Skip to content

fix(eth-engine): mint a fresh JWT for each retried Engine API request - #394

Open
Xowiek wants to merge 1 commit into
circlefin:mainfrom
Xowiek:fix/engine-jwt-per-attempt
Open

fix(eth-engine): mint a fresh JWT for each retried Engine API request#394
Xowiek wants to merge 1 commit into
circlefin:mainfrom
Xowiek:fix/engine-jwt-per-attempt

Conversation

@Xowiek

@Xowiek Xowiek commented Sep 12, 2026

Copy link
Copy Markdown

EngineRpc::rpc_request mints the JWT once, before the request is sent:

.retry(retry_policy)
.bearer_auth(self.auth.generate_token()?)
.send()

RpcRequestBuilder::send then reuses that one string on every retry attempt, and exchange_capabilities retries with ENGINE_EXCHANGE_CAPABILITIES_RETRY_RPC — a constant 3s delay and without_max_times(), so it never gives up.

The Engine API rejects a token whose iat is more than 60 seconds from the server's clock (JWT_MAX_IAT_DIFF in alloy-rpc-types-engine, which is what reth's JwtAuthValidator checks). So if the execution client's auth port takes longer than a minute to come up, the consensus node keeps resending a token the EL can no longer accept, and no amount of further retrying recovers it — the node stays locked out even after the EL is healthy, until it is restarted.

Two attempts 1.1 seconds apart on main carry the byte-identical token:

left:  eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE3ODkyMzc4ODksImV4cCI6bnVsbH0...
right: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE3ODkyMzc4ODksImV4cCI6bnVsbH0...

The payload is {"iat":1789237889,"exp":null} in both — the timestamp is frozen at the first attempt.

The builder now takes the Auth instead of a finished token and mints one inside the retry closure. Only EngineRpc sets it; the IPC transport carries no JWT, and every other Engine method uses NoRetry, so nothing else changes.

The added test asserts the two attempts carry different tokens and that both validate against the secret. It fails on main with the output above. cargo test -p arc-eth-engine passes (96), as do fmt and clippy.

EngineRpc::rpc_request mints the token once, before the request is sent, and
RpcRequestBuilder::send reuses that one string on every retry attempt.
exchange_capabilities retries with a constant 3s delay and without_max_times(),
so it never gives up.

The Engine API rejects a token whose iat is more than 60 seconds from the
server's clock. If the execution client's auth port takes longer than that to
come up, the node keeps resending a token that can no longer be accepted, and
no further retry fixes it — even once the EL is healthy.

Take the Auth in the builder instead of a finished token and mint one per
attempt. Only EngineRpc sets it: the IPC transport carries no JWT and every
other Engine method uses NoRetry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant