fix(eth-engine): mint a fresh JWT for each retried Engine API request - #394
Open
Xowiek wants to merge 1 commit into
Open
fix(eth-engine): mint a fresh JWT for each retried Engine API request#394Xowiek wants to merge 1 commit into
Xowiek wants to merge 1 commit into
Conversation
EngineRpc::rpc_request mints the token once, before the request is sent, and RpcRequestBuilder::send reuses that one string on every retry attempt. exchange_capabilities retries with a constant 3s delay and without_max_times(), so it never gives up. The Engine API rejects a token whose iat is more than 60 seconds from the server's clock. If the execution client's auth port takes longer than that to come up, the node keeps resending a token that can no longer be accepted, and no further retry fixes it — even once the EL is healthy. Take the Auth in the builder instead of a finished token and mint one per attempt. Only EngineRpc sets it: the IPC transport carries no JWT and every other Engine method uses NoRetry.
Xowiek
requested review from
ZhiyuCircle,
ancazamfir,
romac and
sergio-mena
as code owners
September 12, 2026 19:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EngineRpc::rpc_requestmints the JWT once, before the request is sent:RpcRequestBuilder::sendthen reuses that one string on every retry attempt, andexchange_capabilitiesretries withENGINE_EXCHANGE_CAPABILITIES_RETRY_RPC— a constant 3s delay andwithout_max_times(), so it never gives up.The Engine API rejects a token whose
iatis more than 60 seconds from the server's clock (JWT_MAX_IAT_DIFFinalloy-rpc-types-engine, which is what reth'sJwtAuthValidatorchecks). So if the execution client's auth port takes longer than a minute to come up, the consensus node keeps resending a token the EL can no longer accept, and no amount of further retrying recovers it — the node stays locked out even after the EL is healthy, until it is restarted.Two attempts 1.1 seconds apart on
maincarry the byte-identical token:The payload is
{"iat":1789237889,"exp":null}in both — the timestamp is frozen at the first attempt.The builder now takes the
Authinstead of a finished token and mints one inside the retry closure. OnlyEngineRpcsets it; the IPC transport carries no JWT, and every other Engine method usesNoRetry, so nothing else changes.The added test asserts the two attempts carry different tokens and that both validate against the secret. It fails on
mainwith the output above.cargo test -p arc-eth-enginepasses (96), as dofmtandclippy.