fix(devnet): peer the execution and consensus nodes, rename ha-geth to ha-el - #1053
Merged
Conversation
…o ha-el The devnet execution-layer nodes were not peered with each other, and the HA cluster could not reach a usable state at all. Three independent causes, each failing silently: reth had no peers. docker-compose-reth.yml set --disable-discovery but never supplied --trusted-peers, and reth does not read geth's static-nodes.json, so net_peerCount was 0x0 on every reth node. It also mounted no nodekey, so reth minted a random identity per datadir and no peer list could be written ahead of time. The HA execution nodes never peered with each other under either client: they mount static-nodes.json, which lists only morph-el-0 and morph-el-1. The HA consensus nodes never peered with each other either. setup_nodes.py overwrote the persistent_peers generated by --populate-persistent-peers with a hardcoded node-0/node-1/node-2 list. node-1 runs with MORPH_NODE_DERIVATION_VERIFY_MODE=layer1 and never starts tendermint, and node-2 has no compose service, so each ha-node-* had exactly one reachable peer and none of each other. The sequencer hand-over waits for the block pool to report caught up, which never happened, so make devnet-up-cluster* stalled at height 0 - the failure mode already described in ops/README.md. Hardcoding the IDs could not have worked anyway: ops/docker/ has no ha-node0/1/2 directories, so those nodes keep the random key tendermint generates. Changes: - docker-compose-reth.yml points every node at the same nodekey*/ha-nodekey* file geth uses, via --p2p-secret-key, and passes explicit --trusted-peers, so both clients derive the same enode for a node. - static-nodes-cluster.json gives the geth cluster path the peers it lacked. - ha-nodekey0/1/2 and nodekey2 had a trailing newline, which reth rejects with "malformed or out-of-range secret key" while geth tolerates it. Stripped, so one file serves both clients. - setup_nodes.py derives each node ID from the node_key.json that actually ends up installed, and lists only the nodes that run tendermint. Copying the keys moved ahead of building the peer list, since overwriting a node_key.json changes the node's identity. Tendermint RPC now binds 0.0.0.0 so the ports in the README endpoint table are reachable from the host. - docker-compose-cluster.yml is layered before docker-compose-reth.yml. Later -f files win, so the previous order let the cluster file's geth image and entrypoint override the reth override, and devnet-up-cluster-reth silently ran geth for ha-el-*. - ha-geth-0/1/2 renamed to ha-el-0/1/2, since either client can back these services. The .devnet/ha-el* data directories already used that name. Verified with make devnet-clean-build && make devnet-up-cluster-reth: all five execution nodes report 4 peers and identical height, node-0 and the three ha-node-* report 3 peers each, the HA leader produces blocks, and the reth enodes match the values the geth static-nodes files have always used.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughChangesDevnet peering configuration
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk: 🔵 Low · up to The PR fixes devnet node peering and cluster startup, but it also exposes four Tendermint RPC endpoints through host-published ports without established access restrictions and can leave partial configuration after an interrupted setup. It is mergeable with explicit owner awareness and follow-up to constrain RPC access and make setup retries safely regenerate incomplete state. Sequence Diagram(s)sequenceDiagram
participant compose_file_args
participant docker-compose-cluster.yml
participant docker-compose-reth.yml
participant Reth services
compose_file_args->>docker-compose-cluster.yml: add cluster overlay
compose_file_args->>docker-compose-reth.yml: add Reth override last
docker-compose-reth.yml->>Reth services: apply node keys and trusted peers
sequenceDiagram
participant setup_nodes.py
participant node_key.json
participant Tendermint nodes
participant RPC clients
setup_nodes.py->>node_key.json: copy installed keys
setup_nodes.py->>node_key.json: derive peer IDs
setup_nodes.py->>Tendermint nodes: write persistent_peers
Tendermint nodes->>RPC clients: serve RPC on 0.0.0.0:26657
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 3 files. (9 skipped: 9 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
curryxbo
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The devnet execution-layer nodes were not actually peered with each other, and
the cluster could not reach a usable state at all.
Three separate causes, all of which fail silently:
docker-compose-reth.ymlset--disable-discoverybut never supplied
--trusted-peers, and reth does not read geth'sstatic-nodes.json.net_peerCountwas0x0on every reth node. It alsomounted no nodekey, so reth minted a random identity per datadir and no peer
list could have been written in advance.
client. They mount
static-nodes.json, which lists onlymorph-el-0andmorph-el-1, so the three of them only ever reached the two main nodes.setup_nodes.pyoverwrote the
persistent_peersgenerated bytendermint testnet --populate-persistent-peerswith a hardcoded list ofnode-0,node-1,node-2. Of those,node-1runs withMORPH_NODE_DERIVATION_VERIFY_MODE=layer1and never starts tendermint, andnode-2has no compose service, so eachha-node-*had exactly onereachable peer and none of each other. The sequencer hand-over waits for the
block pool to report caught up, which never happens, so
make devnet-up-cluster*stalled at height 0 — the exact failure modealready documented in
ops/README.md.The hardcoded IDs were also unfixable by hand:
ops/docker/has noha-node0/1/2directories, so those nodes keep the random key tendermintgenerates and their IDs differ on every setup.
Changes
Execution-layer peering.
docker-compose-reth.ymlnow points every node atthe same
nodekey*/ha-nodekey*file geth uses, via--p2p-secret-key, andpasses explicit
--trusted-peers. Both clients therefore derive the same enodefor a given node. A new
static-nodes-cluster.jsongives the geth cluster paththe peers it was missing. Peering mirrors the existing geth layout:
morph-el-0/morph-el-1know each other, and the HA nodes dial both plus eachother — which keeps
ha-el-*names out of the non-cluster setup, where they donot resolve.
ha-nodekey0/1/2andnodekey2had a trailing newline, which reth rejects withmalformed or out-of-range secret key(geth accepts it either way). Stripped,so one file works for both clients.
Consensus-layer peering.
setup_nodes.pynow derives each node ID from thenode_key.jsonthat actually ends up installed, and lists only the nodes thatrun tendermint (
node-0and the threeha-node-*). Copying the key files movedahead of building the peer list, because overwriting a
node_key.jsonchangesthe node's identity. Tendermint RPC also binds
0.0.0.0so the published portsin the README endpoint table are reachable from the host.
Compose layering order.
docker-compose-cluster.ymlis now passed beforedocker-compose-reth.yml. Later-ffiles win, so with the previous order thecluster file's
image: morph-geth:latestand geth entrypoint overrode the rethoverride —
make devnet-up-cluster-rethsilently ran geth forha-el-*.Rename.
ha-geth-0/1/2→ha-el-0/1/2, since either client can back theseservices. The
.devnet/ha-el*data directories were already named this way.Verification
make devnet-clean-build && make devnet-up-cluster-reth, then:net_peerCounton all five execution nodes/net_infoonnode-0and the threeha-node-*docker compose configwas also used to confirm the reth override wins forha-el-*and that the geth cluster path is unchanged.python3 -m pytest ops/devnet-morph/tests/test_devnet_config.pycovers thelayering order, the rename, the cluster static-nodes file, the reth key/peer
wiring, the absence of trailing newlines in key files, and that node IDs are
derived after the keys are installed.
Summary by CodeRabbit
New Features
Bug Fixes