Extend nitro-testnode to run consensus and execution separately by bragaigor · Pull Request #177 · OffchainLabs/nitro-testnode

bragaigor · 2026-02-13T23:46:25Z

Extend nitro-testnode to run consensus and execution in different processes

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor · 2026-02-19T17:45:23Z

docker-compose.yaml

+  regular-follower-node:
+    pid: host # allow debugging
+    image: nitro-node-dev-testnode
+    entrypoint: /usr/local/bin/nitro
+    ports:
+      - "127.0.0.1:7447:8547"
+      - "127.0.0.1:7548:8548"
+    volumes:
+      - "seqdata:/home/user/.arbitrum/local/nitro"
+      - "l1keystore:/home/user/l1keystore"
+      - "config:/config"
+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/consensus_config.json
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --node.seq-coordinator.my-url=http://sequencer:8547
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --graphql.enable
+      - --graphql.vhosts=*
+      - --graphql.corsdomain=*
+    depends_on:
+      - geth


we don't really need this but it was very helpful in debugging. I've added it behind --follower-node flag, but let me know if you prefer to remove it

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

Copilot

Pull request overview

Extends nitro-testnode to support running a follower node with Nitro consensus and execution split across separate processes (communicating over RPC), and adds a convenience flag for starting a regular follower node.

Changes:

Add --run-consensus-and-execution-in-different-processes and --follower-node flags to test-node.bash to start additional follower node services in simple mode.
Generate an additional consensus_config.json in scripts/config.ts for the new follower node services.
Add new docker-compose services (consensus-follower-node, execution-follower-node, regular-follower-node) and corresponding volumes, plus a small README update.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File	Description
test-node.bash	Adds CLI flags/help text and wires them into the `NODES` set; adjusts init logging/sleep ordering.
scripts/config.ts	Writes a new `consensus_config.json` derived from the base config.
docker-compose.yaml	Introduces new follower node services for split consensus/execution and a regular follower node, with new volumes/ports.
README.md	Documents how to print an account private key via the `print-private-key` script.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

test-node.bash

scripts/config.ts

docker-compose.yaml

test-node.bash

docker-compose.yaml

README.md

pmikolajczyk41

so running ./test-node.bash --dev --init-force --run-consensus-and-execution-in-different-processes --simple spawned indeed two follower nodes; their logs are not very dynamic, but at least consensus one seems to make progress with lines like:

INFO [03-13|10:00:32.413] InboxTracker                             sequencerBatchCount=9 messageCount=472 l1Block=414 l1Timestamp=2026-03-13T10:00:32+0000

for the execution follower, I guess, I'd have to throw some RPC requests at it

the only thing that looked suspicious was miscommunication between the two:

# consensus follower:
INFO [03-13|09:59:02.958] Reading message result remotely.         msgIdx=294
INFO [03-13|09:59:02.958] rpc response                             method=nitroexecution_resultAtMessageIndex logId=2794 err="result not found"             result=null attempt=0 args=[294] errorData=null
ERROR[03-13|09:59:02.958] Error getting message result             msgIdx=294 err="result not found"

# execution follower
WARN [03-13|09:59:02.958] Served nitroexecution_resultAtMessageIndex reqid=2794 duration="43.625µs" err="result not found"

and this continuous to happen every ~2 minutes; in particular, logs from exec follower look:

WARN [03-13|09:59:02.958] Served nitroexecution_resultAtMessageIndex reqid=2794 duration="43.625µs" err="result not found"
WARN [03-13|10:01:02.619] Served nitroexecution_resultAtMessageIndex reqid=5215 duration="40.541µs" err="result not found"
WARN [03-13|10:02:33.140] Served nitroexecution_resultAtMessageIndex reqid=7036 duration="40.166µs" err="result not found"
WARN [03-13|10:03:33.533] Served nitroexecution_resultAtMessageIndex reqid=8318 duration="37.75µs"  err="result not found"

scripts/config.ts

test-node.bash

docker-compose.yaml

bragaigor · 2026-03-13T13:25:54Z

the only thing that looked suspicious was miscommunication between the two:

I noticed that as well. I'm not sure if that's something related to docker configuration or if we're missing something in the code? Maybe @diegoximenes could shed a light? As in the request above is regarding _resultAtMessageIndex endpoint, but I don't see any other requests being made unless they are not being logged? Do we still need OffchainLabs/nitro#4264 to make some of that work maybe?

I did some further investigation (added some logs to client (consensus-followe) and server (execution-follower)), and I was able to see constant successful communication between them; I was able to spot the following endpoint requests:

_markFeedStart
_headMessageIndex
_setConsensusSyncData
_setFinalityData
_digestMessage

for which all completed successfully. So it seems related to something on server side (execution-follower-node) specific to _resultAtMessageIndex. And after some more digging it seems like whenever a _resultAtMessageIndex is made and execution is behind consensus then execution can't find the block, here are some of the logs I added to execution side:

Time	Requested msgIdx	currentHead	Gap
13:18:59	137	128	9
13:20:29	314	257	57
13:21:59	492	475	17

is it expected for msgIdx to always be behind currentHead? I got currenHead by asking the blockchain s.bc.CurrentBlock(), while msgIdx is the requested msgIdx.

CC: @pmikolajczyk41

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

README.md

diegoximenes · 2026-03-23T13:14:09Z

scripts/config.ts

+    // Use a different persistent.chain name than the sequencer ("local") to avoid
+    // storage lock conflicts between services sharing the same volume.


This comment is confusing to me.
Which services are sharing the same volume?

when we run either execution-follower-node or regular-follower-node where they share the same volume we acquire a lock on that file:
https://github.com/OffchainLabs/go-ethereum/blob/c21e595c40ec1fdc84d4a73fb6e52d9679597698/node/node.go#L322-L330

docker-compose.yaml

diegoximenes · 2026-03-23T13:20:25Z

docker-compose.yaml

+      - --conf.file=/config/base_node_config.json
+      - --node.feed.output.enable
+      - --http.api=net,web3,eth,txpool,debug,timeboost,auctioneer
+      - --node.seq-coordinator.my-url=http://sequencer:8547


not necessary

only kept --conf.file=

diegoximenes · 2026-03-23T13:21:16Z

docker-compose.yaml

+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/base_node_config.json
+      - --node.seq-coordinator.my-url=http://sequencer:8547


not necessary

only kept --conf.file=

diegoximenes · 2026-03-23T13:22:04Z

docker-compose.yaml

+      - "tokenbridge-data:/tokenbridge-data"
+    command:
+      - --conf.file=/config/base_node_config.json
+      - --node.seq-coordinator.my-url=http://sequencer:8547


not necessary

only kept --conf.file=

diegoximenes · 2026-03-23T13:23:49Z

README.md

+
+### `--run-consensus-and-execution-in-different-processes`
+
+Adds a **split-process follower** where consensus and execution run as separate containers communicating over authenticated RPC.


It is not authenticated, this is the flag being used: --node.rpc-server.authenticated=false

test-node.bash

diegoximenes · 2026-03-23T15:45:15Z

Regarding Error getting message result, this is related to this task.

Regarding is it expected for msgIdx to always be behind currentHead?, which is also related to the task that I linked.
It is OK for the requested msgIdx bigger than current head on Execution side.
ConsensusExecutionSyncer can trigger that.
The issue is if the client is not handling it properly, which is the case today.
It should log a debug and not an error as it is today.
The "real test" is comparing the block hashed from the sequencer and the follower node.

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor · 2026-03-23T22:24:37Z

The "real test" is comparing the block hashed from the sequencer and the follower node.

I confirmed that both regular-follower-node and execution-only-follower-node are deriving the same block hashes as the sequencer

bragaigor mentioned this pull request Feb 13, 2026

Enable execution only mode in nitro OffchainLabs/nitro#4386

Open

bragaigor force-pushed the braga/consensus-exec-diff-proc branch from 7192f2a to 52beb21 Compare February 19, 2026 15:54

Extend nitro-testnode to run consensus and exection separately

3b1d3b7

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor force-pushed the braga/consensus-exec-diff-proc branch from 52beb21 to 3b1d3b7 Compare February 19, 2026 16:56

bragaigor changed the base branch from release to master February 19, 2026 16:56

bragaigor commented Feb 19, 2026

View reviewed changes

bragaigor added 2 commits March 11, 2026 07:47

Merge branch 'master' into braga/consensus-exec-diff-proc

e319343

create consensus-local config and clean up

c5a3814

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor marked this pull request as ready for review March 12, 2026 14:01

bragaigor requested review from diegoximenes and pmikolajczyk41 March 12, 2026 14:01

bragaigor assigned diegoximenes Mar 12, 2026

diegoximenes requested a review from Copilot March 12, 2026 16:47

diegoximenes assigned pmikolajczyk41 and unassigned diegoximenes Mar 12, 2026

Copilot started reviewing on behalf of diegoximenes March 12, 2026 16:47 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

pmikolajczyk41 reviewed Mar 13, 2026

View reviewed changes

scripts/config.ts Outdated Show resolved Hide resolved

test-node.bash Outdated Show resolved Hide resolved

test-node.bash Show resolved Hide resolved

docker-compose.yaml Outdated Show resolved Hide resolved

docker-compose.yaml Outdated Show resolved Hide resolved

pmikolajczyk41 assigned bragaigor and unassigned pmikolajczyk41 Mar 13, 2026

Update comments and docker-compose.yaml

f50d921

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor requested a review from pmikolajczyk41 March 13, 2026 13:29

bragaigor assigned pmikolajczyk41 and unassigned bragaigor Mar 13, 2026

pmikolajczyk41 previously approved these changes Mar 13, 2026

View reviewed changes

pmikolajczyk41 assigned diegoximenes and unassigned pmikolajczyk41 Mar 13, 2026

Update README and add comments

bbeb358

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor dismissed pmikolajczyk41’s stale review via bbeb358 March 13, 2026 13:56

bragaigor commented Mar 13, 2026

View reviewed changes

README.md Show resolved Hide resolved

diegoximenes reviewed Mar 23, 2026

View reviewed changes

diegoximenes assigned bragaigor and unassigned diegoximenes Mar 23, 2026

bragaigor added 2 commits March 23, 2026 18:00

remove unecessary config from docker-compose

677818c

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

revert address on README

c8b2b7b

Signed-off-by: Igor Braga <5835477+bragaigor@users.noreply.github.com>

bragaigor requested a review from diegoximenes March 23, 2026 22:20

bragaigor assigned diegoximenes and unassigned bragaigor Mar 23, 2026

		// Use a different persistent.chain name than the sequencer ("local") to avoid
		// storage lock conflicts between services sharing the same volume.


		### `--run-consensus-and-execution-in-different-processes`

		Adds a split-process follower where consensus and execution run as separate containers communicating over authenticated RPC.

Conversation

bragaigor commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmikolajczyk41 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bragaigor commented Mar 13, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

diegoximenes commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bragaigor commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bragaigor commented Feb 13, 2026 •

edited

Loading

diegoximenes commented Mar 23, 2026 •

edited

Loading