Lesson 14 — Testing Stage and ExEx — fixture chains, in-process nodes, golden state

Question

Testing patterns specific to Reth components. Fixture chains (small chains for stage tests), in-process node tests (full pipeline), golden state (deterministic reproducibility).

Principle (minimum model)

• Two layers. Unit tests that call a Stage or drive an ExEx directly, and in-process node tests built with NodeBuilder::testing_node. All the helpers sit behind test-utils cargo features.
• TestStageDB (reth_stages::test_utils). A fresh temp MDBX database plus static-file and RocksDB dirs for each test. db.factory.provider_rw() gives you the provider you pass to Stage::execute / unwind.
• Stage methods are synchronous. fn execute(&mut self, provider: &Provider, input: ExecInput) -> Result<ExecOutput, StageError>. Async readiness lives in poll_execute_ready. Reth's own stage tests seed random blocks from reth_testing_utils::generators and use a crate-private stage_test_suite_ext! macro.
• ExEx harness. reth_exex_test_utils::test_exex_context() returns (ExExContext, TestExExHandle), backed by a temp DB with the mainnet genesis already inserted and a temp WAL. The handle has send_notification_chain_committed / _reorged / _reverted, plus assert_event_finished_height(BlockNumHash) and assert_events_empty(). PollOnce::poll_once() drives your ExEx until it returns Pending.
• In-process node. NodeBuilder::new(NodeConfig::test()).testing_node(runtime).node(EthereumNode::default()).launch() boots a full node on a throwaway datadir. Reth's e2e tests in crates/ethereum/node/tests/e2e/ start exactly like this.
• Real-chain correctness in Reth's CI is not a golden file. ef-tests replays the ethereum/tests and execution-spec-tests fixtures and checks every block's state root. stage.yml runs each stage with reth stage run over mainnet blocks 0–50,000 in the merge queue (its job runs only on merge_group events), so a change is checked before it lands on main. sync.yml syncs mainnet to block 100,000 every 6 hours, then unwinds.

Worked example + steps

Testing Stage and ExEx — fixture chains, in-process nodes, golden state

You walked the Stage trait, the ExEx API, and the NodeBuilder SDK. Now: how does Reth — and the apps that extend it — verify any of this works? A node component that "looks correct" in dev can silently corrupt state for thousands of users in production. The patterns below are the test tooling Reth itself ships, and how every ExEx-based app you build in the Building tier should be tested.

The two test layers any Reth-extending app needs

Reth's own crates use both. Production extensions (ExEx indexers, MEV ExExes, custom app-chains) need both too. Skip either and a class of bugs gets through.

1. Stage unit testing — the canonical pattern

A Stage impl has two required methods besides id: execute (forward sync) and unwind (reorg rollback). Both are plain synchronous functions of the provider and the input checkpoint. The pattern: make a temp DB, seed pre-state, call execute, assert; then call unwind and assert the rollback.

Below is the smallest version that compiles and passes against Reth eb4c15e (deps: reth-stages with test-utils, reth-stages-api):

use reth_stages::test_utils::TestStageDB;
use reth_stages_api::{
    ExecInput, ExecOutput, Stage, StageCheckpoint, StageError, StageId, UnwindInput, UnwindOutput,
};

/// A trivial stage: "processes" every block up to the target and records the checkpoint.
struct CountStage;

impl<Provider> Stage<Provider> for CountStage {
    fn id(&self) -> StageId {
        StageId::Other("Count")
    }

    fn execute(&mut self, _provider: &Provider, input: ExecInput) -> Result<ExecOutput, StageError> {
        Ok(ExecOutput::done(StageCheckpoint::new(input.target())))
    }

    fn unwind(
        &mut self,
        _provider: &Provider,
        input: UnwindInput,
    ) -> Result<UnwindOutput, StageError> {
        Ok(UnwindOutput { checkpoint: StageCheckpoint::new(input.unwind_to) })
    }
}

#[test]
fn execute_then_unwind() {
    let db = TestStageDB::default(); // fresh temp MDBX + static files per test
    let provider = db.factory.provider_rw().unwrap();
    let mut stage = CountStage;

    let out = stage.execute(&provider, ExecInput { target: Some(10), checkpoint: None }).unwrap();
    assert_eq!(out, ExecOutput::done(StageCheckpoint::new(10)));

    let out = stage
        .unwind(&provider, UnwindInput { checkpoint: out.checkpoint, unwind_to: 5, bad_block: None })
        .unwrap();
    assert_eq!(out.checkpoint.block_number, 5);

    provider.commit().unwrap();
}

A real stage reads and writes tables through provider, so a real test seeds them first: TestStageDB::insert_blocks(...) with blocks from reth_testing_utils::generators::random_block_range, then checks the tables after execute and after unwind. create_test_provider_factory() (reth_provider::test_utils) gives you the same kind of throwaway factory without the stage helpers. Either way every test gets its own ephemeral database: no shared state, no cleanup code, no stale fixtures.

🔍 Find in repo. Open paradigmxyz/reth at crates/stages/stages/src/stages/sender_recovery.rs and scroll to mod tests. execute_single_transaction seeds random blocks, runs the stage through a test runner, and asserts the exact ExecOutput. The runner (crates/stages/stages/src/test_utils/runner.rs) is just provider_rw() → stage.execute(&provider, input) → commit(), which is what the test above does by hand.

2. ExEx unit testing — the harness pattern

reth-exex-test-utils gives you an ExExContext and a handle for feeding it synthetic notifications, without booting a node. This test compiles and passes against Reth eb4c15e. An ExEx that keeps +1 per commit and −1 per revert gets 3 commits and 1 revert:

use futures::TryStreamExt;
use reth_execution_types::{Chain, ExecutionOutcome};
use reth_exex::{ExExContext, ExExEvent, ExExNotification};
use reth_exex_test_utils::{test_exex_context, PollOnce};
use reth_node_api::FullNodeComponents;
use reth_trie::{updates::TrieUpdatesSorted, HashedPostStateSorted};
use reth_trie_common::LazyTrieData;
use std::{
    pin::pin,
    sync::{
        atomic::{AtomicI64, Ordering},
        Arc,
    },
};

/// +1 per committed chain, -1 per reverted chain.
async fn counter_exex<Node: FullNodeComponents>(
    mut ctx: ExExContext<Node>,
    count: Arc<AtomicI64>,
) -> eyre::Result<()> {
    while let Some(notification) = ctx.notifications.try_next().await? {
        match &notification {
            ExExNotification::ChainCommitted { .. } => {
                count.fetch_add(1, Ordering::SeqCst);
            }
            ExExNotification::ChainReverted { .. } => {
                count.fetch_sub(1, Ordering::SeqCst);
            }
            ExExNotification::ChainReorged { .. } => {}
        }
        if let Some(committed) = notification.committed_chain() {
            ctx.events.send(ExExEvent::FinishedHeight(committed.tip().num_hash()))?;
        }
    }
    Ok(())
}

#[tokio::test]
async fn three_commits_one_revert() -> eyre::Result<()> {
    let (ctx, mut handle) = test_exex_context().await?;
    let head = ctx.head;
    let genesis = handle.genesis.clone();
    let chain = || {
        Chain::from_block(
            genesis.clone(),
            ExecutionOutcome::default(),
            LazyTrieData::ready(
                Arc::new(HashedPostStateSorted::default()),
                Arc::new(TrieUpdatesSorted::default()),
            ),
        )
    };

    let count = Arc::new(AtomicI64::new(0));
    let mut exex = pin!(counter_exex(ctx, count.clone()));

    for _ in 0..3 {
        handle.send_notification_chain_committed(chain()).await?;
        exex.poll_once().await?; // drive the ExEx until it is Pending again
        handle.assert_event_finished_height((head.number, head.hash).into())?;
    }
    handle.send_notification_chain_reverted(chain()).await?;
    exex.poll_once().await?;
    handle.assert_events_empty(); // a pure revert has no committed chain -> no FinishedHeight

    assert_eq!(count.load(Ordering::SeqCst), 2);
    Ok(())
}

Three details that trip people up:

• You poll; nothing runs in the background. The test owns the ExEx future and calls poll_once(). If the future finishes instead of returning Pending, poll_once returns PollOnceError::FutureIsReady, which matches production, where a returning ExEx brings the node down.
• The notification channel holds one message. test_exex_context creates it with capacity 1, so send → poll → send. Two sends in a row without a poll will block.
• The handle checks events synchronously. assert_event_finished_height takes a BlockNumHash and returns eyre::Result. The only block the harness gives you is handle.genesis, so build any extra fixture blocks yourself.

The shape is taken from the test at the bottom of reth-exex-examples/minimal/src/main.rs. The Arc<AtomicI64> is there so the test can read state the ExEx owns. The Reth Book's Tracking State chapter turns the ExEx into a struct for the same reason. Without the harness you would boot a node, replay real blocks and wait for events. With it, every notification is a single function call.

3. NodeBuilder integration testing — in-process Reth

Sometimes the unit-test layer isn't enough. If your code depends on the interaction between components (a custom pool builder that relies on what the payload builder does, say), you need the assembled node. This boots a whole Ethereum node in-process and passes against Reth eb4c15e (reth-node-builder with test-utils):

use reth_node_builder::{NodeBuilder, NodeConfig, NodeHandle};
use reth_node_ethereum::EthereumNode;
use reth_provider::BlockNumReader;
use reth_tasks::Runtime;

#[tokio::test]
async fn boots_in_process() -> eyre::Result<()> {
    let runtime = Runtime::test();
    let config = NodeConfig::test().with_unused_ports();
    let NodeHandle { node, node_exit_future: _ } = NodeBuilder::new(config)
        .testing_node(runtime.clone()) // ephemeral temp datadir
        .node(EthereumNode::default())
        .launch()
        .await?;

    assert_eq!(node.provider.best_block_number()?, 0);
    Ok(())
}

testing_node(...) points the datadir at a temp directory and opens a test database there. launch() gives you a FullNode whose public fields (node.pool, node.provider, node.network, node.payload_builder_handle) you drive directly. To plug in your own component, swap .node(EthereumNode::default()) for .with_types::<EthereumNode>().with_components(EthereumNode::components().pool(MyPoolBuilder)).with_add_ons(EthereumAddOns::default()), the shape used in crates/ethereum/node/tests/it/builder.rs. Reth's e2e tests (crates/ethereum/node/tests/e2e/pool.rs, dev.rs, …) wrap the launched node in reth_e2e_test_utils::NodeTestContext to submit transactions and advance blocks. On a laptop the test above takes about 3.6 s, against 0.3 s for the stage test, because each test does a full launch. It is still the only layer that catches cross-component bugs.

4. Fixture-chain testing — when canned data isn't enough

For the heaviest checks (does the whole pipeline produce the right state on real history?) Reth doesn't keep a home-made "golden DB" file. Its CI uses three real mechanisms (paths in the eb4c15e checkout):

The "golden" values are external: the fixture's expected post-state, or the block hashes and state roots of the real chain. A regression in any stage shows up as a state-root or hash mismatch, which is a consensus bug, caught before it ships. For your own extension, the reusable pieces are the ef-tests runner pattern (execute, then compare against a published post-state) and reth_e2e_test_utils, which can import a chain RLP file before starting nodes (crates/e2e-test-utils/src/setup_import.rs).

How this connects to the Building tier

The Building tier's test-gate lesson sets this gate for its ExEx-indexer lab:

Fixture chain replay — feed a known sequence of ChainCommitted / ChainReverted notifications and assert your derived state matches a golden reference.

That is §2 of this lesson at the application layer: the same harness, the same synthetic notifications. When you reach Building, this lesson is the precondition.

Drill

1. Read one Reth stage test end-to-end. Open crates/stages/stages/src/stages/sender_recovery.rs in paradigmxyz/reth and read execute_single_transaction top to bottom. Map every helper (TestStageDB, insert_blocks, ExecInput, StageCheckpoint, the runner's execute) to where it's defined under crates/stages/stages/src/test_utils/. 30 minutes.
2. Read one ExEx test using the harness. Reth itself has no in-tree callers of test_exex_context(). Read the test in reth-exex-examples/minimal/src/main.rs and then crates/exex/test-utils/src/lib.rs in Reth. Trace what test_exex_context() builds (temp DB, genesis, WAL, channels) and how the test drives notifications. 30 minutes.
3. Write a stage unit test from scratch. Take the CountStage above. Test that execute advances the checkpoint and unwind rolls it back, using TestStageDB::default(). Get cargo test green. Then make the stage actually read the Headers table and seed it with insert_headers. 1.5 hours.
4. Write an ExEx unit test from scratch. Reproduce the counter test above without looking: 3 commits + 1 revert, counter ends at 2. Then add a send_notification_chain_reorged(old, new) case and decide what the counter should do. 1 hour.
5. Run a NodeBuilder integration test. Read crates/ethereum/node/tests/e2e/dev.rs or pool.rs (they call testing_node()), run them with cargo test -p reth-node-ethereum --test e2e, and read what each test asserts. 1 hour.

After drill 5 you can test any custom Reth component you ship with the same tools the Reth maintainers use.

📺 Further reading

• reth_exex_test_utils — generated docs for the ExEx test harness
• Reth Book — ExEx: Tracking State — why a struct-based ExEx is easier to assert on
• testing/ef-tests in the Reth repo — how consensus fixtures are replayed and checked

Expert continuation

Test discipline at the Reth-component level. At the systems level, two Expert lessons take this further:

• The differential-fuzzing lesson — multi-implementation consensus-correctness testing
• The systems-code auditing lesson — reading Reth patches with an auditor's eye

Summary (3 lines)

• Stages: TestStageDB + synchronous execute / unwind calls on a provider_rw(). Whole node: NodeBuilder::testing_node(...) in-process.
• ExEx: test_exex_context() + send_notification_chain_* + poll_once() + assert_event_finished_height; reorgs via send_notification_chain_reorged / _reverted.
• Real-chain correctness in Reth CI = ef-tests state roots + reth stage run on mainnet 0–50k + a pinned 100k-block mainnet sync. The pattern transfers to any Reth-based codebase.