Lesson 11 — Running a Reth fork in production

Question

Running a custom Reth fork in production. Disciplines: chainspec + upgrade path + monitoring + emergency response.

Principle (minimum model)

• Chainspec. Custom genesis + custom forks. Deterministic; reproducible.
• Upgrade path. Add a new fork at a future block or timestamp (ForkCondition::Block / ForkCondition::Timestamp). Coordinate with users; staged rollout.
• Monitoring. Prometheus + Grafana. Metrics: sync time, peer count, mempool size, gas usage.
• Alerts. Page on critical failures (sync stalled, peer count < 5, OOM, disk filling).
• Emergency response. Documented procedures for: stuck chain, fork resolution, key compromise. Practiced.
• Coordination. Telegram / Discord channels for validators. Status page for users.
• Validator discipline. Documented procedures + on-call rotation + post-mortem culture.
• Production examples of Reth-SDK chains. op-reth (OP Stack, now in ethereum-optimism/optimism under rust/op-reth), Tempo (tempoxyz/tempo), Berachain (berachain/bera-reth). Each has its own node binary built on Reth crates and 24/7 ops.

Worked example + steps

Running a Reth fork in production

It's 3 a.m. Your validator stopped producing blocks 40 minutes ago. The dashboard shows: file-descriptor exhaustion, MDBX page-cache pressure, peer count at 2. None of these would have shown up in unit tests. None of them would have shown up the day you shipped. They show up at month 3, all at once. This lesson is the ops checklist that prevents that 3 a.m. page — build flags, systemd limits, diff testing against vanilla, the deployment topology that lets your fork survive contact with reality.

1. Build & release pipeline

# Max-performance build — what reth's `make maxperf` runs
RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf
# binary: target/maxperf/reth

jemalloc and asm-keccak are already default features of bin/reth, and the release profile already sets strip = "symbols" (use --profile maxperf-symbols if you need symbols for profiling). The trap is your fork's own binary crate: on the library crates (reth-node-core, reth-cli-util, alloy-primitives) these are opt-in features, so copy bin/reth's feature wiring or you ship without them.

2. Systemd unit (or equivalent)

[Service]
ExecStart=/usr/local/bin/reth node --chain /etc/reth/genesis.json --datadir /var/lib/reth --metrics 127.0.0.1:9001
Restart=on-failure
LimitNOFILE=1048576
LimitNPROC=infinity
TasksMax=infinity

The file-descriptor limit matters: P2P connections, RPC connections and the database/static files all consume fds. Reth raises its soft limit to the hard limit at startup (fdlimit::raise_fd_limit in the node launcher), so the limit the unit sets is the real ceiling.

3. Storage discipline

• Separate volumes for DB and logs. Never let logs fill the DB partition.
• NVMe SSDs only. Spinning rust will not keep up with execution.
• Snapshot regularly. reth db copy <dest> copies the MDBX database (bundled mdbx_copy; the static_files directory is separate and must be backed up too), or use filesystem-level snapshots with the node stopped.
• Plan for growth. For Ethereum mainnet, Reth's system requirements list at least 1.2 TB for a full node and 2.8 TB for an archive node, and it keeps growing.

4. Monitoring

What to alert on:

Reth exposes Prometheus metrics when started with --metrics <addr> (DB gauges such as db.table_size and db.freelist included), and ships Grafana dashboards in etc/grafana/dashboards/. Wire them up with alerting on rates of change, not just absolute values.

5. Diff testing

If your fork modifies execution, run continuous diff testing against vanilla Reth on the same blocks:

# Pseudo-code for a diff harness
for block in mainnet[recent_1000]:
    s1 = reth_vanilla.execute(block)
    s2 = reth_fork.execute(block)
    if s1.stateRoot != s2.stateRoot:
        alert("divergence at block", block, s1, s2)

Any unintended divergence — even one storage slot — means a consensus bug. Bug = chain halt for an App-chain.

6. Deployment topology for an App-chain

Minimum:

• ≥ 4 validators in 3 datacenters
• 2 sentries in front of each validator
• A separate archive node (not a validator) for analytics queries
• A separate RPC fleet with rate-limiting and a CDN

Don't run the validator and the public RPC on the same machine. One DDoS and your chain stalls.

7. Upgrade procedure

The hardest part of running a fork is upgrading it without halting the chain.

1. Announce a target activation block height (or timestamp)
2. Ship a new binary whose chainspec schedules the fork at that point — before activation it behaves exactly like the old one
3. At the activation point, the consensus rule changes — guarded by the hardfork check (ForkCondition::Block / ForkCondition::Timestamp)
4. Nodes that haven't upgraded reject the new-rule blocks and fall off — that's why scheduled activation + announcement matters

This is how Ethereum hard forks work (pre-Merge by block number, post-Merge by timestamp); an App-chain is no different, just smaller scale.

8. Reading list

• Reth Book "Run a node" + "Custom chain" sections
• The validator ops post-mortem from any major chain incident — they're gold for ops intuition

You now have a complete picture: develop, profile, extend, deploy, monitor. Welcome to the small club.

Final check: in one sentence, why is "diff testing against vanilla Reth" the highest-value test you can write for a fork? What class of bug does it catch that no unit test ever will? If your answer doesn't mention "consensus" or "the only output that matters is stateRoot," re-read Section 5.

Expert continuation

Running a Reth fork in production sits at the intersection of three Expert lessons in this course:

• The systems-code auditing lesson — reviewing your fork diff with auditor discipline before every upgrade
• The chaos-engineering lesson — controlled failure injection so your fork survives the same failure modes upstream Reth survives
• The open-source contributor lesson — upstreaming generic fixes back to Paradigm/Reth so your fork's diff stays small

Summary (3 lines)

• Production Reth fork = chainspec + upgrade path + monitoring + emergency response.
• Monitoring: Prometheus + Grafana + alerts. Emergency procedures documented + practiced.
• Coordination via Telegram/Discord + status page. Production examples: op-reth / Tempo / Berachain (bera-reth).