Lesson 3 — Hot upgrades and coordinated chain upgrades

Question

It's mainnet hardfork day. The new binary rewrites consensus rules. Tens of thousands of validators are scattered across the world — no master switch, no maintenance window, the chain doesn't stop. And yet at 14:13 UTC, every validator still on the canonical chain simultaneously starts producing blocks under the new rules. How?

Principle (minimum model)

• The coordination mechanism is not "everyone upgrades at the same time". The binary itself knows when to switch = the rules are gated on block height or block timestamp.
• Four activation conditions (Reth's ForkCondition). Block (pre-Merge Ethereum forks) + Timestamp (compared with the block header timestamp — every Ethereum fork since Shanghai) + TTD (total difficulty — the Merge, one-shot) + Never (defined but not scheduled).
• Upgrade before activation, and it's fine. After activation, upgraded validators apply new rules; un-upgraded ones produce a stale fork and drop off the network.
• Five-step rollout. Receive announcement → download + verify new binary → deploy before activation → verify deployment → wait for activation.
• What is being upgraded is the chain spec. The hardfork → ForkCondition table ships with the binary.
• Pre-fork dry run. Testnets fork first → any issue delays mainnet (Pectra reached mainnet only after incidents on Holesky and Sepolia and a further run on the new Hoodi testnet).
• Hot fork ≠ hot software update. Hot fork = consensus rules change / hot software update = no restart. A short restart is fine — the slashing-protection DB survives it.
• Four emergency tiers. Stale blocks (self-heals) / invalid state (coordinated rollback) / fund theft (emergency hardfork) / consensus halt (coordinated reset — rare).
• BFT chains are halt-and-recover by design. >1/3 offline → halt → operators recover → resume. Halting is acceptable (no fork, safety preserved).

Worked example + steps

Hot upgrades and coordinated chain upgrades

Picture mainnet hardfork day. The new binary changes consensus rules. Tens of thousands of validators run it, spread across every continent, every cloud, every home setup. There is no master switch. There is no scheduled maintenance window. The chain cannot pause. And yet at 14:13 UTC, every validator that's going to stay on the canonical chain starts producing blocks under the new rules at the same time — and the ones that didn't upgrade quietly fork off into irrelevance. How?

That coordination problem is the hardest operational problem in blockchains: validators must switch rules in lockstep without ever talking to each other directly. This lesson covers the protocol mechanisms and operational drills that make it work.

1. The core mechanism — height-gated rules

The trick is that the binary already knows when to switch. A hardfork is defined by:

• A block height (or timestamp) at which new rules activate
• A set of new rules (consensus, EVM, gas, etc.)

Validators don't all upgrade simultaneously. They upgrade before the activation height. Then at the activation block, every upgraded validator applies the new rules — same block, same instant, no coordination needed. Validators who haven't upgraded continue with old rules and fall off, producing blocks the rest of the network rejects.

Block 999: All validators (old + new code) accept this block
Block 1000: Activation point
Block 1001: Old code validators reject this (it follows new rules);
           new code validators accept it

After activation, the chain follows the new rules. Old code is explicitly invalid.

2. The activation conditions

Activation is typically:

A timestamp fork is not a wall-clock read: every node compares the same header field, so activation is as deterministic as a block number. On post-Merge Ethereum the consensus layer schedules a fork at an epoch, and the execution layer activates at that epoch's start timestamp (Pectra: epoch 364032 = timestamp 1746612311, 2025-05-07 10:05:11 UTC).

Tempo does the same: each T-fork's activation time is a genesis config field (e.g. t1aTime) that becomes ForkCondition::Timestamp (tempoxyz/tempo, crates/chainspec/src/spec.rs).

3. The upgrade coordination protocol

Operators must:

1. Receive the upgrade announcement (Github issue, Discord, etc.)
2. Download and verify the new binary
3. Deploy to all validator nodes before activation
4. Verify the deployment is correct
5. Wait for the activation block — the new rules apply automatically

If any validator misses step 3, they fall off the chain at activation. They can rejoin by upgrading and syncing back up.

For Ethereum: this is exactly how Merge, Shanghai, Cancun, Pectra activations worked. The Ethereum Foundation's mainnet announcement with client releases typically lands about two weeks before activation (Pectra: announced 2025-04-23, activated 2025-05-07).

The 1% produce a "stale fork" with old rules. The 99% follow the canonical chain with new rules. From the perspective of the 99%, the 1% are simply offline (their blocks are rejected). To recover:

1. Operator notices "my validator is producing rejected blocks"
2. Upgrades the binary
3. Validator syncs to the canonical chain (gets blocks from peers)
4. Resumes signing on the canonical chain

No slashing risk (they were on a different fork, not double-signing on the canonical one). Just inactivity penalty.

4. The upgrade is in the chain spec

For a Reth-based chain, upgrades are encoded in the chain spec (the Rust struct that defines a chain's identity — genesis, fork activations, chain ID). Ethereum mainnet's table as Reth consumes it (alloy-hardforks 0.4.7, src/hardfork/ethereum.rs, excerpt):

pub const fn mainnet() -> [(Self, ForkCondition); 21] {
    [
        (Self::Frontier, ForkCondition::Block(MAINNET_FRONTIER_BLOCK)),
        // ...
        (Self::London, ForkCondition::Block(MAINNET_LONDON_BLOCK)),
        // ...
        (
            Self::Paris,
            ForkCondition::TTD {
                activation_block_number: MAINNET_PARIS_BLOCK,
                fork_block: None,
                total_difficulty: MAINNET_PARIS_TTD,
            },
        ),
        (Self::Shanghai, ForkCondition::Timestamp(MAINNET_SHANGHAI_TIMESTAMP)),
        (Self::Cancun, ForkCondition::Timestamp(MAINNET_CANCUN_TIMESTAMP)),
        (Self::Prague, ForkCondition::Timestamp(MAINNET_PRAGUE_TIMESTAMP)),
        (Self::Osaka, ForkCondition::Timestamp(MAINNET_OSAKA_TIMESTAMP)),
        // ...
    ]
}

When the chain spec ships in a new binary version, validators upgrading to that version get the new activation table. At the activation block, the new rules kick in.

The chain spec IS the upgrade.

5. Pre-fork dry runs

Production chains run upgrades on testnets first:

• Schedule mainnet only after the testnets have forked cleanly
• Run the same fork on testnet
• Validate everything works
• If issues found, delay mainnet

This catches bugs that would otherwise be catastrophic. Pectra is the recent example: it hit problems on Holesky and on Sepolia (EF blog, "Sepolia Pectra Incident Update", 2025-03-05), was run again on a new testnet, Hoodi, and only then activated on mainnet on 2025-05-07.

Tempo also runs a public testnet, Moderato, next to mainnet (tempoxyz/tempo README; crates/chainspec/src/genesis/moderato.json).

6. Hot software updates (vs hot fork)

Two different things:

• Hot fork = upgrade consensus rules. This is what we've been discussing.
• Hot software update = upgrade validator software without restart. This is operational.

For hot software updates:

• Validator software is restarted with new version
• During restart, it's offline (small inactivity penalty)
• New version continues from prior chain state

Most chains accept that a brief restart is fine. The slashing-protection database survives the restart, so no risk of double-signing.

Some advanced setups use:

• Active-passive failover — restart passive node first, then transition signing authority, then restart active node
• Live code patching — extremely rare; only for performance fixes that can't tolerate downtime

7. The emergency response playbook

What if a bug is discovered after deployment?

The 2016 DAO fork is the canonical example of the third row: a coordinated hardfork at block 1,920,000 that moved the drained funds to a refund contract.

Every chain eventually has incidents. The validator set + governance must have a documented playbook before launch.

8. The "halt and recover" pattern

For BFT chains (e.g. Tempo, which uses Simplex consensus via Commonware, or Hyperliquid, whose HyperBFT is a HotStuff variant):

• If >1/3 validators go offline, chain halts (a direct consequence of BFT's >2/3 quorum requirement — no quorum means no progress)
• Operators bring validators back online
• Chain resumes producing blocks

Compared to Ethereum (which has inactivity leak to recover from): BFT chains have a cleaner halt-and-recover. Halt is acceptable because the chain doesn't fork or lose safety; it just stops.

For such chains a halt is the designed outcome of a big incident. Better halt than fork.

9. Practice

1. Read Ethereum's Pectra upgrade announcement
2. Identify: what's in the EIP for activation logic?
3. Sketch: your validator setup for an L1 upgrade. What's the deployment sequence?
4. Identify: under what circumstances would you delay a fork activation?

10. Reading list

• Ethereum hardfork list — see how forks have been coordinated
• Ethereum execution-apis EngineAPI — how EL/CL coordinate fork activation
• Cosmos chain upgrade docs

Final check: in one sentence, why is "all validators upgrade at the exact same time" the wrong mental model for a hardfork? If your answer doesn't reference "height-gated rules in the chain spec," re-read §1-2.

Pass criteria

• Explain why "everyone simultaneously upgrades" is not the actual coordination mechanism.
• List the four activation methods and which chains use each.
• State what happens to validators that upgrade after activation, vs before.
• Walk the five-step rollout for a mainnet hardfork.
• Explain what is actually upgraded (the chain spec's hardfork → ForkCondition table).
• Describe the role of testnet dry runs and the Pectra precedent.
• Distinguish hot fork from hot software update.
• Name the four emergency tiers and the response posture for each.

Summary (3 lines)

• Coordinated upgrades work because the binary knows the activation height or timestamp (chain spec); operators upgrade ahead of activation, and the chain switches itself at the gate.
• Four activation methods, four emergency tiers, five-step rollout. Hot fork ≠ hot software update — short restarts are routine because the slashing-protection DB survives them.
• BFT chains are halt-and-recover; halting is acceptable because it preserves safety over liveness. Final quiz tests recall across keys / slashing / upgrades.