Lesson 3 — Hot upgrades and coordinated chain upgrades

Question

It's mainnet hardfork day. The new binary rewrites consensus rules. Tens of thousands of validators are scattered across the world — no master switch, no maintenance window, the chain doesn't stop. And yet at 14:13 UTC, every validator still on the canonical chain simultaneously starts producing blocks under the new rules. How?

Principle (minimum model)

• The coordination mechanism is not "everyone upgrades at the same time". The binary itself knows when to switch = the rules are height-gated.
• Four activation methods. Block height (deterministic, works for PoW and PoS) + timestamp (wall-clock, less precise) + difficulty (PoW historical) + total difficulty (Ethereum Merge, one-shot).
• Upgrade before activation, and it's fine. After activation, upgraded validators apply new rules; un-upgraded ones produce a stale fork and drop off the network.
• Five-step rollout. Receive announcement → download + verify new binary → deploy before activation → verify deployment → wait for activation.
• What is being upgraded is the chain spec. The activation_block_number table ships with the binary.
• Pre-fork dry run. Testnet does the same fork 2–3 weeks earlier → any issue delays mainnet (Pectra was delayed twice).
• Hot fork ≠ hot software update. Hot fork = consensus rules change / hot software update = no restart. A short restart is fine — the slashing-protection DB survives it.
• Four emergency tiers. Stale blocks (self-heals) / invalid state (coordinated rollback) / fund theft (emergency hardfork) / consensus halt (coordinated reset — rare).
• BFT chains are halt-and-recover by design. >1/3 offline → halt → operators recover → resume. Halting is acceptable (no fork, safety preserved).

Worked example + steps

Hot upgrades and coordinated chain upgrades

Picture mainnet hardfork day. The new binary changes consensus rules. Tens of thousands of validators run it, spread across every continent, every cloud, every home setup. There is no master switch. There is no scheduled maintenance window. The chain cannot pause. And yet at 14:13 UTC, every validator that's going to stay on the canonical chain starts producing blocks under the new rules at the same time — and the ones that didn't upgrade quietly fork off into irrelevance. How?

That coordination problem is the hardest operational problem in blockchains: validators must switch rules in lockstep without ever talking to each other directly. This lesson covers the protocol mechanisms and operational drills that make it work.

1. The core mechanism — height-gated rules

The trick is that the binary already knows when to switch. A hardfork is defined by:

• A block height (or timestamp) at which new rules activate
• A set of new rules (consensus, EVM, gas, etc.)

Validators don't all upgrade simultaneously. They upgrade before the activation height. Then at the activation block, every upgraded validator applies the new rules — same block, same instant, no coordination needed. Validators who haven't upgraded continue with old rules and fall off, producing blocks the rest of the network rejects.

Block 999: All validators (old + new code) accept this block
Block 1000: Activation point
Block 1001: Old code validators reject this (it follows new rules);
           new code validators accept it

After activation, the chain follows the new rules. Old code is explicitly invalid.

2. The activation conditions

Activation is typically:

Modern PoS chains use timestamp for human-readability and block height for precision. Casper FFG uses epoch boundary.

For Tempo or Hyperliquid: timestamp-based activation. "At Unix timestamp X, switch to fork Y."

3. The upgrade coordination protocol

Operators must:

1. Receive the upgrade announcement (Github issue, Discord, etc.)
2. Download and verify the new binary
3. Deploy to all validator nodes before activation
4. Verify the deployment is correct
5. Wait for the activation block — the new rules apply automatically

If any validator misses step 3, they fall off the chain at activation. They can rejoin by upgrading and syncing back up.

For Ethereum: this is exactly how Merge, Shanghai, Cancun, Pectra activations worked. Validators have 1-2 weeks of warning to upgrade.

The 1% produce a "stale fork" with old rules. The 99% follow the canonical chain with new rules. From the perspective of the 99%, the 1% are simply offline (their blocks are rejected). To recover:

1. Operator notices "my validator is producing rejected blocks"
2. Upgrades the binary
3. Validator syncs to the canonical chain (gets blocks from peers)
4. Resumes signing on the canonical chain

No slashing risk (they were on a different fork, not double-signing on the canonical one). Just inactivity penalty.

4. The upgrade is in the chain spec

For a Reth-based chain, upgrades are encoded in the chain spec (the Rust struct that defines a chain's identity — genesis, fork heights, chain ID). From Lesson 5 of Course 1 (Consensus Engineering):

pub enum CustomHardfork {
    Bedrock,
    Canyon,
    Ecotone,
    // ...
}

impl CustomHardfork {
    pub fn activation_block_number(&self, chain: &CustomChain) -> Option<u64> {
        match (self, chain) {
            (Self::Bedrock, CustomChain::Mainnet) => Some(105_235_063),
            (Self::Canyon, CustomChain::Mainnet) => Some(125_000_000),
            // ...
        }
    }
}

When the chain spec ships in a new binary version, validators upgrading to that version get the new activation table. At the activation block, the new rules kick in.

The chain spec IS the upgrade.

5. Pre-fork dry runs

Production chains run upgrades on testnets first:

• Hold mainnet activation 2-3 weeks out
• Run the same fork on testnet
• Validate everything works
• If issues found, delay mainnet

This catches bugs that would otherwise be catastrophic. Ethereum's Pectra fork delay (twice) was due to testnet issues caught during dry runs.

For Tempo: there's likely Tempo Moderato (testnet) for exactly this. The fork sequence goes Moderato → Mainnet, with weeks between.

6. Hot software updates (vs hot fork)

Two different things:

• Hot fork = upgrade consensus rules. This is what we've been discussing.
• Hot software update = upgrade validator software without restart. This is operational.

For hot software updates:

• Validator software is restarted with new version
• During restart, it's offline (small inactivity penalty)
• New version continues from prior chain state

Most chains accept that a brief restart is fine. The slashing-protection database survives the restart, so no risk of double-signing.

Some advanced setups use:

• Active-passive failover — restart passive node first, then transition signing authority, then restart active node
• Live code patching — extremely rare; only for performance fixes that can't tolerate downtime

7. The emergency response playbook

What if a bug is discovered after deployment?

The 2016 DAO incident was a coordinated hardfork to recover stolen funds. The 2024 incident on Polkadot (validator misbehavior) was a coordinated rollback. Each of these had ~24 hour response cycles.

For Tempo: there will eventually be incidents. The validator set + governance must have a documented playbook before launch.

8. The "halt and recover" pattern

For purely BFT chains (Tempo, Hyperliquid):

• If >1/3 validators go offline, chain halts (a direct consequence of BFT's >2/3 quorum requirement — no quorum means no progress)
• Operators bring validators back online
• Chain resumes producing blocks

Compared to Ethereum (which has inactivity leak to recover from): BFT chains have a cleaner halt-and-recover. Halt is acceptable because the chain doesn't fork or lose safety; it just stops.

For Tempo: this means outages are by design during big incidents. Better halt than fork.

9. Practice

1. Read Ethereum's Pectra upgrade announcement
2. Identify: what's in the EIP for activation logic?
3. Sketch: your validator setup for an L1 upgrade. What's the deployment sequence?
4. Identify: under what circumstances would you delay a fork activation?

10. Reading list

• Ethereum hardfork list — see how forks have been coordinated
• Ethereum execution-apis EngineAPI — how EL/CL coordinate fork activation
• Cosmos chain upgrade docs

Final check: in one sentence, why is "all validators upgrade at the exact same time" the wrong mental model for a hardfork? If your answer doesn't reference "height-gated rules in the chain spec," re-read §1-2.

Pass criteria

• Explain why "everyone simultaneously upgrades" is not the actual coordination mechanism.
• List the four activation methods and which chains use each.
• State what happens to validators that upgrade after activation, vs before.
• Walk the five-step rollout for a mainnet hardfork.
• Explain what is actually upgraded (the chain spec / activation_block_number).
• Describe the role of testnet dry runs and the Pectra precedent.
• Distinguish hot fork from hot software update.
• Name the four emergency tiers and the response posture for each.

Summary (3 lines)

• Coordinated upgrades work because the binary knows the activation height (chain spec); operators upgrade ahead of activation, and the chain switches itself at the gate.
• Four activation methods, four emergency tiers, five-step rollout. Hot fork ≠ hot software update — short restarts are routine because the slashing-protection DB survives them.
• BFT chains are halt-and-recover; halting is acceptable because it preserves safety over liveness. Final quiz tests recall across keys / slashing / upgrades.