Lesson 1 — Validator key management: hot keys, HSM, MPC, threshold signatures

Question

It is 3 AM. The staking operator gets paged. A second validator process accidentally started on the standby box — same key, both online, both signed an attestation at the same height. The network sees two valid signatures from the same identity → equivocation → the validator is slashed and forcibly exited. The validator signing key is the validator's economic identity. How do production teams protect it?

Principle (minimum model)

Five requirements that no single solution satisfies. Can sign + never signs twice at the same height/round + not exposed on the public internet + survives operator turnover + survives disaster (HW failure / DC loss).
Four solutions, ordered by sophistication. Config-file hot keys (dev only) → HSM (tamper-resistant hardware, key never leaves) → MPC (split across N-of-M devices) → threshold signatures (BLS, cryptographic share, no reconstruction required).
MPC ≠ threshold signatures. MPC = general-purpose protocol (compute any function over secret shares) / threshold signatures = a specific cryptographic primitive (the signature scheme itself supports secret-sharing natively).
The "two keys" pattern. Withdrawal key (cold, controls stake funds) + signing key (hot, votes / proposes / slashable). Blast radius is bounded: even if the signing key leaks, the funds can't be moved (only slashed).
Four anti-slashing rules. Exactly one signer per identity + slashing-protection DB + fail-closed under uncertainty + survives network partition.
Remote-signer pattern. The validator node holds no keys → it asks the remote signer → the HSM signs after enforcing slashing protection. Web3Signer (Consensys) and Dirk (Attestant) are open-source examples for Ethereum.
Multi-region active-passive. DC1 active + DC2 standby + DC3 cold backup. Transition is manual + only via consensus (to avoid both signing at the same time).

Worked example + steps

Validator key management — hot keys, HSM, MPC, threshold signatures

A staking operator gets paged at 3 AM. A second validator process accidentally started on the standby box — same key, both online, both signing attestations at height 9,801,442. By the time anyone notices, the network has already seen two valid signatures from one identity. That's equivocation (the consensus term for signing two conflicting messages at the same slot), and the protocol slashes them for it. They wake up to a slashed, force-exited validator — for a duplicate process. If many keys were run twice at once, the correlation penalty can take most or all of each stake.

A validator's signing key is its economic identity. Lose it → lose your stake. Leak it → attacker double-signs → slashing → lose your stake. Reuse it → same. This lesson is the operational reality: how production teams keep keys safe, what fails when they don't, and the cryptographic primitives that scale validator sets.

1. The validator key threat model

A validator key must:

Sign blocks/attestations when it's the leader/voter
Never sign two different messages at the same height/round (slashing)
Never be exposed to the open internet
Survive operator turnover (you'll have multiple ops people)
Survive disasters (hardware failure, data center loss)

Each of these is a separate security challenge. No single solution solves all of them.

2. The four solutions, in order of sophistication

2.1 Hot key in a config file

# This is dangerous
echo "0xabc123..." > /var/lib/validator/key.txt

Pros: simple, works. Cons: Anyone with file system access has the key. Backup = clone of key.

Used in development, not production for valuable validators.

2.2 HSM (Hardware Security Module)

An HSM is a tamper-resistant physical device that holds the private key and signs without ever exposing it. AWS CloudHSM, YubiHSM, or dedicated boxes from vendors like Thales.

Workflow:

Validator generates key inside the HSM
Public key is exposed; private key never leaves the device
To sign: validator software sends a hash to the HSM; HSM returns the signature
If validator software is compromised, the attacker can sign anything valid but cannot steal the key itself

Pros: Key never on disk, never in the validator process's memory. Cons: Single device — physical loss = key loss. Backup is hard.

2.3 MPC (Multi-Party Computation)

The key is split across multiple devices, and signing requires N-of-M cooperation. No single device ever holds the full key.

Example: 3 devices in 3 data centers. To sign, 2 of 3 cooperate. Compromise 1 device → attacker has 1/3 of the key, useless. To get to 2/3, you'd need to compromise 2 separate facilities.

Pros: No single device holds the key. Cons: Requires cooperation = latency on every signature. Complex protocol.

2.4 Threshold signatures (the cryptographic version of MPC)

Same idea as MPC, but using threshold signature cryptography (signature schemes specifically designed to be produced by N-of-M shareholders without ever reassembling the key). Each device holds a "share" of the key. Signing produces a normal-looking signature without ever reconstructing the full key.

BLS (a pairing-based signature scheme) supports this natively, which is why it is the usual choice for threshold validator keys:

Each node holds a share of one validator's signing key
Each node signs with its share; any t partial signatures are combined into one final signature
Verifiers don't know it's a threshold signature — they just see a standard BLS sig

Pros: Cryptographically clean. No "reconstruction" step. Cons: Complex setup, key generation ceremony required.

On Ethereum this is what distributed-validator (DVT) clients such as Obol's Charon and SSV do. Tempo also uses BLS threshold keys: its README publishes a "network identity" BLS threshold public key that nodes use to verify consensus finalisation certificates.

MPC is a general protocol to compute functions over secret shares without revealing them — works for any function, including signing. Threshold signatures are a specific cryptographic primitive where signature schemes natively support secret-sharing. Threshold sigs are cleaner; MPC is more flexible.

3. The "two-keys" pattern

The point of this pattern: bound the blast radius of a key leak. Most production validators separate:

Withdrawal key (cold): controls the staked funds. Held offline (paper, hardware wallet)
Signing key (hot): controls voting/proposing. Held online, slashable

If the signing key gets compromised, the attacker can slash the validator (cost: hot stake) but cannot steal funds (withdrawal key is cold). Loss is bounded.

For Ethereum:

Withdrawal credentials (0x01...): cold storage
Validator key (BLS): online for attestations and proposals

For Hyperliquid (per the hyperliquid-dex/node README):

Signer wallet: used only for signing consensus messages (hot)
Validator wallet: holds funds and receives delegation rewards (cold)

4. The slashing-prevention checklist

These four rules are what separates "validator that earns rewards" from "validator that gets slashed." You must guarantee:

Single signer per identity — never run two processes with the same key
Slashing-protection database — track every signed message, refuse to sign anything that would cause slashing
Fail-closed on uncertainty — if you can't verify recent history, don't sign
Network partition tolerance — if you're behind a partition and lose sync, don't sign (you might be on a fork)

Reth is an execution client and has no slashing protection. On Ethereum it lives in the consensus-layer validator client or the remote signer, and EIP-3076 defines the format for moving that history between them. Custom L1s need their own (CometBFT, for example, records the last signed height / round / step in priv_validator_state.json).

Both machines have the same key. Both sign the same epoch's attestation. One is the canonical. The other gets seen by the network as a double-signing event. Slashed. The redundancy attempt becomes the slashing offense.

The fix: active-passive with strict failover (only one node has signing authority at a time, transitions via consensus protocol).

5. The remote signer pattern

Production validators often use remote signers:

[Validator node] --signs blocks via API-->  [Remote signer with HSM]
                                                  |
                                                  +--Tracks signed messages
                                                  +--Refuses dangerous signs
                                                  +--Holds the key in HSM

The validator node never has the key. It connects to a remote signer service that does the actual signing. The remote signer enforces slashing-protection (it refuses to sign two conflicting messages).

Open-source implementations (Ethereum):

Web3Signer (Consensys, Java) — signs for Ethereum consensus and execution clients with keys from external vaults or encrypted on disk; slashing protection in a PostgreSQL database
Dirk (Attestant, Go) — "an Ethereum 2 distributed remote keymanager"; can also split keys across several Dirk instances for threshold signing

Other chains: whether the node can use a remote signer depends on the client, so check its docs. Hyperliquid's node, for example, reads the signer key from a local node_config.json.

6. Multi-region deployment

To survive a data center loss:

Active validator in DC1 (signs blocks)
Standby in DC2 (ready to take over)
Cold backup in DC3 (DR)

The transition between active and standby is the hardest part. It cannot be automatic — risk of both signing simultaneously. Usually:

Active signs block N
Active confirms block N propagated and finalized
Manual operator confirms shutdown
Standby takes signing authority
Standby signs block N+1

For BFT chains with view changes, the cost is a missed slot. For Nakamoto-style chains, even less impact.

7. For my projects

Tempo validator operation

Tempo's public repo (tempoxyz/tempo) manages validators through a ValidatorConfig precompile in which only the contract owner can add validators. If you operate a validator there or on any other L1:

HSM or remote signer for the signing key, if the client supports one
3-region active-passive setup
Double-sign protection in the signing path
Any key that controls funds (stake, rewards) on a dedicated hardware wallet, offline

This is the launch operational checklist for any L1 validator role.

Soltempo / mppsol relayer operations

Any off-chain service that signs with its own key (a bridge relayer, a payment or treasury bot) needs the same discipline:

Don't expose keys in code
Use HSM or similar for production
Rotate keys periodically
Have backups

8. Practice

Compute: if 3 nodes use BLS threshold signing with t=2, how many compromised nodes can still sign? How many before they can sign without consent?
Read Web3Signer docs — slashing-protection section
Identify: what's the slashing risk if you migrate to a new HSM during operation?

9. Reading list

Final check: in one sentence, why is "having backups of the signing key" a slashing vulnerability rather than a feature? If your answer doesn't include "duplicate signers can produce slashable equivocations," re-read §4.

Pass criteria

List the five requirements and explain why no single solution satisfies all of them.
Walk the four solutions in order and the trade-off each makes.
Explain why MPC and threshold signatures are not the same thing.
Describe the two-key pattern and why it bounds the blast radius.
State the four anti-slashing rules with one sentence each.
Sketch the remote-signer pattern and which open-source tools implement it.
Explain why active-passive failover must run through consensus, not a load balancer.

Summary (3 lines)

Validator key management is bounded by five hard requirements; no single tool solves all of them, so production stacks layer HSM + slashing-protection DB + remote signer + active-passive multi-region.
MPC and threshold signatures are different primitives; both solve secret-sharing, but threshold sigs are scheme-native and operationally simpler.
The two-key pattern (withdrawal cold / signing hot) bounds blast radius — even a full signing-key leak cannot move funds, only slash them. Next lesson reads what actually triggers a slashing event.