← Validator Ops Bootcamp — Runbooks, Incidents, and Recovery

Incident Drills

Lesson 3 of 5·CONTENT32 min100 XP

Treat this page as a workbench, not a blog post. The goal is to extract a reusable mental model from the source and carry it into the rest of the Fabrknt stack.

Course

Validator Ops Bootcamp — Runbooks, Incidents, and Recovery

Lesson role

CONTENT

Sequence

3 / 5

Double-sign incident response

Hands-on steps

Run incident simulation:

cd examples/validator-ops
./scripts/simulate-double-signer.sh
cat artifacts/incident-double-signer.log

Extract containment timeline into runbook:

containment start
secondary signer stop
single-writer verification

Add 3 prevention controls to runbook:

signer lease lock
startup guard
duplicate-signer alert

Pass criteria

Incident log includes single-writer=PASS
Runbook has a "Double-sign Containment" section

Verification

grep -q "single-writer=PASS" artifacts/incident-double-signer.log && echo PASS
grep -n "Double-sign Containment" artifacts/validator-ops-runbook.md

← Previous

Observability stack for validator nodes

Next →

Upgrade rollback drill