Incident Drills

Lesson 4 of 5·CONTENT32 min100 XP

Treat this page as a workbench, not a blog post. The goal is to extract a reusable mental model from the source and carry it into the rest of the Fabrknt stack.

Course

Validator Ops Bootcamp — Runbooks, Incidents, and Recovery

Lesson role

CONTENT

Sequence

4 / 5

Upgrade rollback drill

Hands-on steps

Run rollback drill:

cd examples/validator-ops
./scripts/rollback-drill.sh
cat artifacts/rollback-drill.log

Document policy in runbook:

canary window
rollback triggers
post-rollback checks

Pass criteria

Log includes recovery PASS
Runbook has "Rollback Procedure" section

Verification

grep -q "recovery PASS" artifacts/rollback-drill.log && echo PASS
grep -n "Rollback Procedure" artifacts/validator-ops-runbook.md

← Previous

Double-sign incident response

Capstone — production runbook review