← Validator Ops Bootcamp — Runbooks, Incidents, and Recovery

Runbook Foundations

Lesson 2 of 5·CONTENT28 min90 XP

Treat this page as a workbench, not a blog post. The goal is to extract a reusable mental model from the source and carry it into the rest of the Fabrknt stack.

Course

Validator Ops Bootcamp — Runbooks, Incidents, and Recovery

Lesson role

CONTENT

Sequence

2 / 5

Observability stack for validator nodes

Hands-on steps

Simulate lag incident signal:

cd examples/validator-ops
./scripts/simulate-head-lag.sh
cat artifacts/incident-head-lag.log

Triage checklist (write in runbook):

local vs network-wide lag
import latency correlation
peer churn check

Add a "Detection" section to:artifacts/validator-ops-runbook.md.

Expected evidence

artifacts/incident-head-lag.log contains ALERT=PAGE

Verification

grep -q "ALERT=PAGE" artifacts/incident-head-lag.log && echo PASS

← Previous

SLO/SLI for validator operations

Next →

Double-sign incident response