Chapter 20
Protect the invariant before optimizing recovery time
Updated
Automated database failover restores connectivity but leaves divergent writes.
Inspect
Compare the recovery topology with application latency assumptions, replication state, background queues, and the integrity of accepted writes.
Decide
Predefine when to stop writes, pause consumers, serve degraded results, or reconcile. Exercise restore and backlog recovery, not just the failover trigger.
Bring to a review
A recovery drill recording the protected invariant, data-loss exposure, restoration time, and safe replay conditions.
Common trap: A healthy infrastructure dashboard does not prove the recovered application is correct.
Read
- October 21 post-incident analysis A brief network partition exposed incompatible failover and application assumptions.
- Designing Data-Intensive Applications · 2nd edition BookData models, replication, transactions, and partial failure.Focus: Second edition: chapters 5, 6, 8, 9 and 12 on evolution, replication, transactions, failures, and streams. Apply: Defend one consistency contract: stale reads allowed, invariants protected, and recovery under a partition.
- Canarying Releases · The Site Reliability Workbook BookA partial rollout needs a meaningful control and a release decision.Focus: Read representative canaries, comparison signals, and automated rollout decisions. Apply: Define cohort, exposure, observation window, rollback trigger, and how low traffic affects detection.