All articles

A deployment is only as good as its recovery plan

Build a lightweight release process around immutable artifacts, observability, and a rollback procedure you can actually rehearse.

A deployment is only as good as its recovery plan

Keep a known-good artifact

A reliable release starts with a build you can identify. Associate a source revision with each deployable artifact and keep the previous successful artifact available. Dependencies should resolve from a checked-in lockfile. Runtime secrets should come from the environment so the same artifact can run under different environments without carrying private values.

On a small Linux host this can be as simple as keeping each release in its own directory and pointing the service at the current release. On a hosted platform the equivalent is promoting a deployment that has already passed your checks. In either case, verify that a restart uses the expected revision and that you can still route traffic back to the previous version.

Design the migration for overlap

Application versions can overlap during a rollout. Add new columns and tables first, then release code that understands the new schema. Backfill existing rows in batches and check the count and constraints afterward. Remove old columns in a later release only after you confirm that all required readers have moved. This sequence lets the old application keep running during the transition.

Rollback cannot recover data deleted by a destructive migration. Keep a tested backup and document the recovery point before changing production data. A successful backup log is useful evidence, but periodically restoring to a separate environment is how you measure whether the recovery is realistic.

Choose a few observable release checks

Check a public page, one authenticated workflow, and an operation that touches a dependency such as the database. Compare latency and error rates against the pre-release baseline. Give the person releasing a clear threshold for stopping, and write down who will decide whether to roll back. Rehearse this with a staging release so the first attempt happens without a live incident.