Deployment Rollbacks That Actually Work: Beyond kubectl rollout undo - SysRoot

Why your rollback strategy fails at the worst possible moment, and how to build one that survives schema changes, config drift, and stateful services.

The rollback that didn't roll back

At 02:14 a deploy went out, error rates spiked, and the on-call engineer did the thing everyone is trained to do: kubectl rollout undo. The pods flipped back to the previous image in 40 seconds. The errors did not stop. It took another 25 minutes to realise the failed deploy had also run a database migration that added a NOT NULL column, and the "rolled back" old code was now writing rows without that column and getting rejected by Postgres.

That is the core problem with how most small teams think about rollbacks. kubectl rollout undo reverts one thing — the pod template of one Deployment. A production release is almost never one thing. It's an image, a ConfigMap, a set of feature flags, a Helm values change, and increasingly a schema migration. Rolling back the container while leaving the other four in their new state is not a rollback; it's a new, untested, hybrid state that nobody has ever run.

This article is about building rollbacks that actually restore a known-good state, including the hard parts nobody wants to touch: database migrations and config. It's written from the perspective of running this for teams who don't have a dedicated platform group — so it has to be operable at 02:00 by whoever is on call, not just by the person who built it.

What "rollback" has to actually cover

A deploy changes state in several places. A real rollback strategy has to have an answer for each one:

                 DEPLOY unit (must roll back together or not at all)
   ┌─────────────────────────────────────────────────────────────┐
   │                                                               │
   │   [container image]   [ConfigMap/Secret]   [Helm values]      │
   │          │                    │                  │            │
   │          └──────────┬─────────┴──────────────────┘            │
   │                     ▼                                         │
   │              Kubernetes Deployment  ─────►  running Pods      │
   │                     │                                         │
   │                     ▼                                         │
   │           DB schema migration (Postgres)  ◄── the hard part   │
   │                     │                                         │
   └─────────────────────┼─────────────────────────────────────────┘
                         ▼
             app writes/reads must be valid against
             BOTH old and new schema during the window

The container image is the easy 90%. Config and schema are the 10% that turns a 40-second recovery into a 30-minute incident. The two rules that make everything else tractable:

Schema changes must be backward-compatible for at least one release. The old code must be able to run against the new schema. If that isn't true, you cannot safely roll back code without also reversing the migration — and reversing migrations under load is where data loss lives.
Config and image must be version-pinned together. If you roll back the image to v1.4.2 you need the exact ConfigMap that shipped with v1.4.2, not "whatever is in the cluster now."

Making config roll back with the image

The most common silent failure: someone edits a live ConfigMap with kubectl edit, then a rollback flips the image back but the ConfigMap is now three edits ahead. Fix this by treating config as immutable and versioned, and referencing it by hash.

# configmap generated per release, name includes content hash
apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config-8f3a1c   # suffix = sha of contents, immutable
  namespace: prod
immutable: true
data:
  LOG_LEVEL: "info"
  FEATURE_NEW_BILLING: "false"
  DB_STATEMENT_TIMEOUT: "5000"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: prod
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0        # never drop below desired during rollout
      maxSurge: 2
  template:
    metadata:
      annotations:
        # forces pod restart when config changes; ties image<->config
        configHash: "8f3a1c"
    spec:
      containers:
        - name: api
          image: registry.internal/api:v1.5.0
          envFrom:
            - configMapRef:
                name: api-config-8f3a1c
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 3
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { memory: "512Mi" }

Because the ConfigMap name is content-addressed and immutable: true, nobody can drift it in place. Rolling back the Deployment to a previous ReplicaSet automatically pulls the correct api-config-8f3a1c, because the pod template referenced that exact name. Config and image roll back as one unit.

Set maxUnavailable: 0. During a rollback you want zero capacity loss — you're already in an incident.

The migration discipline that makes rollback safe

The rule is expand / migrate / contract, and rollbacks only work if every release is in the "expand" or "contract" phase, never the destructive part while old code might still run.

Bad (breaks rollback):

-- release v1.5.0 - CANNOT roll back to v1.4 code
ALTER TABLE invoices ADD COLUMN currency text NOT NULL;

Good (rollback-safe expand):

-- release v1.5.0: EXPAND. Old code ignores the column, new code fills it.
ALTER TABLE invoices ADD COLUMN currency text;          -- nullable
UPDATE invoices SET currency = 'USD' WHERE currency IS NULL;  -- backfill in batches, see runbook
-- NOT NULL constraint deferred to a LATER release (contract phase)

The NOT NULL constraint and any column drops only land in a later release, once you're confident the previous version will never be rolled back to. This means at any given moment, the release you'd roll back to can read and write the current schema. That single discipline turns database rollback from "restore from backup and lose 25 minutes of writes" into "just roll back the code."

For the contract phase, add the constraint with NOT VALID first so it doesn't take a full-table lock:

-- release v1.6.0: CONTRACT, only after v1.5 is proven stable
ALTER TABLE invoices ADD CONSTRAINT invoices_currency_nn
  CHECK (currency IS NOT NULL) NOT VALID;   -- no full scan, no lock
ALTER TABLE invoices VALIDATE CONSTRAINT invoices_currency_nn;  -- scans, but SHARE UPDATE EXCLUSIVE only

Wiring the safety into CI/CD

The pipeline should refuse to deploy a migration that isn't backward-compatible, and should record exactly what shipped so rollback is deterministic.

# .gitlab-ci.yml (excerpt)
stages: [validate, deploy, verify]

migration-safety:
  stage: validate
  image: ghcr.io/sqlfluff/sqlfluff:2.3.5
  script:
    # fail the build if a migration contains destructive DDL in same release
    - |
      if grep -rEi 'DROP COLUMN|DROP TABLE|ALTER COLUMN .* SET NOT NULL' migrations/pending/; then
        echo "Destructive DDL in pending migrations — split into contract release"; exit 1
      fi

deploy-prod:
  stage: deploy
  environment: production
  script:
    - RELEASE="v${CI_COMMIT_SHORT_SHA}"
    # record the currently-live revision BEFORE we change anything
    - kubectl -n prod rollout history deployment/api | tail -5 > .prev-rollout.txt
    - helm upgrade api ./chart -n prod \
        --set image.tag="$RELEASE" \
        --atomic --timeout 5m           # --atomic auto-rolls-back on failed release
    # tag the git commit that is now live, so rollback target is unambiguous
    - git tag "prod-live-$RELEASE" && git push origin "prod-live-$RELEASE"
  artifacts:
    paths: [.prev-rollout.txt]
    expire_in: 30 days

verify-prod:
  stage: verify
  script:
    - ./scripts/smoke.sh https://api.internal   # fails pipeline -> --atomic already reverted

helm upgrade --atomic --timeout 5m is doing real work here: if the new pods never reach ready within 5 minutes, Helm automatically rolls the release back to the last successful revision. That handles the "deploy never came up" case without a human. The runbook below is for the harder case: the deploy came up healthy but is behaving badly in production.

RUNBOOK: rolling back a bad-but-healthy release

Use this when a deploy is up, passing readiness, but causing elevated errors / latency / bad behaviour. Target: decision within 5 minutes, recovery within 10.

Confirm it's the deploy, not something else. Check the error-rate panel against the deploy timestamp:
```
sum(rate(http_requests_total{namespace="prod",code=~"5.."}[2m]))
  / sum(rate(http_requests_total{namespace="prod"}[2m]))
```
If the spike aligns with the rollout timestamp, proceed. If not, stop — rolling back won't help.

Identify the exact previous good revision:

kubectl -n prod rollout history deployment/api
# note the revision number of the last-known-good, e.g. 41
kubectl -n prod rollout history deployment/api --revision=41   # verify image tag

Check for schema entanglement BEFORE rolling back. Did this release ship a migration?
```
git log --oneline prod-live-v1.4.2..prod-live-v1.5.0 -- migrations/
```
If migrations exist and they were expand-only (nullable columns, new tables) → safe to roll back code alone, go to step 4. If they contain destructive DDL → do not roll back code; go to step 6 (forward-fix).
Roll back code + config as one unit:
```
kubectl -n prod rollout undo deployment/api --to-revision=41
kubectl -n prod rollout status deployment/api --timeout=120s
```
Because config is content-addressed in the pod template, revision 41 pulls its original ConfigMap automatically.

Verify recovery on the same PromQL panel from step 1. Error ratio should return under 0.5% within 2 minutes. Then check Postgres isn't rejecting writes from the reverted code:

SELECT state, count(*), max(now()-query_start) AS longest
FROM pg_stat_activity WHERE datname='app' GROUP BY state;
-- look for a pile of 'idle in transaction' or errors in the log (step 6)

If rollback is unsafe (destructive migration) — forward-fix instead. Disable the offending path via feature flag (config is immutable, so ship a new config revision, don't edit live):
```
helm upgrade api ./chart -n prod \
  --set config.FEATURE_NEW_BILLING=false \
  --set image.tag=v1.5.0 --atomic --timeout 5m
```
This keeps the new schema (which the new code needs) while turning off the broken behaviour.
Announce state. Post the revision now live, whether schema changed, and whether a forward-fix is pending. Freeze deploys until root cause is known.

Failure modes & how we'd debug it

1. Rollback "succeeded" but errors continue

Symptom: rollout status reports success, pods are ready, 5xx rate stays high.

Most likely: schema entanglement. The old code can't run against the migrated schema.

# is Postgres rejecting the old code's queries?
kubectl -n prod logs deploy/postgres --tail=100 | grep -Ei 'ERROR|null value|does not exist'
# on the DB directly:
psql -c "SELECT query, count(*) FROM pg_stat_activity
         WHERE state='active' AND query ILIKE '%invoices%' GROUP BY query;"

If you see null value in column "currency" violates not-null constraint, the migration was not expand-safe. You cannot fix this by rolling back code. Go to forward-fix (runbook step 6) and, if necessary, drop the offending constraint:

ALTER TABLE invoices DROP CONSTRAINT invoices_currency_nn;  -- restores old-code compatibility

2. Old ReplicaSet is gone — nothing to roll back to

Symptom: kubectl rollout undo returns no rollout history found or only one revision.

Cause: revisionHistoryLimit too low (default cleanup) or a kubectl delete rs cleanup script.

kubectl -n prod get rs -l app=api --sort-by=.metadata.creationTimestamp
kubectl -n prod get deploy api -o jsonpath='{.spec.revisionHistoryLimit}'

Fix going forward: set revisionHistoryLimit: 10 in the Deployment spec. For the incident right now, redeploy the known-good tag from git explicitly:

helm upgrade api ./chart -n prod --set image.tag=v1.4.2 --atomic --timeout 5m

This is why the pipeline tags prod-live-<sha> — the last-good artifact is always reconstructable from git even if the cluster forgot it.

3. Rollback triggers a thundering herd on Postgres

Symptom: after rollback, DB CPU pegs, connection count hits the ceiling, latency worse than before.

Cause: all pods restart at once, each opening fresh connections; caches are cold.

# Prometheus: connection saturation
sum(pg_stat_activity_count{datname="app"})
  / on() pg_settings_max_connections

If that ratio approaches 1.0, you're out of connections. Inspect:

SELECT usename, state, count(*) FROM pg_stat_activity
GROUP BY usename, state ORDER BY count DESC;

Mitigation: this is exactly why a pooler sits in front. With PgBouncer in transaction mode the app's connection storm collapses to a small server-side pool. During the incident, throttle the rollout so pods restart gradually — but note rollout undo restarts by the deployment's own strategy, so maxSurge: 2 already staggers it. If it's still too aggressive, temporarily scale down and back up in steps.

4. A rollback that shouldn't have happened (unauthorized change)

Symptom: production reverted and nobody on the team ran it.

This is a security event, not just an ops one. Wazuh should be auditing kubectl/API activity. A minimal rule to flag production Deployment changes from the audit log:

<!-- /var/ossec/etc/rules/local_rules.xml -->
<group name="k8s,audit,">
  <rule id="100210" level="10">
    <decoded_as>json</decoded_as>
    <field name="objectRef.namespace">prod</field>
    <field name="verb">update|patch|delete</field>
    <field name="objectRef.resource">deployments</field>
    <description>Production Deployment modified: $(user.username) $(verb) $(objectRef.name)</description>
    <options>no_full_log</options>
  </rule>
</group>

Point Wazuh at the API server audit log so this fires:

<!-- ossec.conf on the node reading the audit log -->
<localfile>
  <log_format>json</log_format>
  <location>/var/log/kubernetes/audit/audit.log</location>
</localfile>

Now every rollback is attributable. Cross-reference rule 100210 against your change log; an entry with no matching pipeline run is a real finding.

Pre-flight & hardening checklist

Run this before you rely on rollback as your recovery plan — not during the incident.

Numbers we hold teams to

Rollback decision time: ≤ 5 minutes from alert to decision. That requires the error-rate panel and deploy annotations to be one click away.
Code-only rollback recovery: ≤ 2 minutes to healthy after rollout undo completes, given maxUnavailable: 0 and a warm pool.
RPO for a safe (expand-only) rollback: zero — no data is discarded because the old code is compatible with the migrated schema.
RTO for the destructive-migration case: this is why we forbid it in the same release. If it happens anyway, RTO is bounded by restore-from-backup time, typically 20–40 minutes for a mid-sized DB — an order of magnitude worse. The whole discipline exists to never be in that branch.

The takeaway

kubectl rollout undo is a button, not a strategy. A rollback that works under pressure is a property of how you ship: immutable content-addressed config so image and config revert together, expand/migrate/contract migrations so old code always runs against the current schema, --atomic deploys that self-revert on failure, and git-tagged releases so the good state is reconstructable even when the cluster's memory is short. Get those four things right and the 02:14 incident becomes a 40-second non-event instead of a 30-minute investigation into why the rollback didn't roll back. <<<END>>>

Struggling with Managed DevOps?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Managed DevOps, talk to the SysRoot team and we'll help you get it under control.

Back to Blog