Change Freeze Automation: Enforcing Deploy Windows Without a Platform Team - SysRoot

Change Freeze Automation: Enforcing Deploy Windows Without a Platform Team

Every small engineering team eventually has the same painful incident: someone ships a "tiny" change at 17:45 on a Friday, or during your biggest sales event, and now you're debugging production over the weekend. The organizational fix is a change freeze. The naive implementation is a Slack message that says "please don't deploy." That message has a 100% failure rate under pressure.

This article is about making freezes mechanical — enforced by the pipeline, the cluster, and the migration tooling itself — with a clean, auditable break-glass path for genuine emergencies. It's the kind of guardrail we build for clients who don't have a dedicated platform team to babysit release discipline.

Why "please don't deploy" fails

A freeze policy has three enforcement points, and skipping any one of them leaves a hole:

1. CI/CD — the most common deploy path. Blocking here stops 90% of changes.
2. Kubernetes admission — stops out-of-band kubectl apply, Argo CD auto-sync, and Helm upgrades that bypass CI.
3. Database migrations — schema changes cause the ugliest freeze-window incidents because they're hard to roll back.

Enforce all three or don't bother. A freeze that only lives in CI is trivially defeated by kubectl set image.

                      ┌──────────────────────────┐
   git push / MR ───► │  CI/CD (GitLab / Actions) │──┐
                      │  freeze gate job          │  │ blocked
                      └──────────────────────────┘  │
                                                     ▼
   kubectl apply ───► ┌──────────────────────────┐  deny (webhook)
   argocd sync   ───► │  K8s ValidatingWebhook    │──────────────►
   helm upgrade  ───► │  (Kyverno / OPA)          │
                      └──────────────────────────┘
                                                     ▼
   migrate up    ───► ┌──────────────────────────┐  exit 1
                      │  migration wrapper checks │──────────────►
                      │  freeze API + advisory    │
                      └──────────────────────────┘

   Single source of truth: freeze windows as data (ConfigMap / API)

The key design decision: one source of truth. We store freeze windows as data — a ConfigMap in-cluster and a tiny HTTP endpoint for CI — and every enforcement point reads from it. No copy-pasted cron expressions drifting apart.

The source of truth

We represent freezes as a list of windows with timezone-aware start/end and an optional label. Recurring windows (every Friday evening) and one-off windows (Black Friday week) coexist.

# freeze-windows.yaml  — the canonical definition
apiVersion: v1
kind: ConfigMap
metadata:
  name: freeze-windows
  namespace: platform
data:
  windows.yaml: |
    timezone: Europe/London
    recurring:
      - name: weekend
        # Fri 17:00 -> Mon 08:00
        days: [friday, saturday, sunday]
        start: "17:00"
        end: "08:00"          # end applies to the "next business day" boundary
        crosses_midnight: true
      - name: nightly
        days: [monday, tuesday, wednesday, thursday]
        start: "22:00"
        end: "06:00"
        crosses_midnight: true
    fixed:
      - name: black-friday-2025
        from: "2025-11-27T00:00:00"
        to:   "2025-12-02T09:00:00"
    # teams/paths always allowed (health, hotfix pipelines)
    allow_labels:
      - freeze/break-glass

Concrete numbers we default to: weekend freeze Fri 17:00 → Mon 08:00, nightly freeze 22:00 → 06:00 for non-24/7 teams. Adjust for your on-call coverage — a freeze window should never exceed the time you can respond to a break-glass request (target: acknowledge break-glass within 15 minutes).

Enforcement point 1: the CI/CD gate

In GitLab CI, we add a gate job that every deploy job depends on via needs. It calls a small freeze-check service (or evaluates the ConfigMap directly from a vendored copy). The job fails closed — if the freeze service is unreachable, it blocks, because a broken checker during a freeze is safer than an accidental deploy.

# .gitlab-ci.yml  (excerpt)
stages: [test, freeze-gate, deploy]

freeze:check:
  stage: freeze-gate
  image: curlimages/curl:8.10.1
  script:
    - |
      # Break-glass override: commit message or MR label
      if echo "$CI_COMMIT_MESSAGE" | grep -qi '\[break-glass\]'; then
        echo "Break-glass requested. Verifying approval..."
        # Require a signed approval env set by a protected variable / approval job
        if [ "$BREAK_GLASS_APPROVED" != "true" ]; then
          echo "ERROR: [break-glass] used but not approved. See runbook."; exit 1
        fi
        echo "Break-glass approved by: $BREAK_GLASS_APPROVER"; exit 0
      fi
    - |
      STATUS=$(curl -sf --max-time 5 \
        "https://freeze.internal.sysroot.dev/api/v1/status" \
        -H "Authorization: Bearer $FREEZE_TOKEN") || {
          echo "Freeze service unreachable — failing closed."; exit 1; }
      FROZEN=$(echo "$STATUS" | grep -o '"frozen":[a-z]*' | cut -d: -f2)
      WINDOW=$(echo "$STATUS" | grep -o '"window":"[^"]*"' | cut -d'"' -f4)
      if [ "$FROZEN" = "true" ]; then
        echo "DEPLOY BLOCKED — active freeze window: $WINDOW"
        echo "To deploy anyway, follow the break-glass runbook."
        exit 1
      fi
      echo "No active freeze. Proceeding."

deploy:prod:
  stage: deploy
  needs: ["freeze:check"]
  environment: production
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - helm upgrade --install app ./chart -n prod --wait --timeout 5m

The GitHub Actions equivalent uses a reusable workflow, but the logic is identical: a job that resolves freeze status and exits non-zero, gating the deploy job with needs:.

Enforcement point 2: Kubernetes admission with Kyverno

CI gates are bypassable by anyone with cluster credentials. To catch kubectl apply, Argo CD auto-sync, and helm upgrade run locally, we enforce at admission. Kyverno is lighter to run than an OPA/Gatekeeper stack for a small team, and its policy language is readable.

The trick is that Kyverno policies can't easily do timezone math, so we invert the model: a small CronJob patches a single boolean into a ConfigMap (freeze/active: "true|false"), and the policy just reads that boolean. The freeze logic stays in one place (the CronJob), and admission stays fast and deterministic.

# kyverno-freeze-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-writes-during-freeze
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: deny-prod-workload-changes
      match:
        any:
          - resources:
              kinds: [Deployment, StatefulSet, DaemonSet, Rollout]
              namespaces: ["prod", "prod-*"]
      # skip if the object carries the approved break-glass annotation
      exclude:
        any:
          - resources:
              annotations:
                freeze.sysroot.dev/break-glass: "approved"
      context:
        - name: freezeActive
          apiCall:
            urlPath: "/api/v1/namespaces/platform/configmaps/freeze-state"
            jmesPath: "data.active"
      preconditions:
        all:
          - key: "{{ request.operation }}"
            operator: AnyIn
            value: ["CREATE", "UPDATE"]
      validate:
        message: >-
          Change freeze is active. Prod workload changes are blocked.
          Set annotation freeze.sysroot.dev/break-glass=approved after
          following the break-glass runbook.
        deny:
          conditions:
            all:
              - key: "{{ freezeActive }}"
                operator: Equals
                value: "true"

# freeze-toggle-cronjob.yaml — the ONLY thing that computes freeze state
apiVersion: batch/v1
kind: CronJob
metadata:
  name: freeze-toggle
  namespace: platform
spec:
  schedule: "*/5 * * * *"          # re-evaluate every 5 minutes
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: freeze-toggler
          restartPolicy: Never
          containers:
            - name: toggle
              image: ghcr.io/sysroot/freeze-toggle:1.4.0
              args: ["--windows=/config/windows.yaml",
                     "--target-cm=freeze-state",
                     "--namespace=platform"]
              volumeMounts:
                - { name: windows, mountPath: /config }
          volumes:
            - name: windows
              configMap: { name: freeze-windows }

A 5-minute cadence means the worst-case delay between a window boundary and enforcement is 5 minutes. For weekend freezes that's irrelevant; if you need tighter, drop to */1. Don't go below 1 minute — you'll hammer the API server and the resolution isn't worth it.

Enforcement point 3: database migrations

Schema migrations during a freeze are the scenario that keeps us up at night, because a bad migration can't be rolled back with kubectl rollout undo. We wrap the migration command and add a Postgres advisory lock so two concurrent break-glass migrations can't collide.

#!/usr/bin/env bash
# migrate-guard.sh — wraps `migrate up` with freeze + advisory lock
set -euo pipefail

FREEZE_URL="https://freeze.internal.sysroot.dev/api/v1/status"
LOCK_KEY=48213   # arbitrary, stable per-database migration lock id

if [ "${BREAK_GLASS_APPROVED:-}" != "true" ]; then
  frozen=$(curl -sf --max-time 5 "$FREEZE_URL" | grep -o '"frozen":true' || true)
  if [ -n "$frozen" ]; then
    echo "Migrations blocked by active freeze. Break-glass required."; exit 1
  fi
fi

# Serialize migrations; fail fast rather than block a freeze deploy queue
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 <<SQL
SELECT CASE WHEN pg_try_advisory_lock(${LOCK_KEY})
            THEN 'locked' ELSE (SELECT pg_sleep(0), 'busy') END;
SQL

# run migrations, then release
migrate -path ./migrations -database "$DATABASE_URL" up
psql "$DATABASE_URL" -c "SELECT pg_advisory_unlock(${LOCK_KEY});"

We also block DDL at the database level for high-risk tables during freezes using an event trigger, so even a hand-run psql inside prod is caught:

-- ddl_freeze_guard.sql
CREATE OR REPLACE FUNCTION block_ddl_during_freeze()
RETURNS event_trigger AS $$
BEGIN
  IF current_setting('app.freeze_active', true) = 'on'
     AND current_setting('app.break_glass', true) IS DISTINCT FROM 'on' THEN
    RAISE EXCEPTION 'DDL blocked: change freeze active (set app.break_glass to override)';
  END IF;
END;
$$ LANGUAGE plpgsql;

CREATE EVENT TRIGGER ddl_freeze ON ddl_command_start
  WHEN TAG IN ('ALTER TABLE','DROP TABLE','CREATE INDEX','DROP INDEX')
  EXECUTE FUNCTION block_ddl_during_freeze();

The freeze-toggle CronJob also runs ALTER DATABASE app SET app.freeze_active = 'on' when a window opens, keeping the DB in sync with the same source of truth. RPO/RTO context: our target is RPO ≤ 5 min (WAL archiving) and RTO ≤ 30 min on a promotion — a botched freeze-window migration blows straight through both, which is exactly why we gate it.

Observability: prove the freeze is working

You need a Grafana panel that shows freeze state and blocked-attempt counts, or nobody trusts it. Expose two metrics from the freeze-toggle job and the Kyverno metrics endpoint.

{
  "title": "Change Freeze — blocked deploy attempts (1h)",
  "type": "timeseries",
  "targets": [
    {
      "expr": "sum(increase(kyverno_policy_results_total{policy=\"block-writes-during-freeze\",rule_result=\"fail\"}[1h]))",
      "legendFormat": "k8s admission blocks"
    },
    {
      "expr": "max(freeze_window_active)",
      "legendFormat": "freeze active (0/1)"
    }
  ]
}

And an alert that fires if the freeze-toggle job stops running (a dead toggler means stale enforcement):

# prometheus-rule.yaml
groups:
  - name: freeze
    rules:
      - alert: FreezeToggleStale
        expr: time() - freeze_state_last_updated_seconds > 900
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "Freeze state not updated in 15m — enforcement may be stale"

Wire Wazuh to the audit log so security review can see every break-glass event. Kubernetes API audit logs to a file Wazuh reads:

<!-- /var/ossec/etc/rules/local_rules.xml -->
<group name="k8s,freeze,">
  <rule id="100710" level="10">
    <decoded_as>json</decoded_as>
    <field name="objectRef.annotations">break-glass</field>
    <description>K8s break-glass annotation used during change freeze</description>
  </rule>
</group>

Break-glass runbook

Emergencies are real. The path must be fast, logged, and require a second human. Target: from decision to deploy in under 10 minutes.

1. Declare the incident. Open an incident channel. State what's broken and why the freeze must be broken. No incident record → no break-glass.
2. Get a second approver. One person requests, a different on-call engineer approves. This is the only human gate; keep it lightweight but non-optional.
3. Set the approval variable in CI (protected, expires):

   # requires maintainer role; scope to the single pipeline
   glab variable set BREAK_GLASS_APPROVED "true" --scope production --masked
   glab variable set BREAK_GLASS_APPROVER "[email protected]"
   git commit --allow-empty -m "hotfix: restart payment worker [break-glass]"
   git push
   

4. For direct kubectl (CI down): annotate the specific object so Kyverno lets it through:

   kubectl -n prod annotate deployment/payment-worker \
     freeze.sysroot.dev/break-glass=approved --overwrite
   kubectl -n prod set image deployment/payment-worker app=registry/app:hotfix-1
   

5. For a schema migration: set the session override, then migrate:

   BREAK_GLASS_APPROVED=true ./migrate-guard.sh
   # or manual DDL:
   psql "$DATABASE_URL" -c "SET app.break_glass = 'on';" -f 020_hotfix.sql
   

6. Verify the deploy: kubectl -n prod rollout status deploy/payment-worker --timeout=120s.
7. Revoke immediately. Delete the CI variable, remove the annotation:

   glab variable delete BREAK_GLASS_APPROVED --scope production
   kubectl -n prod annotate deployment/payment-worker \
     freeze.sysroot.dev/break-glass- --overwrite
   

8. Post-incident: confirm Wazuh logged rule 100710, attach the audit trail to the incident, and add a follow-up task if the hotfix needs cleanup after the freeze.

Failure modes & how we'd debug it

1. Deploys blocked outside a freeze window (false positive)

Symptom: freeze:check fails at 14:00 Tuesday. First, check what the toggler actually computed:

kubectl -n platform get cm freeze-state -o jsonpath='{.data.active}'; echo
kubectl -n platform logs job/$(kubectl -n platform get job \
  -l app=freeze-toggle --sort-by=.metadata.creationTimestamp \
  -o name | tail -1 | cut -d/ -f2)

The usual culprit is timezone: the container runs UTC, the window says Europe/London, and during BST you're off by an hour. Confirm:

kubectl -n platform exec deploy/freeze-toggle -- date -u

If the CronJob hasn't run recently, check the FreezeToggleStale alert and:

kubectl -n platform get cronjob freeze-toggle -o wide   # LAST SCHEDULE column
kubectl -n platform describe cronjob freeze-toggle | grep -A3 Events

concurrencyPolicy: Forbid plus a hung previous job will silently skip schedules — kill the stuck job.

2. Change slipped through during a freeze (false negative)

Someone deployed at 20:00 Saturday. Query the K8s audit log for who bypassed and how:

grep -E '"namespace":"prod"' /var/log/kubernetes/audit.log \
  | jq 'select(.verb=="update" or .verb=="patch")
        | {t:.requestReceivedTimestamp, user:.user.username,
           obj:.objectRef.name, ua:.userAgent}'

Check whether Kyverno was even in the request path — a webhook failurePolicy: Ignore fails open:

kubectl get validatingwebhookconfiguration kyverno-policy-validating-webhook-cfg \
  -o jsonpath='{.webhooks[*].failurePolicy}'; echo
kubectl -n kyverno logs deploy/kyverno-admission-controller | grep -i freeze

If it's Ignore, that's your hole for prod policies — but flipping to Fail risks blocking all writes if Kyverno is down, so scope Fail narrowly to prod namespaces and keep Kyverno on ≥2 replicas with a PodDisruptionBudget.

3. Freeze service down, CI failing closed everywhere

Symptom: every pipeline blocks, no freeze is scheduled. This is fail-closed working as designed, but it's now an outage of its own. Confirm the service and its dependency:

kubectl -n platform get pods -l app=freeze-api
kubectl -n platform logs deploy/freeze-api --tail=50
curl -sf https://freeze.internal.sysroot.dev/api/v1/status | jq .

Because the CronJob still writes freeze-state independently, admission control is unaffected — only CI gating breaks. Short-term mitigation is the break-glass variable; the real fix is running the freeze-API with 2 replicas and a cached last-known-good response (serve stale for ≤10 min rather than error).

4. Migration wedged behind the advisory lock

A break-glass migration hangs. Find who holds the lock:

SELECT pid, mode, granted, classid, objid
FROM pg_locks WHERE locktype = 'advisory';

SELECT pid, state, wait_event_type, query, now()-query_start AS age
FROM pg_stat_activity WHERE pid IN (SELECT pid FROM pg_locks WHERE locktype='advisory')
ORDER BY age DESC;

If a dead CI job left the lock held, terminate its backend: SELECT pg_terminate_backend(<pid>);. The advisory lock releases automatically when the session dies — the wrapper's explicit unlock is belt-and-suspenders.

Pre-flight checklist

Before you trust this in production:

• Freeze windows defined in one ConfigMap with an explicit timezone field.
• Toggle CronJob container timezone verified against the window timezone (test around a DST boundary).
• CI gate job fails closed on freeze-service errors, with --max-time set.
• Kyverno policy validationFailureAction: Enforce and scoped to prod namespaces only.
• Kyverno webhook failurePolicy reviewed; ≥2 replicas + PDB if using Fail.
• Migration wrapper enforces freeze and advisory lock; DDL event trigger installed.
• Break-glass requires a second approver and auto-expires the CI variable.
• Every break-glass path emits an audit event captured by Wazuh (rule 100710 firing verified).
• Grafana panel shows freeze state + blocked-attempt count; FreezeToggleStale alert wired.
• Break-glass runbook tested in a game-day, timed under 10 minutes.

The trade-off worth naming

Machine-enforced freezes trade a small amount of friction for a large reduction in weekend incidents. The failure mode you're most likely to hit is over-enforcement — blocking a legitimate emergency fix. That's why every layer here has an audited, fast break-glass path rather than a hard wall. A freeze that engineers can't override under pressure will get disabled entirely after the first bad night, and then you're back to "please don't deploy." Build the escape hatch first; enforce second. <<<END>>>

---

Struggling with Managed DevOps?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Managed DevOps, talk to the SysRoot team and we'll help you get it under control.