Image Promotion, Not Rebuild: Shipping the Exact Bytes You Tested - SysRoot

How a small team can guarantee prod runs the identical artifact that passed staging — with digest pinning, provenance, and automated gates.

Image Promotion, Not Rebuild: Shipping the Exact Bytes You Tested

Most small teams have a subtle, expensive bug baked into their delivery pipeline: they rebuild the container image for every environment. Build for dev, run tests, then a separate job builds again for staging, and yet another builds for prod. Each build pulls fresh base layers, re-resolves latest transitive deps, and stamps a new timestamp. By the time bytes reach production, they are not the bytes you tested.

This article is about the fix: build once, then promote the same immutable artifact by digest through environments. No rebuilds, no :latest, no "it passed in staging" surprises. You don't need a platform team to run this — you need a registry, a CI system, and discipline enforced in code.

Why rebuild-per-environment quietly breaks

A Dockerfile that says FROM python:3.12-slim resolves to a tag, and tags move. Between your staging build at 14:02 and your prod build at 16:40, upstream can publish a new 3.12-slim with a patched glibc, a different OpenSSL, or a rebuilt CA bundle. Your tests never saw it.

The failure signature is nasty because it's non-deterministic:

Staging: green. Prod: SSL: CERTIFICATE_VERIFY_FAILED against an internal service.
Staging: 120 MB image. Prod: 138 MB, OOMKilled under the same limits.
A CVE scan that passed on Tuesday's build fails on Friday's identical source.

The root cause is always the same: the tag is a pointer, not an identity. The only stable identity is the content digest — sha256:.... Promotion means carrying that digest forward untouched.

        ┌──────────┐     build once      ┌───────────────────────────┐
 git ──▶│  CI build │────────────────────▶│ registry.sysroot.io/app   │
 tag    └──────────┘                      │  @sha256:9f3c… (immutable) │
                                          └───────────────────────────┘
                                                     │
              promote by DIGEST (no rebuild)         │
        ┌────────────────────┬────────────────────┬─┘
        ▼                    ▼                    ▼
   ┌─────────┐          ┌─────────┐          ┌─────────┐
   │  dev    │  gate ──▶│ staging │  gate ──▶│  prod   │
   │ deploy  │  tests   │ deploy  │  soak+   │ deploy  │
   └─────────┘          └─────────┘  scan    └─────────┘
   same sha256          same sha256          same sha256

The digest is the invariant across the whole diagram. Everything downstream only ever references it.

The build: produce a digest and provenance, then stop

Build once, push once, and capture the digest as a first-class output. Here's a GitHub Actions job using Buildx with reproducibility flags and SBOM/provenance attestations.

# .github/workflows/build.yml
name: build
on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write   # for keyless cosign signing

jobs:
  build:
    runs-on: ubuntu-24.04
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - uses: docker/login-action@v3
        with:
          registry: registry.sysroot.io
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}

      - name: Build & push (immutable, no floating tag as the source of truth)
        id: push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          # SOURCE_DATE_EPOCH pins timestamps for reproducibility
          build-args: |
            SOURCE_DATE_EPOCH=${{ github.event.head_commit.timestamp && 0 }}
          # tag by commit SHA only. NEVER :latest as promotion input.
          tags: registry.sysroot.io/app:${{ github.sha }}
          provenance: mode=max
          sbom: true

      - name: Sign the digest (keyless)
        env:
          COSIGN_EXPERIMENTAL: "1"
        run: |
          cosign sign --yes \
            registry.sysroot.io/app@${{ steps.push.outputs.digest }}

      - name: Emit digest for downstream promotion
        run: echo "Built ${{ steps.push.outputs.digest }}"

Two things matter here. First, steps.push.outputs.digest gives you sha256:... — the identity you will promote. Second, your Dockerfile base image should itself be digest-pinned so even the build is deterministic:

# Dockerfile — pin the base by digest, not tag
FROM python:3.12-slim@sha256:2f3c7...e91 AS base
# Renovate/Dependabot updates the digest via PR, on your schedule — not upstream's.

Promotion, not rebuild

Promotion is a metadata operation. You never touch the layers; you copy the digest reference into an environment-specific tag (or, better, you update a GitOps manifest to point at the digest). crane copy copies by digest without pulling+pushing layers you already have, and preserves the digest exactly.

# Promote the exact digest from staging identity to prod identity.
# No rebuild. crane copies the manifest by digest, byte-for-byte.
SRC="registry.sysroot.io/app@sha256:9f3c...b1"
crane copy "$SRC" "registry.sysroot.io/app:prod"

# Verify the destination tag now resolves to the SAME digest:
crane digest registry.sysroot.io/app:prod
# -> sha256:9f3c...b1   (must match SRC)

For Kubernetes we go one step further: deploy by digest, never by tag. Tags in a cluster are a re-pull lottery. The manifest references the immutable digest:

# k8s/prod/deployment.yaml — image referenced by digest
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: prod
spec:
  replicas: 3
  selector: { matchLabels: { app: app } }
  template:
    metadata:
      labels: { app: app }
    spec:
      containers:
        - name: app
          # digest, not tag — kubelet will never silently pull a different image
          image: registry.sysroot.io/app@sha256:9f3c...b1
          imagePullPolicy: IfNotPresent
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { cpu: "1",    memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 3

The promotion job's only job is to rewrite that digest string and let Argo/Flux reconcile it. With GitOps, promotion becomes a PR:

# .github/workflows/promote.yml (excerpt)
promote-to-prod:
  needs: [gate-staging]
  runs-on: ubuntu-24.04
  environment: production        # GitHub environment protection = manual approval gate
  steps:
    - uses: actions/checkout@v4
      with: { repository: sysroot/gitops-config, token: ${{ secrets.GITOPS_PAT }} }

    - name: Verify signature & provenance BEFORE promotion
      run: |
        cosign verify \
          --certificate-identity-regexp '.*' \
          --certificate-oidc-issuer https://token.actions.githubusercontent.com \
          registry.sysroot.io/app@${{ needs.gate-staging.outputs.digest }}

    - name: Rewrite prod manifest to the tested digest
      run: |
        DIGEST="${{ needs.gate-staging.outputs.digest }}"
        yq -i ".spec.template.spec.containers[0].image = \"registry.sysroot.io/app@${DIGEST}\"" \
          k8s/prod/deployment.yaml
        git config user.name  "sysroot-ci"
        git config user.email "[email protected]"
        git commit -am "promote app to ${DIGEST} to prod"
        git push

The environment: production gate gives you a human approval step in the GitHub UI with zero extra infrastructure. The cosign verify step is the hard gate: if the artifact isn't the signed, tested one, promotion fails.

Enforce it at admission — trust nothing by policy

Configuration can drift. Someone will eventually kubectl set image ... app=app:hotfix. Close that door with an admission policy that rejects tag-based images and unsigned digests in prod.

# kyverno ClusterPolicy: prod must use digests, and they must be signed
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: prod-image-integrity
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-digest
      match:
        any:
          - resources: { kinds: [Pod], namespaces: [prod] }
      validate:
        message: "prod images must be referenced by digest (sha256), not tag"
        pattern:
          spec:
            containers:
              - image: "registry.sysroot.io/*@sha256:*"
    - name: require-signature
      match:
        any:
          - resources: { kinds: [Pod], namespaces: [prod] }
      verifyImages:
        - imageReferences: ["registry.sysroot.io/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/sysroot/app/.github/workflows/build.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"

Now the cluster itself refuses anything that isn't the promoted, signed digest. That's the difference between "we have a policy" and "the system enforces the policy."

Wazuh: catch promotion bypasses and registry tampering

Admission control blocks bad deploys; Wazuh tells you when someone tries. Ship the registry audit log and Kubernetes audit log to Wazuh and add rules for two events: a tag being overwritten, and a Kyverno denial.

<!-- /var/ossec/etc/rules/local_rules.xml -->
<group name="cicd,supply_chain,">

  <!-- A tag that was supposed to be immutable got re-pushed -->
  <rule id="100610" level="12">
    <decoded_as>json</decoded_as>
    <field name="registry.action">push</field>
    <field name="registry.tag">^prod$|^staging$</field>
    <field name="registry.digest_changed">true</field>
    <description>Registry: immutable env tag $(registry.tag) repointed - possible promotion bypass</description>
    <group>authentication_failure,pci_dss_10.2.7,</group>
  </rule>

  <!-- Kyverno rejected a pod in prod for policy violation -->
  <rule id="100611" level="10">
    <decoded_as>json</decoded_as>
    <field name="responseObject.message">prod images must be referenced by digest</field>
    <description>Kyverno denied non-digest image in prod: $(objectRef.name)</description>
    <group>policy_violation,</group>
  </rule>

</group>

Level 12 pages; level 10 goes to the daily review. The point: an image-promotion pipeline is a supply-chain control, and supply-chain controls need monitoring, not just enforcement.

Runbook: promote a build from staging to production

Follow this under pressure. Assume the build already passed staging soak and you have the digest.

Confirm the digest you intend to ship.

DIGEST=$(crane digest registry.sysroot.io/app:staging)
echo "$DIGEST"   # sha256:9f3c...b1

Verify signature and provenance for that exact digest.

cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp 'sysroot/app/.github/workflows/build.yml' \
  "registry.sysroot.io/app@${DIGEST}"

If this fails, stop. You're about to ship an unverified artifact.

Confirm the currently-running prod digest (your rollback target).

kubectl -n prod get deploy app \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
# record this string — it's your rollback image

Open the promotion PR (updates the digest in GitOps).
```
gh workflow run promote.yml -f digest="$DIGEST"
```
Approve the production environment gate in the GitHub UI.

Watch the rollout with a hard timeout.

kubectl -n prod rollout status deploy/app --timeout=180s

Verify prod is running the promoted digest — not a tag, not a re-pull.

kubectl -n prod get pods -l app=app \
  -o jsonpath='{range .items[*]}{.status.containerStatuses[0].imageID}{"\n"}{end}'
# every line must contain the promoted sha256

Smoke test through the real ingress path.

curl -fsS https://app.sysroot.io/healthz && echo OK

If anything is red, roll back to the recorded digest (step 3):
```
kubectl -n prod set image deploy/app app=<recorded-digest-image>
kubectl -n prod rollout status deploy/app --timeout=120s
```
Then revert the GitOps commit so reconciliation doesn't re-apply the bad digest.

Target numbers we hold ourselves to: promotion-to-serving < 5 minutes, rollback < 2 minutes, and zero rebuilds between staging-pass and prod-serve.

Pre-flight checklist

Failure modes & how we'd debug it

1. "Prod is serving different behavior than staging" despite same tag

Symptom: staging green, prod misbehaving, both "on the same version tag." First, stop trusting the tag.

# Are they actually the same bytes?
crane digest registry.sysroot.io/app:staging
crane digest registry.sysroot.io/app:prod

If the digests differ, someone rebuilt instead of promoting. Confirm what's running vs what's declared:

kubectl -n prod get pods -l app=app \
  -o jsonpath='{.items[*].status.containerStatuses[0].imageID}'

imageID shows the resolved digest the kubelet actually pulled. If it doesn't match your intended digest, a floating tag got re-pulled on a node restart. Reasoning: imageID is ground truth; the image: spec field is only intent. Fix: switch the manifest to a digest reference and let Kyverno block the recurrence.

2. Kyverno blocks a legitimate deploy

Symptom: rollout stuck, pods never created.

kubectl -n prod describe replicaset -l app=app | grep -A5 Events
# look for: "admission webhook denied the request"

kubectl -n prod get events --sort-by=.lastTimestamp | tail -20

Cross-check the Kyverno policy report:

kubectl -n prod get policyreport -o wide
kubectl logs -n kyverno deploy/kyverno-admission-controller --tail=100 | grep app

Most common cause: the image is a digest but the signature check fails because the build workflow identity in the policy subject: no longer matches (e.g., branch renamed main→master, or workflow path changed). Reasoning: verifyImages matches on the OIDC subject string exactly. Fix the subject: regex, don't disable the policy.

3. Promotion "succeeded" but nodes still run the old image

Symptom: GitOps shows the new digest, rollout status returns success, but users report old behavior.

# Check per-pod resolved digest across ALL nodes
kubectl -n prod get pods -o wide -l app=app \
  -o custom-columns=NODE:.spec.nodeName,IMG:.status.containerStatuses[0].imageID

If one node shows the old digest, that node failed to pull the new one (registry auth, disk pressure). Inspect the kubelet:

NODE=<offending-node>
kubectl describe node "$NODE" | grep -A3 -i pressure
# then on the node (Proxmox VM host it, or via debug pod):
journalctl -u kubelet --since "10 min ago" | grep -i "pull\|image"

ImagePullBackOff with no space left on device points to disk pressure — evict, prune images (crictl rmi --prune), and re-check. Reasoning: promotion updates intent cluster-wide, but each kubelet pulls independently; one starved node breaks the invariant.

4. Registry tag was repointed out-of-band (Wazuh 100610 fired)

Symptom: pager fires on rule 100610 — the immutable prod tag's digest changed without a promotion PR.

# What does prod point at now vs the last promoted digest in git?
crane digest registry.sysroot.io/app:prod
git -C gitops-config log -1 --format='%H %s' -- k8s/prod/deployment.yaml

If the registry digest and the git-declared digest disagree, someone bypassed the pipeline. Immediately re-assert the declared digest and lock the tag:

crane copy "registry.sysroot.io/app@<git-declared-digest>" registry.sysroot.io/app:prod

Then audit who: correlate the Wazuh alert timestamp with registry access logs and rotate the credential used. Reasoning: your enforcement lives in the cluster and CI, but the registry is a shared write surface — treat an unexpected repoint as a credential compromise until proven otherwise.

The trade-offs, stated plainly

Storage: keeping every SHA-tagged build costs registry space. We keep prod-promoted digests for 90 days, all others 14 days, via a registry GC policy. Cost: a few GB; benefit: instant rollback to any prior prod digest.
Speed: digest verification (cosign verify) adds ~1–2s per promotion. Non-negotiable.
Rigidity: you can no longer "just repush a tag." That's the entire point — the friction is the safety.

For a team without a platform group, this pattern buys the single most valuable guarantee in delivery: what you tested is what you run, provably, every time. Build once, sign it, promote the digest, enforce it at admission, and watch the registry for anyone who tries to route around it. <<<END>>>

Struggling with Managed DevOps?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Managed DevOps, talk to the SysRoot team and we'll help you get it under control.

Back to Blog