Secrets Sprawl: Rotating Credentials Without Taking Down Production - SysRoot

The problem isn't storing secrets. It's rotating them.

Most engineering teams reach a baseline of secrets hygiene fairly quickly. They move credentials out of source code, stop pasting them into Slack, and adopt a secrets manager like HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, or at minimum Kubernetes Secrets with encryption at rest. That part is well-documented and largely solved.

The part nobody solves is rotation.

Ask a small team when they last rotated their PostgreSQL application password, their Stripe API key, or the SSH key baked into their deployment tooling, and the honest answer is usually "never" or "once, during the incident." Secrets get created and then live forever, copied into a dozen places, referenced by services nobody fully remembers, and quietly accumulating risk. This is secrets sprawl: not the absence of a vault, but the inability to change a credential without fear.

The reason teams avoid rotation is simple. The last time someone tried, they took down production. A database password got rotated, half the pods picked it up and half didn't, connections started failing, and the rollback was messy. After that, rotation became a thing you don't touch.

This article is about doing it safely. Not the theory of secret storage, but the operational mechanics of changing a live credential while traffic is flowing.

Why rotation breaks things

Rotation is dangerous because a secret is almost never used in exactly one place. A single database password might be referenced by:

• A web application running across 12 pods
• A background worker fleet
• A nightly batch job
• A read-replica connection in a separate service
• A developer's local .env file that's been there for two years
• A monitoring agent that runs health-check queries
• Your migration tooling in CI

When you change that password at the source, every one of those consumers needs to pick up the new value. They don't do it simultaneously. Applications cache secrets in memory at boot. Connection pools hold open connections authenticated with the old credential. Some services reload config on a timer, some only on restart, some never.

The core failure mode is the flip-cut rotation: you change the credential in one atomic step, the old value stops working immediately, and anything still holding the old value breaks until it restarts. If your services don't all restart cleanly and quickly, you get a partial outage that's hard to diagnose because some requests succeed and some fail.

The two rotation strategies

There are fundamentally two ways to rotate any credential, and choosing the wrong one is the root cause of most rotation outages.

Single-credential rotation (flip-cut). One credential exists. You replace its value. The old value is invalidated. This is simple to reason about but unsafe in distributed systems because there is a window where consumers disagree about which value is valid.

Dual-credential rotation (overlap). Two credentials are valid simultaneously for a transition window. You introduce the new one, migrate consumers over, then revoke the old one. This is the only approach that gives you zero-downtime rotation, but it requires that the underlying system supports more than one valid credential at a time.

Whether you can use overlap depends entirely on the credential type. Most modern systems support it; some legacy ones make it painful.

Designing for overlap by credential type

Database passwords

PostgreSQL does not let one role have two passwords at once, which trips people up. The clean pattern is to use two roles rather than two passwords on one role.

Create app_user_a and app_user_b, both with identical privileges via a shared group role. Your application points at whichever is currently active. To rotate, you switch the application's secret to point at the inactive role (after setting a fresh password on it), let all consumers migrate, then rotate the password on the now-idle role so it's ready for next time.

If managing two roles is too much overhead, Vault's dynamic database secrets solve this differently: Vault generates short-lived database credentials on demand, so rotation is continuous and automatic. Each application instance gets its own credential with a TTL, and Vault revokes them on expiry. This eliminates long-lived database passwords entirely, but it requires every consumer to be Vault-aware, which is a real adoption cost for a small team.

API keys for third-party services

Most serious providers (Stripe, AWS, SendGrid, Datadog) support multiple active keys. This makes overlap trivial: create a second key, deploy it, confirm traffic is flowing on the new key via the provider's dashboard or logs, then revoke the old one. Always verify the new key is actually being used before revoking the old one. "It deployed" is not the same as "it's in use."

The failure case is a provider that only allows one key. For those, you genuinely cannot avoid a brief window. The mitigation is to make the cutover fast and scheduled, not reactive, and to do it during low traffic.

SSH keys and signing keys

SSH supports multiple authorized keys natively. Add the new public key to authorized_keys, deploy the new private key to whatever needs it, confirm access works, then remove the old public key. There is no reason to ever have a hard SSH cutover.

TLS certificates

Certificates rotate cleanly if you automate them. Use cert-manager in Kubernetes or ACME-based tooling and renew well before expiry. The classic outage here is the certificate that expires on a holiday because renewal was manual and the calendar reminder got ignored. Automate it, and alert when a cert is within 30 days of expiry as a backstop.

How consumers actually pick up new secrets

Even with overlap, you have to get the new value into running services. This is where Kubernetes-specific traps appear.

A Kubernetes Secret mounted as a file gets updated automatically over time, but with a delay (kubelet sync period, often up to a minute or more), and only if your application re-reads the file. Most applications read config once at startup and never again. Mounting the updated secret does nothing for a process that cached it.

A Secret consumed as an environment variable is worse: env vars are set at container start and never update while the pod is running. If your password comes from an env block, the only way to pick up a new value is a pod restart.

This leads to a practical rule: decide your reload mechanism before you rotate. Your options are:

1. Rolling restart on rotation. Update the secret, then trigger a rolling restart of the deployment so every pod re-reads it. Simple and reliable. Combined with overlap (old credential still valid), this is zero-downtime because pods roll one at a time while the old value still works.
2. In-app secret reloading. The application watches the secrets file or polls the secrets manager and reconnects when the value changes. More work to build, but no restart needed. Tools like Reloader can watch a Secret and trigger restarts automatically, bridging the gap.
3. Sidecar injection. Vault Agent or similar injects and refreshes secrets, and the app reads from a local file. Powerful but adds operational surface area.

For most small teams, overlap plus rolling restart is the right default. It's boring, it works, and it doesn't require rewriting application config handling.

A safe rotation runbook

Here is a concrete, ordered procedure for rotating a database password with zero downtime using the two-role pattern.

1. Confirm the inactive role exists and has correct privileges. Say the app currently uses app_user_a. Verify app_user_b exists with identical grants.
2. Set a fresh strong password on app_user_b. Do this directly on the database. No consumer uses this role yet, so nothing is affected.
3. Update the secret in your secrets manager to point at app_user_b with the new password. Don't deploy yet.
4. Trigger a rolling restart of the consuming deployments. Pods restart one at a time. During this window both roles are valid, so any pod still on app_user_a continues to work while new pods come up on app_user_b.
5. Verify the cutover. Check active connections on the database (SELECT usename, count(*) FROM pg_stat_activity GROUP BY usename;). Confirm all connections are now on app_user_b and none remain on app_user_a.
6. Rotate the password on app_user_a to a new random value. It's now idle and primed for the next rotation. Do not delete the role; you'll alternate back to it next time.
7. Audit the trailing references. Search CI variables, local .env templates, infra-as-code, and any other consumer for the old password. This is where sprawl hides.

The whole sequence has no moment where a valid credential is unavailable. That's the point.

A realistic failure scenario

A SaaS team gets a security finding: their main database password is six years old and stored in three places. They decide to rotate it on a Friday afternoon. They change the password in AWS Secrets Manager and update the Kubernetes Secret. Nothing happens, because the password is injected as an environment variable and no pods restart.

Thinking the change didn't apply, they manually delete and recreate a few pods. Those pods come up with the new password and connect fine. But the old password was already invalidated when they changed it on the database. The pods they didn't restart are still alive, holding pooled connections on the old credential. As those connections drop and try to re-authenticate, they fail. Background workers start erroring. The on-call engineer sees intermittent database auth failures across half the fleet and can't tell why some requests work.

The fix at 6pm is a full rolling restart of everything, which they hadn't planned for, during peak weekend signup traffic. Nothing was lost, but it was an hour of avoidable degradation.

Everything that went wrong was a process error, not a tooling error. They used a flip-cut on a system that needed overlap, they didn't understand how their consumers loaded the secret, and they rotated reactively instead of with a runbook.

Patching follows the same logic

The rotation discipline generalizes. Patching infrastructure (OS packages, base images, database engine versions) has the same overlap-vs-cutover trade-off. You don't patch a single node in place and hope; you roll new patched nodes alongside old ones, drain traffic gradually, and keep a rollback path. The mental model is identical: maintain a valid path through the change at every moment, never a flip-cut on a live system.

Checklist before you rotate anything

• Do you know every consumer of this secret? If not, find them first.
• Does the underlying system support two valid credentials at once? If yes, use overlap. If no, schedule a low-traffic cutover.
• How does each consumer load the secret: file, env var, or live fetch? This determines whether a restart is required.
• Do you have a way to verify the new credential is in use before revoking the old one?
• Do you have a rollback if the new credential is wrong?
• Is the rotation written as a runbook, or are you improvising?

Secrets sprawl doesn't get fixed by buying a vault. It gets fixed by making rotation a routine, low-drama operation your team can run on a normal Tuesday. Once rotation is safe, frequent rotation becomes possible, and frequent rotation is what actually reduces your exposure.