PodDisruptionBudgets: Why Your Node Drains Stall or Take Down Production

The Two Ways Node Maintenance Goes Wrong

Every Kubernetes cluster eventually needs node maintenance: kernel patches, instance type changes, cluster upgrades, or autoscaler scale-downs. The mechanism behind all of these is the same: a node drain, which evicts pods so they can reschedule elsewhere.

Drains fail in two opposite ways. Either the drain blocks forever because a PodDisruptionBudget (PDB) refuses to allow any eviction, stalling your upgrade. Or there's no PDB at all, the drain rips out every replica of a service at once, and you take a hard outage during what should have been routine maintenance.

Both outcomes come from misunderstanding what a PDB actually controls.

What a PDB Actually Does

A PodDisruptionBudget governs voluntary disruptions only: evictions triggered by kubectl drain, the cluster autoscaler, or other controllers that call the eviction API. It does not protect against involuntary disruptions like a node crashing, an OOMKill, or hardware failure.

A PDB declares either minAvailable or maxUnavailable for a set of pods selected by a label query:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api

When something tries to evict a pod, the API server checks whether doing so would violate the budget. If it would, the eviction is rejected and the drain retries.

That retry loop is the source of most pain.

The Stalled Drain Problem

The most common failure: a single-replica deployment with a minAvailable: 1 PDB. The drain wants to evict the only pod, but doing so drops availability to zero, which violates the budget. The eviction is rejected indefinitely. Your cluster upgrade hangs, and an engineer eventually force-deletes the pod manually, defeating the entire point.

The rule is simple: a PDB can only ever permit a drain if your replica count gives it room to maneuver. With minAvailable: 1 you need at least 2 replicas. With maxUnavailable: 1 you need at least 2 to allow one to move at a time.

Single-replica workloads and PDBs are fundamentally at odds. If a service truly can't run two copies (a stateful leader, for example), accept that draining its node will cause a brief disruption and plan maintenance windows accordingly, rather than pretending a PDB makes it safe.

Percentages Beat Fixed Numbers

Fixed values like maxUnavailable: 1 look safe but scale badly. A service running 3 replicas tolerates losing one during a drain. The same setting on a 30-replica service makes every node drain agonizingly slow, evicting one pod at a time across dozens of nodes.

Percentages adapt:

spec:
  maxUnavailable: 25%

Now a 4-replica service loses one at a time, and a 40-replica service moves ten at a time. Be aware that Kubernetes rounds maxUnavailable percentages down and minAvailable percentages up, both in the conservative direction, so a tiny service may end up with an effective budget of zero. Always sanity-check the math against your actual replica count.

Readiness Gates: The Hidden Trap

A PDB counts pods as available based on readiness, not just existence. If your pods take 90 seconds to become ready, a drain evicting one pod waits until its replacement is fully ready before evicting the next. Slow startup, missing readiness probes, or a cold cache can turn a routine drain into a 40-minute ordeal.

This is usually a sign your readiness probe and warm-up behavior need work, not that your PDB is wrong. The PDB is correctly protecting you; the underlying pod is just slow to recover.

A Practical Default Policy

For most stateless services, this is a sane baseline:

• maxUnavailable: 25% for services with 4 or more replicas.
• For 2-3 replica services, maxUnavailable: 1.
• Never apply a PDB to single-replica deployments; scale them to 2 first or accept the disruption.
• For critical stateful systems (databases, message brokers), set maxUnavailable: 1 and verify your operator coordinates with the eviction API.

Stateful workloads deserve special care. A naive PDB on a database StatefulSet can let a drain evict the primary at the wrong moment. Good operators (for PostgreSQL, Kafka, etcd) integrate with PDBs and add their own logic to protect quorum and leadership. If yours doesn't, treat node drains on those nodes as a manual, supervised operation.

Test the Drain, Not Just the Manifest

The only way to know your PDBs work is to drain a node in staging and watch what happens. Does the drain complete in a reasonable time? Does service availability stay above your SLO during the process? Does anything get stuck?

Run this as part of your upgrade rehearsal, not for the first time during a real production upgrade. A PDB that has never been exercised is just a hopeful YAML comment. Maintenance you can perform calmly on a Tuesday afternoon is the whole goal.