Readiness vs. Liveness Probes: The Misconfiguration That Crashes Healthy Pods - SysRoot

Kubernetes health probes are deceptively simple to configure and dangerously easy to get wrong. A single misconfigured livenessProbe can take a perfectly healthy application and restart it into oblivion under load. We see this pattern constantly: an engineering team copies a probe block from a tutorial, ships it, and months later discovers that their pods are being killed precisely when they're busiest.

This article breaks down what each probe actually does, the failure modes that follow from confusing them, and how to configure probes that protect availability instead of sabotaging it.

Three Probes, Three Jobs

Kubernetes offers three probe types, and each answers a different question:

• Liveness probe: "Is this container broken beyond recovery?" If it fails, the kubelet kills and restarts the container.
• Readiness probe: "Can this container serve traffic right now?" If it fails, the pod is removed from Service endpoints but keeps running.
• Startup probe: "Has this container finished booting?" It disables the other probes until the app is up.

The critical distinction is the consequence. A failed readiness probe is reversible and non-destructive — traffic stops flowing until the pod recovers. A failed liveness probe is a death sentence. Confusing the two is where outages begin.

The Classic Failure: Liveness Probes Under Load

The most common self-inflicted outage looks like this. A team points their liveness probe at /health, an endpoint that runs a database query or checks a downstream dependency. Under normal conditions, it responds in 20ms. Then traffic spikes, the database slows down, and the health endpoint starts taking 2 seconds to respond.

The liveness probe times out. Kubernetes concludes the container is dead and restarts it. The restart drops in-flight requests, the pod rejoins the pool cold, and the remaining pods absorb even more load — making their health checks slow too. Within minutes you have a cascading restart storm, all triggered by a database that was merely slow, not down.

The lesson: liveness probes should test the process, not its dependencies. If your app can't talk to the database, restarting the pod won't fix the database. It only adds chaos. Reserve liveness for genuine deadlock or unrecoverable state — a process that has hung and will never recover on its own.

Readiness Probes Are Where Dependencies Belong

Dependency checks belong in the readiness probe. If the database is unreachable, the right behavior is to stop sending traffic to the pod — not to kill it. When the dependency recovers, the readiness probe passes again and traffic resumes, no restart required.

This also matters during deployments. A pod that's still warming caches or establishing connection pools should fail readiness until it's genuinely ready. Without this, Kubernetes routes traffic to a pod that returns errors the moment it starts, and your rollout looks like an outage.

Startup Probes Fix Slow Boots

Applications that take 30+ seconds to start — JVM services, apps loading large models, anything with heavy initialization — create a dilemma. Set a liveness probe aggressive enough to catch real hangs, and it kills the container before it finishes booting. Set it lenient enough to survive startup, and it's too slow to catch hangs in steady state.

The startup probe resolves this. It runs first, with a generous failureThreshold and periodSeconds, and the liveness and readiness probes don't begin until it passes. This lets you keep tight liveness timing for the running state while tolerating a slow boot.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 5   # allows up to 150s to start
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Practical Configuration Rules

A few rules keep probes from becoming a liability:

• Separate endpoints. Use /healthz for liveness (process-only, no external calls) and /ready for readiness (dependency-aware). Don't share one endpoint.
• Never check dependencies in liveness. No database, no cache, no downstream API.
• Give timeouts headroom. A timeoutSeconds of 1 against an endpoint that occasionally takes 1.5s will flap. Measure real latency and add margin.
• Tune failure thresholds. A single failed check shouldn't restart a pod. Require 3 consecutive failures so a transient blip doesn't trigger a restart.
• Watch for probe-induced load. Probes hitting expensive endpoints every second across hundreds of pods generate real traffic. Keep them cheap.

Closing the Loop

Probes are control loops, and like any control loop they can oscillate destructively when tuned wrong. The single highest-impact change most teams can make is auditing every liveness probe and stripping out dependency checks. Move those to readiness, add a startup probe for slow boots, and give your timeouts and thresholds enough margin to survive a bad afternoon. Done right, probes make degradation graceful. Done wrong, they turn a slowdown into an outage you caused yourself.