Infrastructure - DevOps - Security

When the Scheduler Lies: Diagnosing Kubernetes Bin-Packing Failures Without a Platform Team - SysRoot

How wrong resource requests silently wreck node utilization, cause Pending pods and 3 a.m. evictions — and how a small team can fix it.

The problem nobody staffs for

Most small teams set Kubernetes requests and limits once, copy them across every deployment, and never look again. Then one of three things happens: pods sit Pending while nodes look half-empty in Grafana, a node fills with pods that all spike at once and the kubelet starts evicting, or your cloud bill doubles because every node is 30% utilized and you keep adding more.

All three are the same bug: the scheduler makes decisions on requests, not on actual usage. If your requests are wrong, the scheduler's model of the cluster is wrong, and every downstream capacity decision is built on a lie.

This is exactly the kind of slow-burn infrastructure problem that never gets a ticket until it causes an outage. This article is the operational playbook we use when we take over a cluster and find the scheduler has been flying blind.

How the scheduler actually packs nodes

The kube-scheduler filters nodes that can fit a pod's requests, then scores the survivors. The default NodeResourcesFit plugin uses LeastAllocated scoring, meaning it spreads pods across nodes to balance requested resources — not used resources.

        Pod requests: cpu=500m mem=512Mi
                 │
                 ▼
   ┌──────────── kube-scheduler ────────────┐
   │  FILTER:  node.allocatable >= requests  │  ← Pending if none fit
   │  SCORE:   LeastAllocated (by REQUESTS)  │
   └─────────────────────────────────────────┘
                 │
     ┌───────────┼───────────┐
     ▼           ▼           ▼
 ┌───────┐   ┌───────┐   ┌───────┐
 │node-a │   │node-b │   │node-c │
 │req 90%│   │req 40%│   │req 55%│   ← scheduler sees THIS
 │use 20%│   │use 85%│   │use 30%│   ← reality is THIS
 └───────┘   └───────┘   └───────┘

Node-a is 90% requested but 20% used — it will refuse new pods while sitting nearly idle. Node-b is 40% requested but 85% used — the scheduler happily piles more work onto a node that's already thrashing. That gap between the two rows is where every bin-packing failure lives.

Two numbers govern everything:

  • requests — reservation. The scheduler subtracts this from allocatable. Overshoot it and you strand capacity.
  • limits — ceiling. CPU over limit gets throttled; memory over limit gets OOMKilled. Undershoot it and pods die under load.

The QoS class the kubelet assigns follows directly:

QoSConditionEviction order
Guaranteedrequests == limits, for cpu+memlast
Burstablerequests set, limits higher/absentmiddle
BestEffortnothing setfirst

Under node memory pressure the kubelet evicts BestEffort first, then Burstable pods exceeding their requests, ordered by how far over-request they are. If your production API has no memory request, it's BestEffort and will be the first thing killed when a batch job goes wide.

Measuring the request-to-usage gap

You cannot fix requests you can't measure. Assuming you run kube-state-metrics and cAdvisor (both standard in kube-prometheus-stack), these PromQL queries expose the lie.

CPU: requested vs. actually used, per namespace

# What we reserved
sum by (namespace) (
  kube_pod_container_resource_requests{resource="cpu"}
)

# What we burned (5m rate)
sum by (namespace) (
  rate(container_cpu_usage_seconds_total{container!="",pod!=""}[5m])
)

Per-workload over-provisioning ratio — the single most useful number. Anything consistently above ~4x is wasting money; below ~1.2x is a throttling risk.

sum by (namespace, pod) (
  kube_pod_container_resource_requests{resource="cpu"}
)
/
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Node allocation vs. real usage — this catches the stranded-capacity problem directly.

# Fraction of allocatable CPU that is REQUESTED
sum by (node) (kube_pod_container_resource_requests{resource="cpu"})
/
sum by (node) (kube_node_status_allocatable{resource="cpu"})

# Fraction of allocatable CPU actually USED
sum by (node) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/
sum by (node) (kube_node_status_allocatable{resource="cpu"})

A Grafana alert that fires when nodes are request-saturated but idle — the pattern that makes you add nodes you don't need:

{
  "alert": "NodeRequestSaturatedButIdle",
  "expr": "(sum by (node) (kube_pod_container_resource_requests{resource=\"cpu\"}) / sum by (node) (kube_node_status_allocatable{resource=\"cpu\"}) > 0.85) and (sum by (node) (rate(container_cpu_usage_seconds_total{container!=\"\"}[5m])) / sum by (node) (kube_node_status_allocatable{resource=\"cpu\"}) < 0.35)",
  "for": "30m",
  "labels": { "severity": "warning", "team": "platform" },
  "annotations": {
    "summary": "Node {{ $labels.node }} is >85% requested but <35% used",
    "description": "Requests are over-provisioned. Right-size workloads before scaling out."
  }
}

Sizing rule we apply after measuring: set the CPU request to the p95 of 14 days of usage, and the memory request to the p99. Memory is uncompressible — you cannot throttle it, you can only kill — so give it more headroom than CPU.

# p95 CPU over 14d for a workload — use this as the new request
quantile_over_time(0.95,
  sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="api",container!=""}[5m]))[14d:5m]
)

Getting the manifest right

Here's a Burstable deployment sized from real data, with the throttling and OOM footguns removed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: api
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api }
    spec:
      # Spread across nodes so one node loss doesn't take the service down
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: orders-api }
      containers:
        - name: orders-api
          image: registry.internal/orders-api:1.8.2
          resources:
            requests:
              cpu: "250m"      # p95 measured usage
              memory: "384Mi"  # p99 measured usage
            limits:
              # NO cpu limit on purpose — see note below
              memory: "512Mi"  # hard ceiling, ~1.3x request
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 3

Two deliberate choices worth defending:

  • No CPU limit. CPU is compressible. A CPU limit implemented via CFS quota can throttle a latency-sensitive service even when the node has spare cores, because the throttle is per-quota-period (100ms) not per-second. For request/response services we set the request accurately and omit the limit, letting the pod burst into idle node capacity. For untrusted or noisy batch work we keep a limit. This is a trade-off, not a universal rule.
  • Memory limit ~1.3x request, not open-ended. An open memory limit lets one leaking pod consume the node and trigger a node-wide OOM cascade. A tight ceiling contains the blast radius to a single pod restart.

To stop the whole problem recurring, enforce a floor with a LimitRange so nobody deploys a BestEffort pod into production:

apiVersion: v1
kind: LimitRange
metadata:
  name: enforce-requests
  namespace: api
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default:        { memory: 256Mi }
      min:            { cpu: 50m,  memory: 64Mi }
      max:            { memory: 2Gi }

Failure modes & how we'd debug it

1. Pods stuck Pending while kubectl top nodes shows idle nodes

Classic stranded-capacity symptom. The nodes have free CPU cycles but no free requests.

# See why it can't schedule — the message is literal
kubectl describe pod orders-api-xxxx -n api | grep -A5 Events
# -> "0/6 nodes are available: 6 Insufficient cpu."

# Confirm the gap: requests near 100%, usage low
kubectl describe nodes | grep -A5 "Allocated resources"
kubectl top nodes

Reasoning: describe node shows Requests percentages; top nodes shows actual usage. If Requests is ~100% but usage is 30%, over-provisioned requests are the cause, not real load. Fix the offending workloads' requests (find them with the over-provisioning-ratio query above) rather than adding nodes.

2. Healthy pods getting OOMKilled during traffic spikes

# Confirm the kill reason
kubectl get pod orders-api-xxxx -n api -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# -> OOMKilled

kubectl describe pod orders-api-xxxx -n api | grep -i -A3 "Last State"

# Working set vs. limit — did we actually approach the ceiling?
container_memory_working_set_bytes{namespace="api",pod=~"orders-api.*"}
/
kube_pod_container_resource_limits{namespace="api",resource="memory"}

Reasoning: if working set hits 1.0, the limit is genuinely too low — raise the memory request/limit to the p99 + headroom. If it spikes only during GC or a specific endpoint, it's a leak or unbounded buffer in the app; raising the limit only delays the crash. Distinguish the two before touching the manifest.

3. Latency spikes with plenty of free CPU — hidden throttling

The sneakiest one. Requests look fine, node has spare cores, but p99 latency doubles.

# Fraction of CFS periods that got throttled
rate(container_cpu_cfs_throttled_periods_total{namespace="api"}[5m])
/
rate(container_cpu_cfs_periods_total{namespace="api"}[5m])

Reasoning: anything above ~0.05 (5% of periods throttled) means a CPU limit is choking the pod during bursts even though the node is idle. Either raise the CPU limit substantially or remove it and rely on an accurate request. This is invisible in kubectl top because average usage stays low — the throttling happens in sub-100ms bursts.

4. Node NotReady after eviction storm — and did anyone touch it?

kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp
# On the node itself:
journalctl -u kubelet --since "20 min ago" | grep -i "evict\|MemoryPressure"

Reasoning: MemoryPressure in kubelet logs means the node crossed its eviction threshold (default memory.available<100Mi). The kubelet evicts BestEffort → over-request Burstable pods to reclaim memory. The fix is proper requests + LimitRange, so no pod is BestEffort and the eviction math is predictable.

Because eviction storms and node changes are also a security-relevant signal, we surface them in Wazuh. A rule watching the kubelet journal on each node:

<group name="kubernetes,eviction,">
  <rule id="100420" level="8">
    <decoded_as>json</decoded_as>
    <field name="SYSLOG_IDENTIFIER">kubelet</field>
    <match>attempting to reclaim|eviction manager</match>
    <description>Kubelet is evicting pods under node resource pressure</description>
  </rule>
  <rule id="100421" level="10" frequency="5" timeframe="120">
    <if_matched_sid>100420</if_matched_sid>
    <description>Eviction storm: 5+ eviction events in 2 minutes</description>
  </rule>
</group>

Feed the journal into Wazuh via ossec.conf on each node:

<localfile>
  <log_format>journald</log_format>
  <location>SYSLOG_IDENTIFIER=kubelet</location>
</localfile>

Runbook: right-sizing a workload under pressure

Use this when a service is being throttled/OOMKilled or when you've been paged for NodeRequestSaturatedButIdle.

  1. Confirm the QoS class — this tells you eviction risk immediately. 
    kubectl get pod -n api -l app=orders-api \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.qosClass}{"\n"}{end}'
  2. Pull 14 days of real usage. Run the p95 CPU and p99 memory queries from earlier in Grafana Explore for the target namespace/pod.
  3. Check for throttling with the CFS query. If >5%, the CPU limit is the culprit — not the request.
  4. Compute new values. CPU request = p95. Memory request = p99. Memory limit = p99 × 1.3, rounded up. Drop the CPU limit for latency-sensitive services.
  5. Apply as a patch, not a full re-deploy, so you can watch it roll. 
    kubectl set resources deployment/orders-api -n api \
  --requests=cpu=250m,memory=384Mi \
  --limits=memory=512Mi
  6. Watch the rollout and probes. 
    kubectl rollout status deployment/orders-api -n api --timeout=120s
kubectl get pods -n api -l app=orders-api -w
  7. Verify the gap closed. Re-run the node request-vs-usage PromQL. Request % should drop; usage % unchanged.
  8. If pods now go Pending, you've correctly reserved real capacity and the cluster is genuinely too small — scale the node pool by one and re-check, don't revert the requests.
  9. Record the new numbers in the deployment manifest in Git. A kubectl set that isn't committed will vanish on the next CI deploy.
  10. Silence the alert only after usage-based verification, not before.

Wiring the guardrails into CI/CD

The durable fix is preventing unsized manifests from ever reaching the cluster. This GitLab CI job fails the pipeline if any container lacks requests, using kubeconform for schema plus a small check:

validate-resources:
  stage: validate
  image: alpine:3.20
  before_script:
    - apk add --no-cache yq
  script:
    - |
      for f in k8s/**/*.yaml; do
        kinds=$(yq '.kind' "$f")
        echo "$kinds" | grep -qE 'Deployment|StatefulSet|DaemonSet' || continue
        missing=$(yq '.spec.template.spec.containers[]
          | select(.resources.requests.cpu == null
                or .resources.requests.memory == null)
          | .name' "$f")
        if [ -n "$missing" ]; then
          echo "FAIL: containers missing requests in $f: $missing"
          exit 1
        fi
      done
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

Pair it with the LimitRange above: CI catches it at review, the LimitRange catches anything that slips through, and the Grafana alert catches drift over time. Three layers, each cheap.

Pre-flight & hardening checklist

  • Every production container has both a CPU and memory request set from measured p95/p99, not guessed.
  • Memory limit set at ~1.3x request; no unbounded memory limits in production.
  • CPU limits removed for latency-sensitive services after verifying CFS throttling is <5%.
  • No BestEffort pods in production namespaces (verify with the QoS jsonpath query).
  • LimitRange deployed in every application namespace with sane defaultRequest and min.
  • NodeRequestSaturatedButIdle alert active in Grafana with a 30m for clause.
  • topologySpreadConstraints set so losing one node can't take a whole service down.
  • CI blocks merges of Deployments/StatefulSets missing requests.
  • Kubelet eviction events shipped to Wazuh (rule 100420/100421) and to on-call.
  • Node pool right-sized after fixing requests, not before — capacity added only when real usage justifies it.
  • New resource values committed to Git; no undocumented kubectl set resources in production.

The bottom line

Bin-packing failures aren't a scaling problem — they're a measurement problem. The scheduler is only as good as the requests you feed it, and most teams feed it fiction. Close the gap between requested and used, enforce a floor in CI and via LimitRange, watch the request-vs-usage ratio in Grafana, and you'll recover stranded capacity, stop the 3 a.m. evictions, and stop paying for nodes you never actually used. None of it requires a standing platform team — it requires doing the measurement once, correctly, and wiring the guardrails so it stays fixed. <<<END>>>

Struggling with Managed DevOps?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Managed DevOps, talk to the SysRoot team and we'll help you get it under control.

Back to Blog

Published on: July 9, 2026
Tags: kubernetes, devops, observability, capacity-planning