Memory Overcommit and Noisy Neighbors: Tuning VM Density Without Killing Performance - SysRoot

The promise and the trap of overcommit

Virtualization sells you a comforting lie: that the sum of your guests can exceed the physical resources of the host. Most of the time, that lie holds. Your VMs rarely all peak at once, so you can pack 200 GB of allocated guest memory onto a 128 GB host and nobody notices. This is overcommit, and it is the single biggest lever for reducing hardware spend on both VMware and Proxmox.

It is also the most common source of mysterious, intermittent performance problems that take days to diagnose. When the lie stops holding — during a backup window, a traffic spike, or a host failover that suddenly doubles the load on a survivor — the hypervisor starts making decisions on your behalf. It reclaims memory, it queues CPU, and it pages guest RAM to disk. Your application doesn't crash. It just gets slow in ways that don't show up in any single VM's metrics.

This article is about finding the line between healthy consolidation and dangerous over-density, specifically for memory and CPU, on VMware vSphere and Proxmox VE, including hybrid estates where you run both.

How memory overcommit actually works

Before tuning, you need to know which mechanism is reclaiming memory, because they have wildly different performance costs.

VMware's reclamation hierarchy

ESXi reclaims memory under contention in a specific order, from cheapest to most destructive:

1. Transparent Page Sharing (TPS) — deduplicates identical memory pages. Largely neutered for cross-VM sharing since 2014 for security reasons; intra-VM sharing remains.
2. Ballooning — the vmmemctl driver inside VMware Tools inflates a balloon inside the guest, forcing the guest OS to give up its least-used pages using its own paging logic. This is the polite mechanism: the guest decides what to evict.
3. Memory compression — compresses pages that would otherwise be swapped, keeping them in RAM.
4. Hypervisor swapping — ESXi swaps guest memory to a .vswp file on disk, with no knowledge of what's hot or cold. This is the destructive one. Latency goes from nanoseconds to milliseconds.

The failure mode you want to avoid is hypervisor swapping. If you see non-zero Memory Swapped (SWCUR) or active swap-in (SWR/s) on a host, latency-sensitive guests on that host are already suffering.

Proxmox and the KVM/KSM model

Proxmox VE uses KVM, and its tooling is blunter. The two relevant mechanisms:

• KSM (Kernel Samepage Merging) — Proxmox's equivalent of TPS, merging identical pages across guests. It kicks in by default when host memory usage crosses a threshold (controlled by ksmtuned). It saves real RAM but burns CPU scanning pages, and the savings evaporate the moment guests write to merged pages.
• Ballooning — Proxmox supports virtio ballooning. If you set a VM's memory with a minimum lower than the assigned maximum, the balloon driver dynamically reclaims down to the minimum under host pressure.
• Host-level swapping — if you overcommit past what KSM and ballooning recover, the Linux host kernel starts swapping. Same disaster as ESXi swapping, except now you're also at the mercy of the host's swappiness and your swap device's speed.

The critical Proxmox-specific risk: the OOM killer. If you disable swap on the host (common advice to avoid swap-induced latency) and then overcommit memory, the host kernel can invoke the OOM killer and terminate a QEMU process — killing an entire VM instantly, with no graceful shutdown. This is the Proxmox equivalent of a node failure, self-inflicted by over-density.

The noisy neighbor problem

Overcommit interacts with the noisy neighbor effect: one VM's behavior degrading its co-tenants. It shows up in three resources.

Memory

A single VM that suddenly allocates aggressively — a database loading a large index, a build job, a memory leak — pushes the host into contention. The hypervisor balloons or swaps other VMs to satisfy it. The victim isn't the noisy VM; it's the quiet web server next to it that suddenly can't keep its working set in RAM.

CPU and the ready-time trap

CPU overcommit is measured by CPU Ready (VMware) or steal time (Proxmox/Linux guests, visible as st in top). Ready time is the milliseconds a vCPU spent runnable but waiting for a physical core. The classic mistake is provisioning fat VMs: an 8-vCPU guest on a host with 16 logical cores needs to co-schedule all 8 vCPUs (or enough of them) simultaneously. Under load, that VM waits more than a 2-vCPU guest would.

A practical rule: a single 8-vCPU VM that uses 10% CPU is worse for the cluster than four 2-vCPU VMs at the same total load, because of co-scheduling overhead. Right-size vCPU counts down aggressively.

Storage and the backup window

The most common noisy-neighbor incident in practice isn't CPU or RAM — it's a backup or snapshot consolidation saturating shared storage IOPS, which then causes memory pressure because guests can't flush dirty pages fast enough. Density problems often manifest at 2 a.m. during backups, not during peak traffic.

A realistic scenario

A 30-person SaaS company runs two Proxmox hosts, 128 GB each, in a manual failover pair. Across them: 18 VMs totaling 200 GB of assigned RAM. Day to day, actual usage sits around 95 GB per host, so KSM keeps things comfortable and nobody thinks about it.

One host's PSU fails. The team migrates the survivors' VMs onto the remaining host — now 18 VMs, 200 GB assigned, on one 128 GB box. KSM ramps up, scanning frantically, burning a CPU core. Memory usage climbs past 128 GB. The host has 8 GB of swap on a SATA SSD. The kernel starts swapping QEMU memory.

The Postgres VM, which had been doing 2 ms queries, now sees random 40 ms stalls as its buffer cache pages get swapped out and back. The application's p99 latency goes from 80 ms to 1.2 s. No VM has crashed. Every VM's own CPU graph looks fine. The on-call engineer spends two hours staring at application logs before someone runs vmstat 1 on the host and sees si/so columns pinned.

The root cause was never the PSU. It was that the cluster was provisioned with no headroom for the loss of one node. N+1 redundancy means each host must be able to absorb the others' load without overcommitting into swap.

Setting the density line

Here is the operational discipline. None of this is exotic; it's just rarely enforced.

1. Measure consumed, not assigned

Assigned memory is fiction. What matters is active or consumed memory. On VMware, watch Active and Consumed per VM. On Proxmox, the assigned figure overstates reality; use guest-level metrics and host free -m. Size your overcommit ratio against consumed peaks, not assignments.

2. Define a hard headroom budget

Reserve capacity explicitly:

• Memory: keep host consumed memory under ~80% of physical under normal load, leaving room for hypervisor overhead and failover absorption.
• CPU: keep average host CPU under 70–75%, and track CPU Ready / steal time as the real signal, not raw utilization.
• Failover: in an N-host cluster, sum of all VMs must fit within (N-1) hosts' usable capacity without swapping.

3. Set reservations for the workloads that matter

Not all VMs are equal. For latency-sensitive guests — databases, message brokers, anything with an SLA — set a memory reservation in VMware so that RAM can never be ballooned or swapped away. In Proxmox, set the balloon minimum equal to the assigned memory (effectively disabling ballooning for that VM) for critical guests. You're trading some density for guaranteed performance on the workloads where it counts.

4. Right-size vCPUs downward

Audit every VM's vCPU count against actual CPU usage. If a VM rarely exceeds the equivalent of two cores, give it two. Wide VMs are co-scheduling tax you pay constantly.

5. Decide your swap posture deliberately

On Proxmox, the choice is between two bad outcomes under extreme pressure: slow VMs (swap enabled) or dead VMs (swap disabled, OOM killer). The pragmatic stance for most teams: keep a small, fast swap on the host as a shock absorber, set vm.swappiness low (1–10) so the kernel only swaps under genuine pressure, and alert aggressively on any swap activity so you fix density before it becomes routine. Treat swap-in as an incident signal, not a normal operating state.

6. Alert on the right metrics

The metrics that catch density problems early:

• VMware: Memory Ballooned, Memory Swapped, CPU Ready (% per vCPU).
• Proxmox/KVM: host si/so from vmstat, KSM pages shared and CPU consumed by ksmd, guest steal time.

Most teams alert on guest CPU and memory, which are exactly the metrics that look healthy during a density incident. Alert on the host's reclamation activity instead.

Hybrid VMware + Proxmox considerations

Many teams run VMware for legacy or vendor-mandated workloads and Proxmox for everything new, often to escape licensing costs. The density risk in hybrid estates is inconsistent policy: VMware's reservations and DRS give you automated load balancing and admission control, while Proxmox gives you none of that out of the box. A VM that's safely placed by DRS on VMware can be manually migrated onto an already-stressed Proxmox host with no admission check stopping you.

If you run both, document the per-platform density rules explicitly, because the tooling won't enforce them for you on the Proxmox side. Consider a thin capacity-tracking spreadsheet or, better, a Prometheus dashboard that shows committed-vs-physical per host across both platforms in one view. The goal is that no migration decision is made without seeing real headroom.

A short pre-consolidation checklist

Before you increase VM density on any host, confirm:

1. Host consumed memory stays under 80% at the new density, measured at peak.
2. The cluster still survives losing one host without swapping.
3. Latency-sensitive VMs have memory reservations (VMware) or balloon-min == max (Proxmox).
4. vCPU counts are right-sized; no VM is wider than it needs to be.
5. Backup/snapshot windows don't overlap with peak load on the same storage.
6. Alerts exist on ballooning, swapping, CPU Ready, and steal time — not just guest utilization.
7. Swap posture is deliberate and documented, with swap-in treated as an incident.

The bottom line

Overcommit is a legitimate, valuable tool. Hardware is expensive and most workloads are bursty, so consolidation pays for itself. The discipline isn't avoiding overcommit — it's knowing exactly where your line is, reserving headroom for failover, protecting the workloads that have SLAs, and alerting on hypervisor reclamation rather than guest metrics that lie to you during an incident.

The teams that get burned aren't the ones who overcommit. They're the ones who overcommit by accident, discover their true density only when a host fails, and then spend a sleepless night learning the difference between ballooning and swapping the hard way.

---

Struggling with Bare Metal & VM Management?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Bare Metal & VM Management, talk to the SysRoot team and we'll help you get it under control.