vCPU Topology and NUMA: The Hidden Performance Tax on Your VMs - SysRoot

vCPU Topology and NUMA: The Hidden Performance Tax on Your VMs

Most teams provision VMs by picking a number. "Give it 16 vCPUs and 64GB of RAM." The hypervisor accepts the request, the VM boots, and everyone moves on. Months later a database starts showing erratic tail latency, a JVM garbage collector stalls under load, or a packet-processing service drops throughput by 30% with no obvious cause. CPU graphs look fine. Memory isn't saturated. The host isn't even particularly busy.

The culprit is frequently NUMA topology: how the guest's virtual CPUs and memory map onto the physical sockets, cores, and memory controllers of the host. This is one of the least visible aspects of bare metal and VM management, and it punishes you silently. This article walks through what actually happens, where VMware and Proxmox differ, the common misconfigurations, and a checklist you can apply this week.

What NUMA actually is, briefly

Modern dual-socket servers are NUMA (Non-Uniform Memory Access) machines. Each physical CPU socket has its own memory controller and a bank of RAM directly attached to it. That's a NUMA node. A core on socket 0 can access memory attached to socket 0 quickly. Accessing memory attached to socket 1 has to traverse the inter-socket interconnect (UPI on Intel, Infinity Fabric on AMD), which adds latency and consumes shared bandwidth.

The penalty for remote memory access is real but not catastrophic in isolation: typically 1.5x to 2x the latency of local access. The problem is scale. A workload doing millions of memory operations per second, with half of them crossing the interconnect, accumulates a continuous tax. Worse, the interconnect is a shared resource, so a single misplaced VM degrades neighbors too.

Single-socket AMD EPYC and some Intel parts further subdivide a socket into multiple NUMA domains internally (NPS settings, sub-NUMA clustering). So even a one-socket host can be a multi-NUMA machine. Do not assume single socket means single node.

How VMs get this wrong

When you create a VM larger than a single physical NUMA node, the hypervisor has two choices. It can spread the VM's vCPUs and memory across multiple nodes (creating a wide VM), or it can try to expose multiple virtual NUMA nodes (vNUMA) to the guest so the guest OS can make smart scheduling decisions.

Things break in three common ways.

Failure mode 1: the oversized VM that straddles nodes

Suppose your host has two sockets, each with 16 physical cores and 192GB of RAM local to that socket. You create a VM with 24 vCPUs. That VM cannot fit inside one node. The hypervisor scatters its vCPUs across both sockets, and its memory ends up partly local, partly remote, depending on allocation timing.

Now the guest OS has no idea any of this is happening unless vNUMA is configured. It treats all 24 vCPUs as uniform, schedules a latency-sensitive thread on a vCPU backed by socket 0, but that thread's data lives in memory on socket 1. Every access pays the interconnect tax. Throughput drops and tail latency becomes unpredictable because placement is effectively random across reschedules.

Failure mode 2: misleading vCPU topology

By default many platforms present vCPUs as sockets with one core each. A 16-vCPU VM might be advertised to the guest as 16 sockets, 1 core per socket. The guest OS and applications make NUMA and cache-locality decisions based on this fake topology. The JVM, PostgreSQL, SQL Server, and the Linux scheduler all read topology to decide thread placement. Garbage feeds them garbage decisions.

Failure mode 3: memory ballooning and migration shuffling layout

Even a correctly placed VM can drift. Live migration (vMotion, Proxmox migration) moves a VM to a host with a different topology. Memory hot-add, ballooning, and overcommit can cause memory to be reallocated remotely. A VM that was perfectly local on Monday is straddling nodes on Friday after a maintenance window, and nobody notices because nothing alerts on NUMA locality.

A realistic scenario

A fintech team runs a Redis-backed pricing service on a Proxmox cluster. Each node is a dual-socket box, 16 cores per socket, 256GB RAM split evenly. The pricing VM was provisioned with 20 vCPUs and 96GB RAM because "it was busy." P99 latency on the pricing API sat around 4ms, which was tolerable but the team wanted lower.

During an incident review they noticed P99 spiked to 40ms after every migration, then slowly settled. The pattern matched maintenance windows. Investigation showed the 20-vCPU VM could never fit in a 16-core node, so it was always split across both sockets. Redis is single-threaded for command execution and extremely sensitive to memory latency. Roughly half its hot dataset lived on the remote socket.

The fix was structural, not tuning. They cut the VM to 14 vCPUs and 90GB RAM so it fit inside one NUMA node, pinned it to that node, and disabled ballooning for it. P99 dropped to 1.8ms and stopped spiking after migrations. The smaller VM was faster than the larger one, which is the counterintuitive lesson NUMA teaches over and over: a VM sized to fit a node beats a bigger VM that straddles two.

VMware specifics

VMware's ESXi scheduler is NUMA-aware and does a reasonable job by default, but defaults have edges.

• Wide-VM and vNUMA: ESXi automatically exposes vNUMA to the guest when a VM exceeds 8 vCPUs (the numa.vcpu.min threshold) and spans more than one physical node. If your VM has, say, 10 vCPUs but fits in one node, vNUMA may not be presented, which is usually fine. The trap is when you have hosts with small nodes.
• Cores per socket: The Cores per Socket setting in the VM hardware directly influences vNUMA layout. Historically people set this to manipulate Windows licensing. Doing so can force a vNUMA topology that doesn't match the physical host and hurts performance. Modern guidance: leave cores-per-socket aligned with the physical NUMA node size, or let ESXi decide.
• Latency-sensitive setting: For truly latency-critical VMs, the Latency Sensitivity = High setting plus full CPU and memory reservations gives the VM exclusive pinned cores and local memory. This is powerful but costly: you give up overcommit on those cores entirely.
• vMotion across heterogeneous hosts: vNUMA topology is computed at power-on and does not change on migration. Migrating a VM whose vNUMA was sized for a host with 16-core nodes onto a host with 12-core nodes creates a mismatch. Keep clusters homogeneous where you can.

Proxmox specifics

Proxmox sits on KVM/QEMU and gives you more direct, lower-level control, which means more rope.

• The numa flag: Enabling NUMA on a VM (numa: 1) lets you define virtual NUMA nodes and map them to host nodes. Without it, a large VM gets a flat topology and the host scheduler does best-effort placement.
• vCPU topology: Set sockets, cores, and vcpus deliberately. Presenting sockets: 2, cores: 8 for a 16-vCPU VM that you intend to span two host sockets is reasonable; presenting sockets: 16, cores: 1 is almost always wrong.
• CPU pinning: Proxmox supports taskset/cgroup-based affinity and, via the hostpids/affinity options or manual vcpu pinning, you can bind vCPUs to specific physical cores. This is the strongest guarantee but disables the flexibility you bought virtualization for. Use it only on dedicated, latency-sensitive hosts.
• Hugepages and memory locality: For high-throughput VMs, backing guest memory with hugepages allocated per NUMA node (hugepages: 1024 with node binding) reduces TLB pressure and keeps memory local. This needs host-level configuration and reduces flexibility.
• Ballooning: The default balloon device can reclaim and reallocate guest memory in ways that break locality. For NUMA-sensitive VMs, set the balloon minimum equal to the maximum to effectively disable it.

How to diagnose NUMA problems

You cannot fix what you cannot see. The symptom is usually "variable latency or throughput that doesn't correlate with CPU utilization."

• On a Linux guest, numactl --hardware shows the topology the guest believes it has. Compare it against reality.
• On the host, numastat -c qemu (Proxmox/KVM) shows per-VM memory distribution across nodes. Lots of remote memory for one VM is your smoking gun.
• On ESXi, esxtop in memory mode shows NUMA statistics: N%L (percentage of memory local to the VM). Anything well below ~80% on a busy VM is suspicious.
• Watch for migration-correlated regressions. If latency degrades after every maintenance window and recovers slowly, suspect layout drift.

A practical checklist

Use this when provisioning or auditing latency-sensitive VMs.

1. Know your host node size first. Record cores per socket and RAM per socket for every host model in the cluster. This is the unit you size against.
2. Size VMs to fit inside one NUMA node when possible. If a VM needs 14 vCPUs and your node has 16 cores, keep it on one node rather than rounding up to 20.
3. If a VM must span nodes, enable vNUMA and present a vCPU topology that mirrors the physical layout (e.g., 2 sockets of 8 cores, not 16 single-core sockets).
4. Reserve memory for latency-critical VMs and pin them so their memory stays local. Disable ballooning for those VMs.
5. Keep clusters homogeneous. Mixed node sizes break vNUMA assumptions on migration.
6. Verify after migration. Add a post-maintenance check of memory locality for your top latency-sensitive VMs.
7. Measure, don't guess. Confirm with numastat, esxtop, or in-guest numactl before and after any change.

The trade-off you are actually making

NUMA optimization pulls against the entire reason you virtualized: flexibility, overcommit, and easy live migration. Pinning vCPUs and reserving memory turns a flexible VM into something close to a dedicated machine. That is the right call for a database primary or a packet-processing service, and the wrong call for a fleet of stateless web workers where the scheduler's flexibility is worth more than a few microseconds.

The discipline is to classify workloads. Identify the handful of VMs where memory latency genuinely matters, treat them as NUMA-sensitive citizens with proper sizing and pinning, and let everything else float. The mistake is treating all VMs identically and then being surprised when the one workload that cares about locality misbehaves.

NUMA is invisible until it isn't. Build the host topology into your provisioning decisions, measure locality on the VMs that matter, and you eliminate a whole class of unexplained latency before it ever pages someone.

---

Struggling with Bare Metal & VM Management?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Bare Metal & VM Management, talk to the SysRoot team and we'll help you get it under control.