Proxmox Live Migration: What Actually Breaks When You Move a Running VM - SysRoot

A production deep-dive into live migration failure modes, shared-storage requirements, and a runbook for evacuating a node without downtime.

Live migration is the feature everyone assumes "just works" until the first time a node has bad RAM and you need to evacuate 18 production VMs at 02:00 without a maintenance window. Then you discover that migration speed is gated by your network MTU, that one VM has a PCI passthrough device and refuses to move at all, and that the storage you thought was shared is actually local ZFS on each node.

This article is about the concrete mechanics of Proxmox VE live migration: how it works, the exact preconditions, the failure modes we've actually hit in production, and a runbook for draining a node safely. Everything here is copy-pasteable against PVE 8.x.

Direct answer: Proxmox LXC live migration is not supported

Proxmox VE does not support true live migration for LXC containers. A container migration is a stop/start operation: Proxmox stops the container on the source node, moves or reuses its storage on the destination, and starts it again. That means there is downtime, even if the storage is shared and the restart is quick.

The migration command pattern is:

pct migrate <id> node-b --restart

The --restart behavior means the container is stopped on the source and restarted on the destination. CRIU references or online-style wording do not change that stop/restart behavior or provide true live memory migration for LXC containers as described here.

Shared storage can reduce migration time because the destination can reuse the container's storage instead of copying it, but it does not preserve the running container's in-memory state or make the operation live.

This is different from QEMU/KVM VM live migration, where Proxmox can copy guest memory while the VM keeps running and then perform a short final stop-copy handoff. LXC containers do not have the same live memory migration path in Proxmox VE.

If you see errors or search for phrases like proxmox lxc live migration not supported, criu not supported, or restart migration, the practical answer is:

VMs can live-migrate when CPU, storage, network and device preconditions are met.
LXC containers should be planned as restart migrations.
Stateful LXC services need an application-level failover plan, a maintenance window, or conversion to a VM if true live migration is required.
Shared storage reduces copy time, but it does not turn LXC migration into live migration.

For production maintenance, inventory containers separately from VMs and communicate a downtime budget for every pct migrate operation.

How live migration actually works

There are two completely different code paths, and confusing them is the root of most failures.

                 LIVE MIGRATION (running VM)
  ┌──────────────┐                         ┌──────────────┐
  │   node-a     │                         │   node-b     │
  │  ┌────────┐  │   1. memory pre-copy    │  ┌────────┐  │
  │  │ VM 101 │──┼──── (dirty-page) ──────►│  │ VM 101 │  │
  │  │ RUNNING│  │   2. final stop-copy    │  │ paused │  │
  │  └────────┘  │   3. CPU state handoff  │  └────────┘  │
  └──────┬───────┘   4. resume on node-b   └──────┬───────┘
         │                                        │
         └────────────┬───────────────────────────┘
                      ▼
              SHARED STORAGE  (Ceph / NFS / iSCSI-LVM)
              disk image stays put — only RAM moves

                 WITHOUT shared storage:
              + disk is also copied block-by-block
                (qemu drive-mirror) → much slower, more failure surface

With shared storage (Ceph RBD, NFS, iSCSI+LVM), only RAM and CPU state move. Proxmox does an iterative memory pre-copy: it copies all guest RAM while the VM runs, then re-copies pages dirtied during the copy, repeating until the remaining dirty set is small enough to copy in a sub-second stop-the-world window.

With local storage (local ZFS, local LVM, directory), the disk also has to be mirrored over the network with drive-mirror. This is what Proxmox calls --with-local-disks. It works, but the failure surface and duration explode.

If your migration "hangs at 80%," you almost always have a VM dirtying memory faster than your migration link can copy it. More on that below.

Preconditions that bite you

Live migration requires:

Matching or compatible CPU types. If VMs run with cpu: host, the destination CPU must be the same family/flags. Mixing a Skylake and an EPYC node and migrating a host-CPU VM will fault the guest on resume. Use a named model like x86-64-v3 for a heterogeneous cluster.
No un-migratable hardware. PCI passthrough (hostpci0), USB passthrough of physical devices, and args: with raw QEMU device handles block live migration. These can only be migrated offline.
A real migration network. By default migration uses the cluster (corosync) network — the worst possible choice, because saturating it can fence nodes.

Configure a dedicated migration network in /etc/pve/datacenter.cfg:

# /etc/pve/datacenter.cfg
migration: secure,network=10.10.50.0/24
# 'secure' tunnels over SSH (encrypted, ~1-2 Gbit ceiling on a single core)
# 'insecure' uses a raw TCP socket — far faster on 10/25GbE, LAN-only

For a trusted, isolated 10GbE+ storage/migration VLAN we use insecure because the SSH tunnel becomes a single-core bottleneck around 2 Gbit/s. On 25GbE the difference is the gap between a 40-second migration and a 6-minute one.

Set migration bandwidth and parallelism so an evacuation doesn't starve replication or storage:

# /etc/pve/datacenter.cfg
migration: insecure,network=10.10.50.0/24
# cap aggregate migration bandwidth (in MiB/s) to leave headroom for Ceph
bwlimit: migration=600

You can also override per-VM at migrate time with --bwlimit.

Failure modes & how we'd debug it

1. Migration stalls at high percentage and never converges

Symptom: Task log shows memory transfer reaching ~90% then bouncing, never completing. The VM is a busy PostgreSQL or Redis box dirtying RAM faster than the link copies it.

Inspect the task and live progress:

# find the running migration task
qm list
# tail the migration task log directly
tail -f /var/log/pve/tasks/active
# or via the UPID
cat /var/log/pve/tasks/$(ls -t /var/log/pve/tasks/index | head)/...

# directly ask QEMU for migration stats on the source node
qm monitor 101
# then at the (qemu) prompt:
(qemu) info migrate
# look at: dirty pages rate, throughput, total time, remaining ram

If dirty pages rate (in pages/sec × 4KiB) exceeds your effective throughput, pre-copy will never converge. Two fixes:

a) Enable auto-converge / increase downtime tolerance. Proxmox sets a default max downtime of 100ms. Raising it lets the final stop-copy window absorb more dirty pages:

# allow up to 500ms stop-the-world window for this migration
qm monitor 101
(qemu) migrate_set_parameter downtime-limit 500
(qemu) migrate_set_parameter max-bandwidth 1250000000   # ~10 Gbit in bytes/s
(qemu) migrate_set_capability auto-converge on

auto-converge throttles the guest vCPUs to slow the dirty rate — a transparent ~5-20% CPU hit during the final phase, far better than a failed migration.

b) Migrate during a quieter window, or for a stubborn DB, accept a short offline migration (sub-second on shared storage with qm stop/qm start is not live; use qm migrate --online 0 only if you can tolerate the downtime).

2. "no such logical volume" / disk not found on destination

Symptom: Migration fails immediately with a storage error. Root cause: the VM disk lives on storage that isn't actually available on the destination node, or storage nodes= restrictions exclude the target.

# check what storages exist and which nodes they're bound to
pvesm status
cat /etc/pve/storage.cfg

# example output you DON'T want for migration:
# zfspool: local-zfs
#         pool rpool/data
#         content images,rootdir
#         nodes node-a          <-- restricted to node-a only!

If it says nodes node-a, that pool is single-node by config and the only path is --with-local-disks (block copy) or no migration at all. Check the VM's actual disk reference:

qm config 101 | grep -E 'scsi|virtio|sata|ide'
# scsi0: local-zfs:vm-101-disk-0,size=64G   <-- local, will need block copy

To move a VM off local ZFS to another node:

qm migrate 101 node-b --online --with-local-disks --targetstorage local-zfs
# requires ZFS pool of the same name on node-b; Proxmox uses zfs send/recv

3. VM won't migrate at all — passthrough or CPU mismatch

Symptom: can't migrate VM which uses local devices or guest kernel panic / unsupported-instruction fault right after resume on the destination.

# check for migration-blocking devices
qm config 101 | grep -E 'hostpci|usb|args'
# hostpci0: 0000:81:00.0   <-- GPU/NIC passthrough: live migration impossible

# check CPU model
qm config 101 | grep '^cpu'
# cpu: host   <-- only safe if all nodes share the exact CPU

Fix for the CPU case across a mixed cluster — set the lowest common model and reboot the guest in a window:

qm set 101 --cpu x86-64-v2-AES

For passthrough VMs, the only option is offline migration:

qm stop 101
qm migrate 101 node-b
# re-attach equivalent host device on node-b BEFORE start if PCI addr differs
qm start 101

4. Migration succeeds but VM is sluggish / corosync nearly fenced

Symptom: Migration ran on the corosync network and saturated it; you see retransmit warnings and almost lost quorum.

Check corosync health and where migration traffic went:

journalctl -u corosync --since "10 min ago" | grep -iE 'retransmit|token|fault'
# Look for: "Retransmit List:" or "FAILED TO RECEIVE"
corosync-cfgtool -s          # ring status / faults

The fix is structural — never share corosync and migration on one link. Confirm migration: ...network= points to a non-corosync subnet, and rate-limit with bwlimit. This is exactly why we set bwlimit: migration=600 above.

Observability: know before it hurts

Proxmox exposes metrics natively. Point PVE at an InfluxDB/Graphite endpoint or scrape the pve-exporter for Prometheus. A few alerts we actually run:

# prometheus rules: proxmox-migration.rules.yml
groups:
- name: proxmox.migration
  rules:
  - alert: ProxmoxNodeMemoryPressure
    # high mem usage makes evacuation slow & risky; act before you must
    expr: (pve_memory_usage_bytes / pve_memory_size_bytes) > 0.85
    for: 10m
    labels: {severity: warning}
    annotations:
      summary: "Node {{ $labels.id }} mem >85% — evacuation will be slow"

  - alert: ProxmoxClusterQuorumLost
    expr: pve_node_info * on(instance) group_left pve_quorate == 0
    for: 1m
    labels: {severity: critical}
    annotations:
      summary: "Proxmox cluster lost quorum — migrations & HA frozen"

A Grafana panel for migration-link saturation during an evacuation (Prometheus, node_exporter on the migration NIC):

{
  "title": "Migration NIC throughput (10.10.50.x)",
  "type": "timeseries",
  "targets": [
    {
      "expr": "rate(node_network_transmit_bytes_total{device=\"eno2\"}[1m]) * 8",
      "legendFormat": "tx bits/s {{instance}}"
    }
  ],
  "fieldConfig": { "defaults": { "unit": "bps" } }
}

For the security/audit angle, Wazuh on each PVE node catches who triggered migrations and any SSH brute force against the migration network (which on insecure mode is otherwise an open TCP socket — keep it on an isolated VLAN, full stop):

<!-- /var/ossec/etc/rules/local_rules.xml -->
<group name="proxmox,">
  <rule id="100210" level="3">
    <if_sid>5715</if_sid>
    <match>pvedaemon</match>
    <description>Proxmox: pvedaemon authenticated action</description>
  </rule>
  <rule id="100211" level="10" frequency="6" timeframe="120">
    <if_matched_sid>5710</if_matched_sid>
    <dstip>10.10.50.0/24</dstip>
    <description>Proxmox: repeated SSH failures on migration VLAN</description>
  </rule>
</group>

RUNBOOK: Evacuate a Proxmox node for maintenance

Use this when you must drain node-a (failing disk, RAM errors, kernel upgrade) with minimal downtime. Assumes shared storage (Ceph/NFS) or matching local ZFS pools, and a healthy quorum.

Confirm cluster health and quorum first. Never start an evacuation on a degraded cluster.

pvecm status            # expect "Quorate: Yes"
ha-manager status       # check HA resource states
ceph -s 2>/dev/null     # if using Ceph: HEALTH_OK, no recovery in flight

Inventory what's on the node and flag blockers.

qm list                                  # all VMs on this node
for v in $(qm list | awk 'NR>1{print $1}'); do
  echo "== VM $v =="
  qm config $v | grep -E 'hostpci|usb|^cpu|^scsi|^virtio'
done

Note any hostpci/usb VMs — those need an offline window.

Set the node into HA maintenance mode (drains HA-managed VMs automatically, respects HA groups):
```
ha-manager crm-command node-maintenance enable node-a
```

Cap migration bandwidth so storage/Ceph recovery isn't starved (temporary, datacenter-wide is fine):

# verify current setting
grep migration /etc/pve/datacenter.cfg
# if needed, per-migration override is used in step 5

Live-migrate non-HA VMs explicitly, one at a time, watching each:

# shared storage:
qm migrate 101 node-b --online --bwlimit 600
# local ZFS, same pool name on target:
qm migrate 102 node-b --online --with-local-disks --targetstorage local-zfs --bwlimit 400

# watch progress
tail -f /var/log/pve/tasks/active

For passthrough/blocked VMs, schedule the offline move (do these last, in the agreed window):

qm shutdown 105 --timeout 120     # graceful guest shutdown
qm migrate 105 node-b
# verify host device mapping on node-b, then:
qm start 105

Confirm the node is empty.
```
qm list           # should show no running VMs
pct list          # don't forget LXC containers!
```
Containers (pct) migrate with pct migrate <id> node-b --restart — LXC live migration is not supported; it's a stop/restart.
Do the maintenance (replace disk, apt dist-upgrade, reboot).

Bring the node back and re-balance.

ha-manager crm-command node-maintenance disable node-a
pvecm status              # confirm rejoined & quorate
# migrate VMs back if you pin workloads to specific nodes

Post-checks: confirm Ceph HEALTH_OK, no corosync retransmits, Grafana migration NIC back to baseline, no new Wazuh 100211 alerts.

Pre-flight checklist

Run through this before any planned evacuation or as a quarterly readiness audit.

Concrete sizing & targets we hold ourselves to

Migration link: dedicated 10GbE minimum for any node carrying >8 VMs; 25GbE if VMs are RAM-heavy (DBs, in-memory caches). On insecure 25GbE we sustain ~18 Gbit/s; on secure (SSH) we plan for ~1.5 Gbit/s and size evacuation time accordingly.
Per-VM live migration of a quiet 8GiB VM on shared storage: ~6–10 seconds end-to-end, with a sub-100ms stop window.
Full node evacuation (16 VMs, ~256GiB RAM total, 10GbE, bwlimit 600 MiB/s): budget ~12–18 minutes serialized, longer if VMs are write-heavy.
Stop-copy downtime target: 100ms default; we allow up to 500ms via downtime-limit for busy VMs before falling back to auto-converge.
RTO for an unplanned node failure with HA enabled: ~2 minutes (HA fence + restart on surviving node) — note this is a restart, not a live migration; the VM reboots. Live migration only protects you for planned maintenance.

The takeaway

Live migration is not magic; it's an iterative memory copy gated by your network and your guest's write rate, sitting on top of strict storage and CPU preconditions. The teams that get burned are the ones who never tested an evacuation until they needed one. Put migration on its own network, cap its bandwidth, standardize your CPU model, know which VMs can't move, and rehearse the runbook on a non-critical node before the disk warning light comes on. The difference between a 15-minute planned drain and a 3-hour incident is almost entirely preparation.

Struggling with Bare Metal & VM Management?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Bare Metal & VM Management, talk to the SysRoot team and we'll help you get it under control.

Back to Blog