Ephemeral Self-Hosted CI Runners on Proxmox: Stop Trusting Persistent Runners - SysRoot

How to run isolated, single-use GitHub Actions runners on Proxmox VMs so a poisoned build can't leak secrets or corrupt the next job.

Ephemeral Self-Hosted CI Runners on Proxmox

Most small teams that outgrow GitHub-hosted runners do the same thing: spin up one fat VM, install the runner as a service, and point every repo at it. It works for about three months. Then someone notices that a node_modules from a feature branch leaked into a main build, or a job left a docker login credential in ~/.docker/config.json that the next job happily reused, or /tmp filled up and builds started failing at 2 a.m.

Persistent self-hosted runners are a standing security and reliability liability. The runner process runs untrusted code from every PR. If you reuse the machine, you reuse whatever the last job left behind — files, env vars, kernel modules, cached layers, mounted secrets.

This article covers the pattern we deploy for clients who want self-hosted runner economics without the persistent-state footgun: ephemeral, single-use runner VMs on Proxmox, registered just-in-time, destroyed after one job, and watched by Wazuh. No reused filesystem. No reused credentials. One job, one VM, then qm destroy.

Why ephemeral, and what "ephemeral" actually means

GitHub's runner supports a --ephemeral flag. A runner started with it accepts exactly one job, then deregisters and exits. That's necessary but not sufficient — the process exits, but the VM is still there with all the dirty state. True isolation means the machine is also disposable.

Our target properties:

Blast radius = one job. A malicious PR build cannot read the next build's secrets or write to its workspace.
No persistent runner registration tokens on disk beyond a single job's lifetime.
RTO for "all runners gone" < 5 minutes — the orchestrator just provisions new VMs from a template.
Cold-to-ready provisioning < 60s using a pre-baked Proxmox template plus cloud-init.

                      GitHub Actions
                    (workflow_job webhook)
                            │
                            ▼
                 ┌────────────────────────┐
                 │  runner-orchestrator    │  (small VM / systemd timer)
                 │  - listens for "queued" │
                 │  - calls Proxmox API    │
                 └───────────┬─────────────┘
                             │ qm clone + cloud-init
                             ▼
        Proxmox node ┌───────────────────────────────┐
                     │  template 9000 (linked clone)  │
                     │   ├─ vmid 5001  job A  ──┐      │
                     │   ├─ vmid 5002  job B    │      │
                     │   └─ vmid 5003  job C    │      │
                     └──────────────────────────┼──────┘
                             one job each       │
                             then qm destroy ◄──┘
                             │
                             ▼  (logs/auditd)
                          Wazuh agent → Wazuh manager

The golden template

Everything starts from one Proxmox template. Build it once, keep it minimal, never let a job mutate it. We use a cloud-image base (Debian 12 here) and qm to create it.

# On the Proxmox node — create the runner template VM (vmid 9000)
cd /var/lib/vz/template/iso
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2

qm create 9000 \
  --name ci-runner-template \
  --memory 4096 --cores 2 \
  --net0 virtio,bridge=vmbr1 \
  --scsihw virtio-scsi-single \
  --ostype l26 --agent enabled=1

qm importdisk 9000 debian-12-genericcloud-amd64.qcow2 local-zfs
qm set 9000 --scsi0 local-zfs:vm-9000-disk-0
qm set 9000 --ide2 local-zfs:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --serial0 socket --vga serial0

# Resize root so Docker and a build fit (32 GB)
qm resize 9000 scsi0 32G

Before converting VM 9000 into a template, boot it once and install the prerequisites that every clone needs. The following is an example provisioning sequence to run inside that VM. Stage a reviewed GitHub Actions runner archive and Wazuh agent package from your approved artifact source, then set RUNNER_ARCHIVE and WAZUH_AGENT_DEB to their local paths.

# Inside VM 9000 during its one-time provisioning boot
set -euo pipefail

RUNNER_ARCHIVE=/tmp/actions-runner-linux-x64.tar.gz
WAZUH_AGENT_DEB=/tmp/wazuh-agent.deb

apt-get update
apt-get install -y \
  auditd \
  ca-certificates \
  curl \
  docker.io \
  jq \
  qemu-guest-agent

systemctl enable docker qemu-guest-agent auditd

# Create the account and directory used by runner-oneshot.service.
id runner >/dev/null 2>&1 || useradd --create-home --shell /bin/bash runner
usermod -aG docker runner
install -d -o runner -g runner /home/runner/actions-runner

# Install the reviewed runner archive without configuring or registering it.
tar -xzf "$RUNNER_ARCHIVE" -C /home/runner/actions-runner
chown -R runner:runner /home/runner/actions-runner
(
  cd /home/runner/actions-runner
  ./bin/installdependencies.sh
)

# Install the agent only. Manager details and enrollment happen per clone.
apt-get install -y "$WAZUH_AGENT_DEB"
systemctl disable wazuh-agent || true

Do not run the runner's config.sh while building the template, and do not place a GitHub PAT, runner registration token, Wazuh enrollment credential, or reusable SSH key in the image. The runner is registered just in time by each clone. The Wazuh agent should likewise be enrolled at clone boot using environment-specific manager details and enrollment material rather than credentials stored in the template.

Verify the image before shutting it down:

set -euo pipefail

test -x /home/runner/actions-runner/config.sh
test -x /home/runner/actions-runner/run.sh
docker --version
auditctl -s
systemctl is-enabled docker
systemctl is-enabled qemu-guest-agent
dpkg-query -W wazuh-agent

# The runner must still be unconfigured in the reusable image.
test ! -f /home/runner/actions-runner/.runner
test ! -f /home/runner/actions-runner/.credentials
test ! -f /home/runner/actions-runner/.credentials_rsaparams

Clean one-time provisioning state, shut down the VM, and only then convert VM 9000 into the immutable template:

# Inside VM 9000
apt-get clean
cloud-init clean --logs
shutdown -h now

# Back on the Proxmox node after VM 9000 has stopped
qm status 9000
qm template 9000

The resulting template contains the runner binary, Docker, auditd, the QEMU guest agent, and the Wazuh agent, but no runner registration. Key point: no registration token lives in the template. Tokens are injected per clone.

The per-job cloud-init user-data is generated dynamically by the orchestrator:

#cloud-config
hostname: ci-runner-${VMID}
manage_etc_hosts: true
users:
  - name: runner
    groups: [docker]
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    shell: /bin/bash
write_files:
  - path: /etc/systemd/system/runner-oneshot.service
    permissions: '0644'
    content: |
      [Unit]
      Description=Ephemeral GitHub Actions runner
      After=network-online.target docker.service
      Wants=network-online.target
      [Service]
      Type=oneshot
      User=runner
      WorkingDirectory=/home/runner/actions-runner
      ExecStart=/home/runner/actions-runner/run-once.sh
      # The whole VM is killed after the runner exits
      ExecStopPost=/usr/bin/systemctl poweroff
      [Install]
      WantedBy=multi-user.target
  - path: /home/runner/actions-runner/run-once.sh
    permissions: '0755'
    owner: runner:runner
    content: |
      #!/usr/bin/env bash
      set -euo pipefail
      cd /home/runner/actions-runner
      ./config.sh \
        --url https://github.com/${GH_ORG} \
        --token ${RUNNER_TOKEN} \
        --name ci-runner-${VMID} \
        --labels proxmox,ephemeral,linux,x64 \
        --ephemeral --unattended --disableupdate
      # --ephemeral => accept exactly one job, then exit
      ./run.sh
runcmd:
  - [ systemctl, enable, --now, runner-oneshot.service ]

The ${RUNNER_TOKEN} is a registration token with a ~1 hour TTL fetched from the GitHub API at clone time, not a PAT. Even if a job exfiltrates it, it's single-use and short-lived.

On the Proxmox node — build the runner template (vmid 9000)

cd /var/lib/vz/template/iso wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2

qm create 9000
--name ci-runner-template
--memory 4096 --cores 2
--net0 virtio,bridge=vmbr1
--scsihw virtio-scsi-single
--ostype l26 --agent enabled=1

qm importdisk 9000 debian-12-genericcloud-amd64.qcow2 local-zfs qm set 9000 --scsi0 local-zfs:vm-9000-disk-0 qm set 9000 --ide2 local-zfs:cloudinit qm set 9000 --boot order=scsi0 qm set 9000 --serial0 socket --vga serial0

Resize root so docker + a build fit (32 GB)

qm resize 9000 scsi0 32G

qm template 9000


The runner binary, Docker, and the Wazuh agent get baked into the template via a one-time provisioning boot. We do that with cloud-init `runcmd` on a throwaway clone, then re-snapshot the template. Key point: **no registration token lives in the template.** Tokens are injected per-clone.

The per-job cloud-init user-data is generated dynamically by the orchestrator:

```yaml
#cloud-config
hostname: ci-runner-${VMID}
manage_etc_hosts: true
users:
  - name: runner
    groups: [docker]
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    shell: /bin/bash
write_files:
  - path: /etc/systemd/system/runner-oneshot.service
    permissions: '0644'
    content: |
      [Unit]
      Description=Ephemeral GitHub Actions runner
      After=network-online.target docker.service
      Wants=network-online.target
      [Service]
      Type=oneshot
      User=runner
      WorkingDirectory=/home/runner/actions-runner
      ExecStart=/home/runner/actions-runner/run-once.sh
      # The whole VM is killed after the runner exits
      ExecStopPost=/usr/bin/systemctl poweroff
      [Install]
      WantedBy=multi-user.target
  - path: /home/runner/actions-runner/run-once.sh
    permissions: '0755'
    owner: runner:runner
    content: |
      #!/usr/bin/env bash
      set -euo pipefail
      cd /home/runner/actions-runner
      ./config.sh \
        --url https://github.com/${GH_ORG} \
        --token ${RUNNER_TOKEN} \
        --name ci-runner-${VMID} \
        --labels proxmox,ephemeral,linux,x64 \
        --ephemeral --unattended --disableupdate
      # --ephemeral => accept exactly one job, then exit
      ./run.sh
runcmd:
  - [ systemctl, enable, --now, runner-oneshot.service ]

The ${RUNNER_TOKEN} is a registration token with a ~1 hour TTL fetched from the GitHub API at clone time, not a PAT. Even if a job exfiltrates it, it's single-use and short-lived.

The orchestrator

You can use the official actions-runner-controller on Kubernetes, but for teams running Proxmox and not a full K8s platform, a small webhook listener that drives the Proxmox API is simpler and has fewer moving parts. Here's the core provisioning logic, triggered on the workflow_job queued event.

#!/usr/bin/env bash
# provision-runner.sh — called by the webhook handler on workflow_job=queued
set -euo pipefail

GH_ORG="acme-co"
PVE_NODE="pve01"
TEMPLATE=9000
NET_BRIDGE="vmbr1"

# 1. Pick a free vmid in the ephemeral range
VMID=$(pvesh get /cluster/nextid)
[[ "$VMID" -lt 5000 || "$VMID" -gt 5999 ]] && VMID=$(( 5000 + RANDOM % 1000 ))

# 2. Get a single-use runner registration token (TTL ~1h)
RUNNER_TOKEN=$(curl -sS -X POST \
  -H "Authorization: Bearer ${GH_APP_TOKEN}" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/orgs/${GH_ORG}/actions/runners/registration-token" \
  | jq -r .token)

# 3. Render cloud-init user-data
export VMID GH_ORG RUNNER_TOKEN
envsubst < user-data.tpl.yaml > "/var/lib/vz/snippets/runner-${VMID}.yaml"

# 4. Linked clone from template (fast, copy-on-write on ZFS)
qm clone "$TEMPLATE" "$VMID" \
  --name "ci-runner-${VMID}" \
  --pool ci-runners

qm set "$VMID" --cicustom "user=local:snippets/runner-${VMID}.yaml"
qm set "$VMID" --ipconfig0 "ip=dhcp"
qm set "$VMID" --tags "ephemeral,ci,${GH_ORG}"

# 5. Boot
qm start "$VMID"
echo "Provisioned runner VM ${VMID}"

Tear-down is just as important. The runner deregisters itself on --ephemeral exit, and the VM powers itself off via ExecStopPost. A reaper sweeps stopped ephemeral VMs every minute so dead state never accumulates:

#!/usr/bin/env bash
# reap-runners.sh — run from a systemd timer every 60s
set -euo pipefail
for VMID in $(qm list | awk '$3=="stopped" && $2 ~ /^ci-runner-/ {print $1}'); do
  # Only reap VMs in the ephemeral range with the ephemeral tag
  if qm config "$VMID" | grep -q 'tags:.*ephemeral'; then
    echo "Reaping stopped runner ${VMID}"
    qm destroy "$VMID" --purge --destroy-unreferenced-disks 1
    rm -f "/var/lib/vz/snippets/runner-${VMID}.yaml"
  fi
done

# Safety net: kill any ephemeral VM running longer than 90 min (stuck job)
for VMID in $(qm list | awk '$3=="running" && $2 ~ /^ci-runner-/ {print $1}'); do
  UPTIME=$(qm guest cmd "$VMID" get-time 2>/dev/null || echo 0)
  # Simpler: track start time from a sidecar file written at provision
  START=$(stat -c %Y "/var/lib/vz/snippets/runner-${VMID}.yaml" 2>/dev/null || echo 0)
  AGE=$(( $(date +%s) - START ))
  if [[ "$AGE" -gt 5400 ]]; then
    echo "Force-reaping stuck runner ${VMID} (age ${AGE}s)"
    qm stop "$VMID" && sleep 5 && qm destroy "$VMID" --purge
  fi
done

Capacity and sizing numbers

This is where ephemeral runners scare people: "won't I spawn 200 VMs?" You cap it. Concrete sizing we use on a single Proxmox node with 32 cores / 128 GB:

Runner VM: 2 vCPU, 4 GB RAM, 32 GB linked-clone disk. Linked clones on ZFS only consume delta blocks; a freshly-booted runner uses ~1.5 GB on disk.
Max 12 concurrent runners per node (24 vCPU committed, 48 GB RAM, leaving headroom for Proxmox + the orchestrator). The webhook handler refuses to provision beyond that and lets the job queue in GitHub.
Provision-to-ready: 35–55s with a warm ZFS ARC and a DHCP lease ready. Cold first boot after a host reboot: ~90s.
Reaper interval: 60s. Stuck-job hard timeout: 90 min.

Wazuh: watching what the untrusted code does

Because each VM runs untrusted PR code, you want auditd + Wazuh on it, shipping to your manager. Bake the agent into the template and enroll via the agent-auth flow at boot. The high-value signal: a build process touching things a build has no business touching — outbound connections to weird IPs, reads of /etc/shadow, or attempts to escape the workspace.

<!-- /var/ossec/etc/rules/local_rules.xml on the Wazuh manager -->
<group name="ci_runner,">

  <!-- Detect a CI job reading sensitive files -->
  <rule id="100210" level="10">
    <if_group>syscheck,audit</if_group>
    <field name="audit.file.name">/etc/shadow|/root/.ssh/|/var/ossec/etc/client.keys</field>
    <description>CI runner process accessed a sensitive file: $(audit.file.name)</description>
    <group>ci_runner_suspicious,pci_dss_10.2.1,</group>
  </rule>

  <!-- Outbound connection from runner to non-allowlisted host -->
  <rule id="100211" level="8">
    <if_sid>2502</if_sid>
    <hostname>ci-runner-</hostname>
    <description>Unexpected outbound network activity from ephemeral CI runner</description>
    <group>ci_runner_network,</group>
  </rule>

  <!-- Runner VM lived longer than expected (correlation with reaper logs) -->
  <rule id="100212" level="7">
    <decoded_as>json</decoded_as>
    <field name="event">runner_force_reaped</field>
    <description>Ephemeral runner force-reaped after exceeding max lifetime</description>
  </rule>

</group>

Auditd rules in the template that feed those Wazuh rules:

# /etc/audit/rules.d/ci.rules
-w /etc/shadow -p r -k sensitive_read
-w /root/.ssh -p r -k sensitive_read
-w /var/ossec/etc/client.keys -p r -k sensitive_read
-a always,exit -F arch=b64 -S connect -F a2=16 -k outbound_conn

A level-10 alert on 100210 from a runner means a PR build tried to read /etc/shadow. That is never a legitimate build action — page someone.

Runbook: a poisoned PR is suspected of exfiltrating from a runner

Use this when Wazuh fires 100210/100211 or someone reports a leaked secret from CI.

Identify the live VM and freeze it (don't destroy yet — you want forensics).

# From the Wazuh alert, get the hostname e.g. ci-runner-5042
ssh pve01
VMID=5042
qm config $VMID | grep -E 'name|tags'
# Suspend, don't stop — keeps RAM state for inspection
qm suspend $VMID

Pull the exact job that ran on it. The runner name maps 1:1 to the job.

curl -sS -H "Authorization: Bearer ${GH_APP_TOKEN}" \
  "https://api.github.com/orgs/acme-co/actions/runners" \
  | jq '.runners[] | select(.name=="ci-runner-5042")'

Find the workflow run and the PR/commit. Note the author and the head SHA.

Snapshot the disk for offline analysis.

qm snapshot $VMID forensics-$(date +%s)
zfs list -t snapshot | grep vm-${VMID}
# Mount read-only on the host for inspection

Rotate any secret that runner could have seen. Assume total compromise of everything in that job's environment. Pull the secret list:
```
gh api repos/acme-co/$REPO/actions/secrets | jq '.secrets[].name'
```
Rotate each at its source (see secrets-rotation runbook), then update via gh secret set.

Confirm the registration token can't be reused. Ephemeral tokens are single-use and expire in ~1h, but revoke org-wide to be safe:

# Remove the runner registration if it lingered
curl -sS -X DELETE -H "Authorization: Bearer ${GH_APP_TOKEN}" \
  "https://api.github.com/orgs/acme-co/actions/runners/${RUNNER_ID}"

Quarantine the offending PR. Disable runner auto-provisioning for that repo until reviewed:

# Flag in orchestrator config; webhook handler will skip provisioning
echo "$REPO" >> /etc/runner-orchestrator/quarantine.list
systemctl reload runner-orchestrator

Destroy after forensics captured.

qm destroy $VMID --purge
rm -f /var/lib/vz/snippets/runner-${VMID}.yaml

Post-incident: confirm no other VMs from the same template were modified (they're linked clones — the template is read-only, verify its ZFS dataset has no writes).

Failure modes & how we'd debug it

1. Runners provision but never pick up jobs

Symptom: VMs boot, GitHub shows "Idle" runners, but the job stays queued.

# On the runner VM
journalctl -u runner-oneshot.service --no-pager | tail -50
# Common: label mismatch. The workflow asks for runs-on: [self-hosted, gpu]
# but the runner only has labels: proxmox,ephemeral,linux,x64

Check the workflow's runs-on against the --labels in run-once.sh. Also verify the registration token wasn't already expired at boot:

grep -i 'token' /home/runner/actions-runner/_diag/*.log | tail
# "Response status code does not indicate success: 404 (Not Found)"
# => token expired between clone and boot. Reduce provision-to-boot lag.

2. Disk fills on the Proxmox node — clones fail

Symptom: qm clone returns unable to create image: zfs error.

zpool list                       # check CAP%
zfs list -o name,used,refer,usedbysnapshots | sort -k4 -h | tail
# Forensic snapshots or orphaned clones accumulating?
qm list | awk '$2 ~ /^ci-runner-/'      # count live runners

Almost always: the reaper timer died, so stopped VMs aren't being destroyed. Verify:

systemctl status reap-runners.timer
systemctl list-timers | grep reap
journalctl -u reap-runners.service --since "1 hour ago"

If stopped runners are piling up, run the reaper manually and re-enable the timer. Also check for orphaned forensics-* ZFS snapshots from old incidents.

3. A job hangs forever and pins capacity

Symptom: GitHub shows a runner "Active" for 40+ minutes; new jobs queue because you've hit the 12-runner cap.

# Find long-running ephemeral VMs
ssh pve01 'for v in $(qm list | awk "\$2 ~ /^ci-runner-/ {print \$1}"); do
  echo -n "$v age: "; echo $(( $(date +%s) - $(stat -c %Y /var/lib/vz/snippets/runner-$v.yaml) ))s
done'

Check whether it's a legitimately slow job or a deadlocked one. Inspect via the guest agent:

qm guest exec 5042 -- ps aux --sort=-%cpu | head

The 90-min hard cap in the reaper will eventually kill it, but if it's blocking the queue, force-reap now and let GitHub re-queue the job. This is exactly why ephemeral + a hard lifetime cap beats persistent runners: a wedged build can't poison the next one, it just costs one VM.

4. Wazuh agent on the runner never reports

Symptom: no events from ci-runner-* even though VMs are running.

# On the runner
systemctl status wazuh-agent
/var/ossec/bin/agent_control -i $(cat /var/ossec/etc/client.keys | awk '{print $1}')
tail -f /var/ossec/logs/ossec.log
# "Unable to connect to manager" => template baked a stale manager IP,
# or agent-auth didn't run at boot because of a DHCP race.

Because runners are short-lived, an agent that takes 30s to enroll may miss the whole job. Pre-enroll with agent-auth in cloud-init runcmd before systemctl --now enable runner-oneshot.service, and gate the runner start on the agent being connected.

Pre-flight & hardening checklist

The trade-off, stated plainly

Ephemeral runners cost you ~45 seconds of provisioning latency per job and a bit of orchestration code. In return, the security model becomes trivial to reason about: one job, one machine, then gone. No shared caches to poison, no leftover credentials, no /tmp accumulation, no "which previous build broke this runner." For a team without a dedicated platform group, that simplicity is worth far more than the saved seconds — debugging a persistent-runner state leak at 2 a.m. costs a lot more than 45 seconds.

If your runners are currently persistent VMs that "just work," they work right up until an untrusted PR teaches you why they shouldn't.

Struggling with Managed DevOps?

You don't have to solve it alone. SysRoot provides senior DevOps and infrastructure expertise without the cost of building a full in-house team. If your team is wrestling with Managed DevOps, talk to the SysRoot team and we'll help you get it under control.

Back to Blog