CitrixPot: NetScaler / Citrix ADC Gateway honeypot (initial public release)

deploy/VM-ACCESS.md

@@ -0,0 +1,58 @@
+# Deployment notes (template)
+
+Replace every `<...>` placeholder with your own values. Keep real addresses and keys out of
+the repository; put them in `/etc/decoy.env` (see `.env.example`) or your own secret store.
+
+## Topology
+
+- Proxmox host: `<PROXMOX_HOST>` (management network, not public).
+- Collector VM: stores evidence and serves the decoy. QEMU user networking only, no bridge NIC.
+- Execution VM (optional): disposable, isolated guest for real payload behaviour. Requires
+  `restrict=on`, no bridge NIC, no host mounts, and must never be enabled on a public listener.
+- Edge proxy (e.g. Nginx Proxy Manager on `<EDGE_PROXY_IP>`): terminates TLS and forwards to a
+  source-preserving ingress on the Proxmox host.
+
+## Source-preserving ingress
+
+`ingress_proxy.py` accepts a TCP connection and prepends a PROXY-v1 header (the real client
+address) before the decoy reads any bytes. Restrict who may reach it:
+
+```
+INGRESS_BIND=<PROXMOX_HOST>
+INGRESS_PORT=18443
+INGRESS_TARGET=127.0.0.1
+INGRESS_TARGET_PORT=19443
+INGRESS_ALLOW_FROM=<EDGE_PROXY_IP>,127.0.0.1,<PROXMOX_HOST>
+```
+
+At the decoy, trust only the edge hop for client-IP headers:
+
+```
+HONEYPOT_PROXY_PROTOCOL_FROM=<QEMU_GATEWAY>        # e.g. 10.0.2.2 for QEMU user networking
+HONEYPOT_TRUSTED_FORWARDED_FROM=<EDGE_PROXY_IP>,127.0.0.1
+```
+
+Add a host firewall rule so only the edge proxy and the host itself can reach the origin port.
+Example (adapt to your firewall tooling):
+
+```
+iptables -N DECOY-ORIGIN
+iptables -A DECOY-ORIGIN -i lo -j RETURN
+iptables -A DECOY-ORIGIN -s <EDGE_PROXY_IP> -j RETURN
+iptables -A DECOY-ORIGIN -s <PROXMOX_HOST> -j RETURN
+iptables -A DECOY-ORIGIN -j DROP
+iptables -I INPUT 1 -p tcp --dport 18443 -j DECOY-ORIGIN
+```
+
+## Management access
+
+Use your own SSH keys and a jump host. Do not enable agent/X11/port forwarding on the
+management key (`no-agent-forwarding,no-port-forwarding,no-X11-forwarding`).
+
+## Operational notes
+
+- QEMU `filter-dump` truncates its pcap on guest start; archive it before restarting the
+  execution VM.
+- The decoy is fail-closed: when the evidence quota or disk fails it stops rather than
+  overwriting. Archive evidence before restarting.
+- Never enable real command execution on an internet-facing listener.

deploy/cloudflare-ips.txt

@@ -0,0 +1,24 @@
+# Cloudflare published edge ranges (https://www.cloudflare.com/ips). Refresh periodically.
+# Only hops inside these networks may set CF-Connecting-IP / X-Forwarded-For for the decoy.
+173.245.48.0/20
+103.21.244.0/22
+103.22.200.0/22
+103.31.4.0/22
+141.101.64.0/18
+108.162.192.0/18
+190.93.240.0/20
+188.114.96.0/20
+197.234.240.0/22
+198.41.128.0/17
+162.158.0.0/15
+104.16.0.0/13
+104.24.0.0/14
+172.64.0.0/13
+131.0.72.0/22
+2400:cb00::/32
+2606:4700::/32
+2803:f800::/32
+2405:b500::/32
+2405:8100::/32
+2a06:98c0::/29
+2c0f:f248::/32

deploy/decoy-cleanup.cron

@@ -0,0 +1,5 @@
+# Hourly disposable VM223 reset. Evidence is archived, never automatically pruned.
+# Stop intake/VM and require operator review when archive budget/reserve is reached.
+SHELL=/bin/sh
+PATH=/usr/sbin:/usr/bin:/sbin:/bin
+0 * * * * root /usr/bin/python3 /opt/decoy-ingress/reset-execution.py --run >> /var/lib/vz/honeypot/reset-cron.log 2>&1

deploy/decoy-guest-network.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Redirect disposable guest TCP to recording relay
+After=network-online.target
+Before=decoy-relay.service
+[Service]
+Type=oneshot
+ExecStart=/bin/sh /opt/decoy/guest-network.sh
+RemainAfterExit=true
+[Install]
+WantedBy=multi-user.target

deploy/decoy-ingress.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Source-preserving ingress for the NetScaler decoy
+After=network-online.target
+[Service]
+EnvironmentFile=/etc/decoy.env
+DynamicUser=yes
+# --allow-from limits who may reach the origin to the edge proxy and trusted hops (see .env).
+ExecStart=/usr/bin/python3 /opt/decoy-ingress/ingress_proxy.py --bind=${INGRESS_BIND} --port=${INGRESS_PORT} --target=${INGRESS_TARGET} --target-port=${INGRESS_TARGET_PORT} --allow-from=${INGRESS_ALLOW_FROM}
+Restart=on-failure
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+MemoryMax=128M
+TasksMax=32
+[Install]
+WantedBy=multi-user.target

deploy/decoy-network.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=External recording and policy relay for disposable execution VM
+After=network-online.target
+[Service]
+ExecStart=/usr/bin/python3 /opt/decoy/network_capture.py --role broker --quota-mib 2048 --listen 0.0.0.0 --port 19090 --evidence /var/lib/decoy/network --peers /var/lib/decoy/allowed-peers.json
+WorkingDirectory=/opt/decoy
+UMask=0077
+Restart=no
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/var/lib/decoy
+ProtectHome=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+MemoryMax=384M
+TasksMax=128
+[Install]
+WantedBy=multi-user.target

deploy/decoy-relay.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Disposable guest transparent TCP and DNS relay
+After=network-online.target
+[Service]
+User=decoy-relay
+ExecStart=/usr/bin/python3 /opt/decoy/network_capture.py --role relay --listen 127.0.0.1 --port 15001 --broker 10.0.2.100 --broker-port 19090 --dns-port 53
+Restart=on-failure
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+RestrictAddressFamilies=AF_INET AF_UNIX
+MemoryMax=256M
+TasksMax=128
+[Install]
+WantedBy=multi-user.target

deploy/decoy-viewer.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=Local read-only honeypot evidence viewer (loopback only)
+After=network-online.target
+[Service]
+EnvironmentFile=/etc/decoy.env
+ExecStart=/usr/bin/python3 /opt/decoy/viewer.py --bind=${VIEWER_BIND} --port=${VIEWER_PORT} --log-dir=${VIEWER_LOG_DIR} --hide-ip=${VIEWER_HIDE_IP}
+WorkingDirectory=/opt/decoy
+UMask=0077
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadOnlyPaths=/var/lib/decoy
+RestrictAddressFamilies=AF_INET
+MemoryMax=256M
+TasksMax=32
+Restart=no
+[Install]
+WantedBy=multi-user.target

deploy/decoy-watchdog.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Stop disposable execution VM on capture failure or quota
+After=pve-guests.service
+[Service]
+ExecStart=/usr/bin/python3 /opt/decoy-ingress/watchdog.py
+Restart=always
+RestartSec=2
+UMask=0077
+[Install]
+WantedBy=multi-user.target

deploy/decoy.service

@@ -0,0 +1,25 @@
+[Unit]
+Description=NetScaler decoy HTTP and evidence collector
+After=network-online.target decoy-network.service
+Requires=decoy-network.service
+[Service]
+EnvironmentFile=/etc/decoy.env
+# Simulation only: attacker commands are parsed, stored and answered, never executed.
+# Real execution additionally requires --vm-config + --allow-real-execution and an isolated guest;
+# never enable it on an internet-facing listener. All values come from /etc/decoy.env.
+ExecStart=/usr/bin/python3 /opt/decoy/honeypot.py --bind=${HONEYPOT_BIND} --port=${HONEYPOT_PORT} --proxy-protocol-from=${HONEYPOT_PROXY_PROTOCOL_FROM} --hide-guest-files --trusted-forwarded-from=${HONEYPOT_TRUSTED_FORWARDED_FROM} --fetch-payloads --log-dir=${HONEYPOT_LOG_DIR} --evidence-mib=${HONEYPOT_EVIDENCE_MIB}
+WorkingDirectory=/opt/decoy
+UMask=0077
+Restart=no
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/var/lib/decoy
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+MemoryMax=768M
+TasksMax=128
+[Install]
+WantedBy=multi-user.target

deploy/guest-network.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+set -eu
+[ -f /etc/decoy-disposable-guest ]
+id decoy-relay >/dev/null 2>&1 || useradd --system --no-create-home --shell /usr/sbin/nologin decoy-relay
+ip route replace default via 10.0.2.2
+iptables -t nat -N DECOY 2>/dev/null || true
+iptables -t nat -F DECOY
+iptables -t nat -A DECOY -m owner --uid-owner decoy-relay -j RETURN
+iptables -t nat -A DECOY -d 127.0.0.0/8 -j RETURN
+iptables -t nat -A DECOY -d 10.0.2.100/32 -j RETURN
+iptables -t nat -A DECOY -p tcp -j REDIRECT --to-ports 15001
+iptables -t nat -C OUTPUT -p tcp -j DECOY 2>/dev/null || iptables -t nat -A OUTPUT -p tcp -j DECOY
+printf 'nameserver 127.0.0.1\noptions timeout:2 attempts:1\n' > /etc/resolv.conf

deploy/manual-test.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# Manual end-to-end probe of a deployed decoy. Read-only: every request is recorded as
+# evidence; nothing is executed. Configure via environment (see .env.example).
+#   BASE_URL   e.g. http://127.0.0.1:8443 or https://your-decoy.example
+#   HOST_HEADER  Host header the edge proxy routes on (default: localhost)
+set -u
+B=${1:-${BASE_URL:-http://127.0.0.1:8443}}
+H="Host: ${HOST_HEADER:-localhost}"
+printf '\000\377MZ-inert-manual-payload\r\n' > /tmp/manual-payload.bin
+printf -- '--testboundary\r\nContent-Disposition: form-data; name="file"; filename="../../manual-multipart.bin"\r\nContent-Type: application/octet-stream\r\n\r\nINERT-MULTIPART-BODY\r\n--testboundary--\r\n' > /tmp/manual-multipart.bin
+
+code() {
+  local label=$1; shift
+  printf '%-42s -> %s\n' "$label" "$(curl -sS --max-time 10 -o /dev/null -w '%{http_code} %{size_download}b' "$@")"
+}
+
+echo "== target $B =="
+code "GET / (login page)"        -H "$H" "$B/"
+code "GET /favicon.ico"          -H "$H" "$B/favicon.ico"
+code "GET background.jpg"        -H "$H" "$B/logon/LogonPoint/receiver/images/common/ReceiverFullScreenBackground.jpg"
+code "GET logo.png"              -H "$H" "$B/vpn/media/citrixgateway_logo_white.png"
+code "GET login.js"              -H "$H" "$B/logon/LogonPoint/login.js"
+code "HEAD /"                    -I -H "$H" "$B/"
+code "GET /nonexistent"          -H "$H" "$B/nonexistent"
+code "GET path traversal"        --path-as-is -H "$H" "$B/../../etc/passwd"
+code "OPTIONS /"                 -X OPTIONS -H "$H" "$B/"
+code "POST auth form"            -H "$H" -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=demo&passwd=demo' "$B/nf/auth/doAuthentication.do"
+code "PUT binary payload"        -X PUT -H "$H" -H 'Content-Type: application/octet-stream' --data-binary @/tmp/manual-payload.bin "$B/sample.bin"
+code "POST multipart upload"     -H "$H" -H 'Content-Type: multipart/form-data; boundary=testboundary' --data-binary @/tmp/manual-multipart.bin "$B/upload"
+code "POST oversize body"        -X POST -H "$H" -H 'Content-Length: 999999999' --max-time 4 "$B/"
+code "POST ambiguous framing"    -X POST -H "$H" -H 'Transfer-Encoding: chunked' -H 'Content-Length: 5' --data '0' "$B/"
+code "GET injection in User-Agent" -H "$H" -A 'pitboss PPE unexpectedly died NSPPE;:`id>/netscaler/ns_gui/manual-test.txt`;# X' "$B/"
+
+echo -n "auth XML body: "
+curl -sS --max-time 8 -H "$H" -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=x&passwd=y' "$B/nf/auth/doAuthentication.do" | head -c 160
+echo
+l=0; while [ $l -lt 12 ]; do l=$((l+1)); curl -sS --max-time 5 -o /dev/null -H "$H" --data 'login=a&passwd=b' "$B/nf/auth/doAuthentication.do"; done
+echo "sent 12 rapid auth attempts (brute-force threshold is 10)"
+python3 - "$B" "${HOST_HEADER:-localhost}" <<'PY'
+import socket, sys, urllib.parse
+url = urllib.parse.urlsplit(sys.argv[1])
+host = sys.argv[2]
+s = socket.create_connection((url.hostname, url.port or 80), timeout=8)
+s.sendall(f'GET / HTTP/1.1\r\nHost: {host}\r\n\r\n'.encode()
+          + f'GET /favicon.ico HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\n\r\n'.encode())
+data = b''
+while True:
+    chunk = s.recv(65536)
+    if not chunk:
+        break
+    data += chunk
+print(f'keep-alive pipelined -> {data.count(b"HTTP/1.1 ")} responses, {len(data)} bytes')
+PY
+echo "done"

deploy/provision-pair.sh

@@ -0,0 +1,63 @@
+#!/bin/bash
+# Run on the designated Proxmox host. Preparation only; never execute payloads here.
+set -euo pipefail
+cd /var/lib/vz/honeypot
+expected=$(awk '$2 == "debian-12-genericcloud-amd64.qcow2" {print $1}' SHA512SUMS)
+[[ -n "$expected" ]]
+printf '%s  debian.qcow2\n' "$expected" | sha512sum --check -
+[[ -s admin.pub ]]
+command -v nc >/dev/null
+printf '#!/bin/sh\nexec nc 127.0.0.1 19223\n' > forward-execution-ssh
+printf '#!/bin/sh\nexec nc 127.0.0.1 19090\n' > forward-collector-telemetry
+chmod 700 forward-execution-ssh forward-collector-telemetry
+for vmid in 222 223; do
+  if qm config "$vmid" >/dev/null 2>&1; then
+    echo "VM $vmid exists; refusing replacement" >&2; exit 1
+  fi
+done
+for vmid in 222 223; do
+  role=collector; mem=2048; port=19222; mac=52:54:00:22:02:22
+  packages='[python3, openssh-client, openssh-server, iptables, qemu-guest-agent]'
+  args="-netdev user,id=decoy,hostfwd=tcp:127.0.0.1:19443-:8443,hostfwd=tcp:127.0.0.1:19222-:22,hostfwd=tcp:127.0.0.1:19090-:19090,guestfwd=tcp:10.0.2.101:22-cmd:/var/lib/vz/honeypot/forward-execution-ssh -device virtio-net-pci,netdev=decoy,mac=$mac"
+  if [[ $vmid = 223 ]]; then
+    role=execution; mem=4096; port=19223; mac=52:54:00:22:02:23
+    packages='[python3, strace, curl, wget, netcat-openbsd, iptables, openssh-server, qemu-guest-agent]'
+    # Trusted package bootstrap only. restrict=on MUST replace this before payloads.
+    args="-netdev user,id=decoy,hostfwd=tcp:127.0.0.1:19223-:22,guestfwd=tcp:10.0.2.100:19090-cmd:/var/lib/vz/honeypot/forward-collector-telemetry -device virtio-net-pci,netdev=decoy,mac=$mac"
+  fi
+  mkdir -p "seed-$vmid" /var/lib/vz/template/iso
+  cat > "seed-$vmid/user-data" <<CLOUD
+#cloud-config
+hostname: decoy-$role
+manage_etc_hosts: true
+disable_root: false
+ssh_pwauth: false
+users:
+  - name: root
+    lock_passwd: true
+    ssh_authorized_keys:
+      - $(cat admin.pub)
+package_update: true
+packages: $packages
+runcmd:
+  - [mkdir, -p, /opt/decoy, /netscaler/ns_gui, /nsconfig]
+  - [systemctl, enable, --now, qemu-guest-agent]
+  - [touch, /var/lib/decoy-bootstrap-complete]
+CLOUD
+  if [[ $vmid = 223 ]]; then printf '  - [touch, /etc/decoy-disposable-guest]\n' >> "seed-$vmid/user-data"; fi
+  printf 'instance-id: decoy-%s-20260928\nlocal-hostname: decoy-%s\n' "$vmid" "$role" > "seed-$vmid/meta-data"
+  cat > "seed-$vmid/network-config" <<NET
+version: 2
+ethernets:
+  decoy:
+    match:
+      macaddress: "$mac"
+    dhcp4: true
+NET
+  genisoimage -quiet -output "/var/lib/vz/template/iso/decoy-seed-$vmid.iso" -volid cidata -joliet -rock "seed-$vmid"
+  qm create "$vmid" --name "decoy-$role" --memory "$mem" --cores 2 --cpu host --ostype l26 --scsihw virtio-scsi-pci --agent enabled=1 --onboot 0 --serial0 socket --vga serial0 --args "$args" --description "Dedicated NetScaler honeypot $role; QEMU user networking only. VM223 restrict=on mandatory before attacker execution."
+  qm importdisk "$vmid" debian.qcow2 local-lvm >/dev/null
+  qm set "$vmid" --scsi0 "local-lvm:vm-$vmid-disk-0" --ide2 "local:iso/decoy-seed-$vmid.iso,media=cdrom" --boot order=scsi0
+  qm resize "$vmid" scsi0 12G
+  qm start "$vmid"
+done

deploy/reset-execution.py

@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Hourly VM223 reset, preserving its full disk and pcap first. Never prunes evidence.
+
+--check is read-only. --run stops intake and VM223, archives disk/pcap, validates,
+then rolls back decoy-clean. On any failure it leaves VM223/intake stopped and
+sets the host STOP latch. RAM is not preserved; collector evidence is untouched.
+"""
+import argparse
+import datetime
+import fcntl
+import hashlib
+import json
+import os
+from pathlib import Path
+import shutil
+import stat
+import subprocess
+import time
+import uuid
+
+ROOT = Path('/var/lib/vz/honeypot')
+ARCHIVES = ROOT / 'archives'
+DISK = Path('/dev/pve/vm-223-disk-0')
+LIMIT = 16 * 1024**3
+RESERVE = 4 * 1024**3
+
+
+def command(*args, timeout=60):
+    return subprocess.run(args, check=True, capture_output=True, text=True,
+                          timeout=timeout).stdout
+
+
+def qm(*args, timeout=60):
+    return command('/usr/sbin/qm', *args, timeout=timeout)
+
+
+def guest(script):
+    result = json.loads(qm('guest', 'exec', '222', '--timeout', '20', '--',
+                           '/bin/sh', '-c', script, timeout=25))
+    if not result.get('exited') or result.get('exitcode') != 0:
+        raise RuntimeError('collector guest operation failed')
+    return result.get('out-data', '')
+
+
+def sync_dir(path):
+    fd = os.open(path, os.O_RDONLY | os.O_DIRECTORY)
+    try:
+        os.fsync(fd)
+    finally:
+        os.close(fd)
+
+
+def write_file(path, data):
+    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
+    with os.fdopen(fd, 'wb') as stream:
+        stream.write(data)
+        stream.flush()
+        os.fsync(stream.fileno())
+    sync_dir(path.parent)
+
+
+def bounds(min_free=RESERVE):
+    used = sum(p.stat().st_size for p in ARCHIVES.rglob('*') if p.is_file()) if ARCHIVES.exists() else 0
+    if used >= LIMIT:
+        raise RuntimeError('archive 16GiB budget reached; evidence retained')
+    if shutil.disk_usage(ROOT).free < min_free:
+        raise RuntimeError('host free-space reserve reached; evidence retained')
+
+
+def valid_config(config):
+    values = dict(line.split(': ', 1) for line in config.splitlines() if ': ' in line)
+    if (values.get('name') != 'decoy-execution' or
+            not values.get('scsi0', '').startswith('local-lvm:vm-223-disk-0,') or
+            'restrict=on' not in values.get('args', '') or
+            'file=/var/lib/vz/honeypot/execution.pcap' not in values.get('args', '') or
+            any(k.startswith('net') and k[3:].isdigit() for k in values)):
+        raise RuntimeError('VM223 identity or isolation mismatch')
+
+
+def validate():
+    if os.geteuid() != 0:
+        raise RuntimeError('run on designated Proxmox host as root')
+    if ROOT.is_symlink() or not ROOT.is_dir() or (ARCHIVES.exists() and ARCHIVES.is_symlink()):
+        raise RuntimeError('invalid evidence directory')
+    if (ROOT / 'STOP').exists():
+        raise RuntimeError('preexisting host STOP latch requires operator review')
+    if not stat.S_ISBLK(DISK.stat().st_mode):
+        raise RuntimeError('execution disk is not a block device')
+    current = qm('config', '223')
+    baseline = qm('config', '223', '--snapshot', 'decoy-clean')
+    valid_config(current)
+    valid_config(baseline)
+    guest('test -f /etc/decoy-collector && test ! -e /var/lib/decoy/STOP')
+    return current, baseline
+
+
+def digest(path):
+    hashed = hashlib.sha256()
+    with path.open('rb') as stream:
+        while chunk := stream.read(1024**2):
+            bounds()
+            hashed.update(chunk)
+        os.fsync(stream.fileno())
+    return hashed.hexdigest()
+
+
+def install_admin_ca():
+    public = ROOT / 'admin-ca.authorized'
+    if not public.exists():
+        return
+    line = public.read_text().strip()
+    if not line.startswith('cert-authority,principals="decoy-admin",restrict,pty ssh-ed25519 ') or len(line) > 4096 or '\n' in line:
+        raise RuntimeError('invalid dedicated admin CA authorization')
+    script = ("import pathlib,os; p=pathlib.Path('/root/.ssh/authorized_keys'); "
+              "p.parent.mkdir(mode=0o700,exist_ok=True); "
+              "lines=p.read_text().splitlines() if p.exists() else []; "
+              "line=" + repr(line) + "; "
+              "p.write_text('\\n'.join(lines if line in lines else lines+[line])+'\\n'); "
+              "p.chmod(0o600)")
+    result = json.loads(qm('guest', 'exec', '223', '--timeout', '10', '--', '/usr/bin/python3', '-c', script))
+    if not result.get('exited') or result.get('exitcode') != 0:
+        raise RuntimeError('admin certificate authorization restore failed')
+
+
+def run():
+    # Permanent lock inode; never unlink it while other cron instances may wait.
+    with open('/run/decoy-reset.lock', 'a') as lock:
+        try:
+            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            print('Reset already in progress')
+            return
+        validate()
+        changed = False
+        try:
+            command('systemctl', 'stop', 'decoy-ingress.service')
+            changed = True
+            command('systemctl', 'stop', 'decoy-watchdog.service')
+            guest('systemctl stop decoy.service decoy-network.service')
+            qm('stop', '223')
+            if 'status: stopped' not in qm('status', '223'):
+                raise RuntimeError('execution VM did not stop')
+            # Recheck after intake stops: never erase a concurrent capture-failure latch.
+            current, baseline = validate()
+            bounds(8 * 1024**3)
+            stamp = datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%SZ')
+            ARCHIVES.mkdir(mode=0o700, exist_ok=True)
+            archive = ARCHIVES / (stamp + '-' + uuid.uuid4().hex)
+            archive.mkdir(mode=0o700)
+            sync_dir(ARCHIVES)
+            write_file(archive / 'vm223.conf', current.encode())
+            write_file(archive / 'baseline.conf', baseline.encode())
+            partial = archive / 'execution.qcow2.partial'
+            # Reserve file exclusively before qemu-img writes it; protected parent0700.
+            write_file(partial, b'')
+            with (archive / 'convert.log').open('xb') as output:
+                process = subprocess.Popen(['qemu-img', 'convert', '-c', '-f', 'raw', '-O',
+                                            'qcow2', str(DISK), str(partial)], stdout=output, stderr=output)
+                deadline = time.monotonic() + 900
+                try:
+                    while process.poll() is None:
+                        bounds()
+                        if time.monotonic() > deadline:
+                            raise RuntimeError('disk archive timeout')
+                        time.sleep(0.5)
+                    if process.returncode:
+                        raise RuntimeError('disk archive conversion failed; inspect retained convert.log')
+                finally:
+                    if process.poll() is None:
+                        process.kill()
+                        process.wait()
+                    output.flush()
+                    os.fsync(output.fileno())
+            bounds()
+            check = command('qemu-img', 'check', '--output=json', str(partial), timeout=300)
+            command('qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', str(DISK), str(partial), timeout=300)
+            write_file(archive / 'disk-check.json', check.encode())
+            disk_hash = digest(partial)
+            image = archive / 'execution.qcow2'
+            partial.rename(image)
+            sync_dir(archive)
+            pcap = ROOT / 'execution.pcap'
+            if pcap.is_symlink() or not pcap.is_file():
+                raise RuntimeError('pcap is missing or not a regular file')
+            pcap_hash = digest(pcap)
+            pcap_size = pcap.stat().st_size
+            if sum(p.stat().st_size for p in ARCHIVES.rglob('*') if p.is_file()) + pcap_size >= LIMIT:
+                raise RuntimeError('pcap would exceed archive budget')
+            pcap.rename(archive / 'execution.pcap')
+            sync_dir(ROOT)
+            sync_dir(archive)
+            metadata = {'timestamp_utc': stamp, 'vmid': 223, 'baseline': 'decoy-clean',
+                        'disk_sha256': disk_hash, 'pcap_sha256': pcap_hash,
+                        'disk_bytes': image.stat().st_size, 'pcap_bytes': pcap_size,
+                        'memory_preserved': False, 'state': 'validated_before_rollback'}
+            write_file(archive / 'manifest.json', (json.dumps(metadata, indent=2) + '\n').encode())
+            bounds()
+            # A preexisting STOP is never cleared. Any late latch aborts reset.
+            if (ROOT / 'STOP').exists():
+                raise RuntimeError('host STOP appeared during archive')
+            guest('test ! -e /var/lib/decoy/STOP')
+            qm('rollback', '223', 'decoy-clean', timeout=300)
+            valid_config(qm('config', '223'))
+            write_file(ROOT / 'execution.pcap', b'')
+            guest('test ! -e /var/lib/decoy/STOP && systemctl start decoy-network.service decoy.service && '
+                  'systemctl is-active --quiet decoy-network.service decoy.service')
+            qm('start', '223')
+            deadline = time.monotonic() + 90
+            while True:
+                try:
+                    ready = json.loads(qm('guest', 'exec', '223', '--timeout', '5', '--',
+                        '/bin/sh', '-c', 'test -f /etc/decoy-disposable-guest && systemctl is-active --quiet decoy-relay decoy-guest-network', timeout=10))
+                    if ready.get('exited') and ready.get('exitcode') == 0:
+                        break
+                except (subprocess.SubprocessError, ValueError):
+                    pass
+                if time.monotonic() >= deadline:
+                    raise RuntimeError('restored guest did not become ready')
+                time.sleep(2)
+            install_admin_ca()
+            command('systemctl', 'start', 'decoy-watchdog.service')
+            command('systemctl', 'start', 'decoy-ingress.service')
+            write_file(archive / 'reset-complete.json',
+                       json.dumps({'completed_time_ns': time.time_ns()}).encode())
+            print(f'Reset VM223; disk and packet evidence retained at {archive}')
+        except BaseException:
+            if changed:
+                # Best effort fail-closed; never resume intake or roll back on failure.
+                for args in [('systemctl', 'stop', 'decoy-ingress.service'),
+                             ('/usr/sbin/qm', 'stop', '223')]:
+                    try:
+                        command(*args)
+                    except Exception:
+                        pass
+                try:
+                    guest('systemctl stop decoy.service decoy-network.service')
+                except Exception:
+                    pass
+                try:
+                    write_file(ROOT / 'STOP', b'automatic reset failed; operator review required\n')
+                except FileExistsError:
+                    pass
+            raise
+        finally:
+            if changed:
+                command('systemctl', 'start', 'decoy-watchdog.service')
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=__doc__)
+    mode = parser.add_mutually_exclusive_group(required=True)
+    mode.add_argument('--run', action='store_true')
+    mode.add_argument('--check', action='store_true')
+    args = parser.parse_args()
+    os.umask(0o077)
+    if args.check:
+        validate()
+        bounds(8 * 1024**3)
+        print('Reset preflight passed; no changes made')
+    else:
+        run()

deploy/restrict-pair.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+"""Run once on Proxmox after trusted cloud-init completes, before deploying payload code."""
+import json
+from pathlib import Path
+import shlex
+import subprocess
+
+ROOT = Path('/var/lib/vz/honeypot')
+PCAP = ROOT / 'execution.pcap'
+if PCAP.exists():
+    raise SystemExit('Existing capture found: refusing a restart that would truncate evidence')
+
+def guest(vm, command):
+    result = json.loads(subprocess.check_output(
+        ['qm', 'guest', 'exec', str(vm), '--', '/bin/sh', '-c', command], text=True))
+    if result.get('exitcode') != 0:
+        raise RuntimeError(result)
+    return result.get('out-data', '')
+
+for vm in (222, 223):
+    guest(vm, 'test -f /var/lib/decoy-bootstrap-complete && cloud-init status --long')
+keys = {vm: guest(vm, 'cat /etc/ssh/ssh_host_ed25519_key.pub').strip() for vm in (222, 223)}
+pub = guest(222, 'mkdir -p /root/.ssh; chmod 700 /root/.ssh; '
+            'test -f /root/.ssh/decoy_execution_ed25519 || '
+            'ssh-keygen -q -t ed25519 -N "" -f /root/.ssh/decoy_execution_ed25519; '
+            'cat /root/.ssh/decoy_execution_ed25519.pub').strip()
+guest(223, 'printf "%s\\n" ' + shlex.quote(
+    'no-agent-forwarding,no-port-forwarding,no-X11-forwarding ' + pub) +
+    ' >> /root/.ssh/authorized_keys; chmod 600 /root/.ssh/authorized_keys')
+guest(222, 'printf "%s\\n" ' + shlex.quote('10.0.2.101 ' + keys[223]) +
+      ' > /root/.ssh/known_hosts; chmod 600 /root/.ssh/known_hosts; '
+      'cp /root/.ssh/known_hosts /root/.ssh/decoy_execution_known_hosts')
+config = ('Host decoy-execution\n  HostName 10.0.2.101\n  User root\n'
+          '  IdentityFile /root/.ssh/decoy_execution_ed25519\n  IdentitiesOnly yes\n'
+          '  StrictHostKeyChecking yes\n  BatchMode yes\n  ConnectTimeout 5\n')
+guest(222, 'printf "%s" ' + shlex.quote(config) +
+      ' > /root/.ssh/config; chmod 600 /root/.ssh/config')
+subprocess.run(['qm', 'shutdown', '223', '--timeout', '60'], check=True)
+ROOT.chmod(0o700)
+PCAP.touch(mode=0o600, exist_ok=False)
+# -sandbox applies a QEMU seccomp filter: deny obsolete syscalls and privilege elevation.
+args = ('-netdev user,id=decoy,restrict=on,hostfwd=tcp:127.0.0.1:19223-:22,'
+        'guestfwd=tcp:10.0.2.100:19090-cmd:/var/lib/vz/honeypot/forward-collector-telemetry '
+        '-device virtio-net-pci,netdev=decoy,mac=52:54:00:22:02:23 '
+        '-object filter-dump,id=decoycap,netdev=decoy,'
+        'file=/var/lib/vz/honeypot/execution.pcap,maxlen=512 '
+        '-sandbox on,obsolete=deny,elevateprivileges=deny')
+subprocess.run(['qm', 'set', '223', '--args', args], check=True)
+subprocess.run(['qm', 'start', '223'], check=True)
+for vm, key in keys.items():
+    print(f'decoy-{vm} {key}')  # Public host keys only.

deploy/vm-config.json

@@ -0,0 +1 @@
+{"host":"10.0.2.101","port":22,"identity":"/root/.ssh/decoy_execution_ed25519","known_hosts":"/root/.ssh/decoy_execution_known_hosts","vmid":223,"allowed_peers":"/var/lib/decoy/allowed-peers.json","stop_file":"/var/lib/decoy/STOP"}

deploy/watchdog.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Proxmox-side fail-closed monitor; only the disposable VM223 is stopped."""
+import json
+import pathlib
+import shutil
+import subprocess
+import time
+
+DIRECTORY = pathlib.Path('/var/lib/vz/honeypot')
+
+
+def qm(*args):
+    return subprocess.run(['/usr/sbin/qm', *args], capture_output=True, text=True, timeout=25, check=True).stdout
+
+
+def stop(reason):
+    with (DIRECTORY / 'watchdog-events.jsonl').open('a') as output:
+        output.write(json.dumps(dict(time_ns=time.time_ns(), event='execution_vm_stopped', reason=reason)) + '\n')
+        output.flush()
+    qm('stop', '223')
+    # This latch prevents unattended auto-restart. Operator must verify evidence before clearing.
+    (DIRECTORY / 'STOP').touch(mode=0o600)
+
+
+def main():
+    last_health = 0
+    while True:
+        try:
+            if 'status: running' not in qm('status', '223'):
+                time.sleep(2)
+                continue
+            reason = None
+            pcap = DIRECTORY / 'execution.pcap'
+            if (DIRECTORY / 'STOP').exists():
+                reason = 'host stop latch'
+            elif pcap.exists() and pcap.stat().st_size >= 512 * 1024 * 1024:
+                reason = 'pcap quota'
+            elif shutil.disk_usage(DIRECTORY).free < 4 * 1024**3:
+                reason = 'host free space reserve'
+            elif time.monotonic() - last_health >= 10:
+                last_health = time.monotonic()
+                usage = subprocess.run(['/usr/sbin/lvs', '--noheadings', '-o', 'data_percent', 'pve/data'], capture_output=True, text=True, check=True, timeout=5)
+                if float(usage.stdout.strip()) >= 85:
+                    stop('Proxmox thin pool reserve')
+                    continue
+                try:
+                    result = json.loads(qm('guest', 'exec', '222', '--timeout', '10', '--',
+                        '/bin/sh', '-c',
+                        'test ! -e /var/lib/decoy/STOP && '
+                        'systemctl is-active --quiet decoy.service decoy-network.service'))
+                    if result.get('exitcode') != 0 or not result.get('exited'):
+                        reason = 'collector services stopped or capture failed'
+                except (OSError, ValueError, subprocess.SubprocessError):
+                    reason = 'collector health unavailable'
+            if reason:
+                stop(reason)
+        except (OSError, subprocess.SubprocessError) as error:
+            print(type(error).__name__, flush=True)
+        time.sleep(1)
+
+
+if __name__ == '__main__':
+    main()