homelab-codex-ws/jobs/deploy-runner/README.md

deploy-runner — host-side executor for redeploy actions

Runs on each node that should be able to execute an operator-approved redeploy. It is the missing half of the self-healing loop: the control-plane executor can generate and dispatch redeploys, but nothing on the node could ever perform one.

What was broken

The executor ran scripts/deploy/deploy-node.sh <node> <service> inside its own container. That script ignores both arguments, and expects a repo at ${HOME}/homelab-codex-ws — inside the container HOME=/home/homelab, so it exited at line 18 with Error: Repository not found. Behind that failure sat three more: no git, no docker CLI in the image, and — had it ever got that far — it would have deployed the executor host's entire service set, not the action's target node/service. Every healthcheck_failed → redeploy dead-ended (recon docs/architecture/RECON-multiagent-2026-07-27.md, D14/D15).

How it works now

supervisor      → actions/pending/<id>.json           (unchanged)
operator        → actions/approved/<id>.json          (unchanged, HITL)
executor (vps)  → actions/deploy/<node>/<id>.json     ← dispatch only, no execution
deploy-runner   ← rsync-pull (node initiates; VPS never connects to a node)
                → scripts/deploy/deploy-service.sh --force-recreate
                → events/<node>/evt-…-action_result-….json → rsync-push
executor        → completed / failed                  (via _reconcile_running_actions)

Design points, and why:

• Host-level, not a container. Compose resolves relative bind mounts and the project name against the filesystem of whatever runs it. In a container those resolve to container paths that the daemon then interprets as host paths — a silent way to produce broken mounts, or to land in a different Compose project where --remove-orphans deletes the running stack. On the host it behaves exactly like a human deploy.sh.
• Independent of node-agent (own rsync pull, own result push) so it can redeploy node-agent itself — the solaria case, where node-agent has been docker-blind for weeks.
• Separate inbox from actions/dispatch/<node>/: node-agent deletes and failure-reports every file in its own inbox that is not container_restart.
• No git pull. A redeploy reconciles the node to the checkout it already has. Shipping new code stays a human scripts/deploy/deploy.sh action.
• No --build, no --remove-orphans, always --force-recreate. Plain up -d is a no-op when config is unchanged — exactly the unhealthy-container case; building on the 4 GiB VPS mid-incident is an OOM risk.

Every action is validated before anything runs (action.py): type must be redeploy, node must match this node, the service name must be a plain kebab-case name, the service must be listed in hosts/<node>/services.yaml, and its compose file must exist. Nothing from the action payload is ever executed — only a validated service name reaches the deploy script. Actions are idempotent (marker in state/processed-deploy-actions/), single-instance (flock), and time-boxed (DEPLOY_TIMEOUT_SECS, default 600 s, below the executor's REDEPLOY_TIMEOUT_SECS of 900 s so the node reports before the control plane gives up).

Install (per node)

Not deployed by deploy.sh — it is a host-level systemd unit, like jobs/documents-ingest/. On the target node:

# 1. config
sudo mkdir -p /opt/homelab/config/deploy-runner
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
        /opt/homelab/config/deploy-runner/env
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
sudoedit /opt/homelab/config/deploy-runner/env      # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS

# 2. units
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now homelab-deploy-runner.timer

# 3. verify (no action queued yet → one clean, empty run)
sudo systemctl start homelab-deploy-runner.service
journalctl -u homelab-deploy-runner.service -n 30 --no-pager

Requirements on the node: repo checkout at REPO_PATH, the service user in the docker group, python3 + PyYAML, rsync, and (remote nodes only) an ssh key that reaches the VPS — the same one node-agent already uses for event shipping.

On the VPS leave VPS_EVENTS_HOST empty: the executor writes into the same /opt/homelab mount, so there is nothing to pull and nothing to push.

Operating it

• Dispatched but not yet collected: ls /opt/homelab/actions/deploy/<node>/ on the VPS.
• Per-action deploy output: /opt/homelab/logs/deploy-runner/<action_id>-<ts>.log on the node.
• Runner activity: journalctl -u homelab-deploy-runner.service.
• A redeploy for a service with its own deploy-local.sh (control-plane) is reported as failed with an explanatory message — those need an operator deploy, by design.
• To let an already-processed action run again, remove its marker: rm /opt/homelab/state/processed-deploy-actions/<action_id>.done.

Tests

tests/ — validation and rejection cases, the event format contract against the executor's real parser, the docker compose argv contract of scripts/deploy/deploy-service.sh, and an end-to-end run of the runner itself with a stubbed docker.

python3 -m pytest jobs/deploy-runner/tests -q