homelab-codex-ws/kb/services/job-deploy-runner.md

okf	type	visibility	status	updated	links
0.1	service	private	active	2026-08-03	../decisions/deploy-runner-uzasadnienie.md ../runbooks/deploy-runner-install.md

deploy-runner — host-side executor for redeploy actions

Runs on each node that should be able to execute an operator-approved redeploy. It is the missing half of the self-healing loop: the control-plane executor can generate and dispatch redeploys, but nothing on the node could ever perform one.

How it works now

supervisor      → actions/pending/<id>.json           (unchanged)
operator        → actions/approved/<id>.json          (unchanged, HITL)
executor (vps)  → actions/deploy/<node>/<id>.json     ← dispatch only, no execution
deploy-runner   ← rsync-pull (node initiates; VPS never connects to a node)
                → scripts/deploy/deploy-service.sh --force-recreate
                → events/<node>/evt-…-action_result-….json → rsync-push
executor        → completed / failed                  (via _reconcile_running_actions)

Design points, and why:

• Host-level, not a container. Compose resolves relative bind mounts and the project name against the filesystem of whatever runs it. In a container those resolve to container paths that the daemon then interprets as host paths — a silent way to produce broken mounts, or to land in a different Compose project where --remove-orphans deletes the running stack. On the host it behaves exactly like a human deploy.sh.
• Independent of node-agent (own rsync pull, own result push) so it can redeploy node-agent itself — the solaria case, where node-agent has been docker-blind for weeks.
• Separate inbox from actions/dispatch/<node>/: node-agent deletes and failure-reports every file in its own inbox that is not container_restart.
• No git pull. A redeploy reconciles the node to the checkout it already has. Shipping new code stays a human scripts/deploy/deploy.sh action.
• No --build, no --remove-orphans, always --force-recreate. Plain up -d is a no-op when config is unchanged — exactly the unhealthy-container case; building on the 4 GiB VPS mid-incident is an OOM risk.

Every action is validated before anything runs (action.py): type must be redeploy, node must match this node, the service name must be a plain kebab-case name, the service must be listed in hosts/<node>/services.yaml, and its compose file must exist. Nothing from the action payload is ever executed — only a validated service name reaches the deploy script. Actions are idempotent (marker in state/processed-deploy-actions/), single-instance (flock), and time-boxed (DEPLOY_TIMEOUT_SECS, default 600 s, below the executor's REDEPLOY_TIMEOUT_SECS of 900 s so the node reports before the control plane gives up).