homelab-codex-ws/jobs/deploy-runner/README.md

# deploy-runner — host-side executor for `redeploy` actions

Runs on each node that should be able to execute an operator-approved
`redeploy`. It is the missing half of the self-healing loop: the control-plane
executor can generate and dispatch redeploys, but nothing on the node could ever
perform one.

## What was broken

The executor ran `scripts/deploy/deploy-node.sh <node> <service>` **inside its own
container**. That script ignores both arguments, and expects a repo at
`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it
exited at line 18 with `Error: Repository not found`. Behind that failure sat
three more: no `git`, no `docker` CLI in the image, and — had it ever got that
far — it would have deployed the **executor host's** entire service set, not the
action's target node/service. Every `healthcheck_failed → redeploy` dead-ended
(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15).

## How it works now

```
supervisor      → actions/pending/<id>.json           (unchanged)
operator        → actions/approved/<id>.json          (unchanged, HITL)
executor (vps)  → actions/deploy/<node>/<id>.json     ← dispatch only, no execution
deploy-runner   ← rsync-pull (node initiates; VPS never connects to a node)
                → scripts/deploy/deploy-service.sh --force-recreate
                → events/<node>/evt-…-action_result-….json → rsync-push
executor        → completed / failed                  (via _reconcile_running_actions)
```

Design points, and why:

- **Host-level, not a container.** Compose resolves relative bind mounts and the
  project name against the filesystem of whatever runs it. In a container those
  resolve to container paths that the daemon then interprets as host paths — a
  silent way to produce broken mounts, or to land in a different Compose project
  where `--remove-orphans` deletes the running stack. On the host it behaves
  exactly like a human `deploy.sh`.
- **Independent of node-agent** (own rsync pull, own result push) so it can
  redeploy `node-agent` itself — the solaria case, where node-agent has been
  docker-blind for weeks.
- **Separate inbox** from `actions/dispatch/<node>/`: node-agent deletes and
  failure-reports every file in its own inbox that is not `container_restart`.
- **No `git pull`.** A redeploy reconciles the node to the checkout it already
  has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain
  `up -d` is a no-op when config is unchanged — exactly the unhealthy-container
  case; building on the 4 GiB VPS mid-incident is an OOM risk.

Every action is validated before anything runs (`action.py`): type must be
`redeploy`, `node` must match this node, the service name must be a plain
kebab-case name, the service must be listed in `hosts/<node>/services.yaml`, and
its compose file must exist. Nothing from the action payload is ever executed —
only a validated service *name* reaches the deploy script. Actions are
idempotent (marker in `state/processed-deploy-actions/`), single-instance
(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the
executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the
control plane gives up).

## Install (per node)

Not deployed by `deploy.sh` — it is a host-level systemd unit, like
`jobs/documents-ingest/`. On the target node:

```bash
# 1. config
sudo mkdir -p /opt/homelab/config/deploy-runner
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
        /opt/homelab/config/deploy-runner/env
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
sudoedit /opt/homelab/config/deploy-runner/env      # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS

# 2. units
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now homelab-deploy-runner.timer

# 3. verify (no action queued yet → one clean, empty run)
sudo systemctl start homelab-deploy-runner.service
journalctl -u homelab-deploy-runner.service -n 30 --no-pager
```

Requirements on the node: repo checkout at `REPO_PATH`, the service user in the
`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key
that reaches the VPS — the same one node-agent already uses for event shipping.

On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same
`/opt/homelab` mount, so there is nothing to pull and nothing to push.

## Operating it

- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy/<node>/` on the VPS.
- Per-action deploy output: `/opt/homelab/logs/deploy-runner/<action_id>-<ts>.log` on the node.
- Runner activity: `journalctl -u homelab-deploy-runner.service`.
- A redeploy for a service with its own `deploy-local.sh` (control-plane) is
  reported as **failed** with an explanatory message — those need an operator
  deploy, by design.
- To let an already-processed action run again, remove its marker:
  `rm /opt/homelab/state/processed-deploy-actions/<action_id>.done`.

## Tests

`tests/` — validation and rejection cases, the event format contract against the
executor's real parser, the `docker compose` argv contract of
`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself
with a stubbed `docker`.

```bash
python3 -m pytest jobs/deploy-runner/tests -q
```