da151fc8d3
Executor odpalal scripts/deploy/deploy-node.sh <node> <service> wewnatrz
swojego kontenera: skrypt ignoruje oba argumenty i wymaga repo w
${HOME}/homelab-codex-ws (w kontenerze HOME=/home/homelab) -> exit 1 w 18.
linii. Za tym brak git, brak klienta docker w obrazie, a gdyby przeszedl —
deploy calego zestawu uslug hosta executora zamiast wezla z akcji. Kazdy
redeploy padal (recon D14/D15; 18 pending / 0 completed).
Redeploy idzie teraz ta sama sciezka pull co container_restart — VPS nigdy
nie inicjuje polaczenia do wezla:
executor -> actions/deploy/<node>/<id>.json
-> deploy-runner (systemd na hoscie) rsync-pull, walidacja, deploy
-> action_result event -> executor rozlicza completed/failed
- scripts/deploy/deploy-service.sh: deploy jednej uslugi, wspoldzielony z
deploy-node.sh, wiec inwokacja compose (a przez to nazwa projektu) jest
identyczna jak przy deployu recznym
- jobs/deploy-runner/: host-level, nie kontener — compose rozwiazuje
wzgledne bindy i nazwe projektu tak jak przy deployu czlowieka;
niezalezny od node-agenta, wiec potrafi zredeployowac takze jego
- walidacja: tylko typ redeploy, node musi sie zgadzac, usluga musi byc w
hosts/<node>/services.yaml, zadna tresc z payloadu nie trafia do shella
- --force-recreate bez --build i bez --remove-orphans: redeploy to
rekoncyliacja, nie wysylka kodu
- executor: REDEPLOY_TIMEOUT_SECS=900, /repo zjechany do :ro (nieuzywany)
248 testow zielonych; deploy-node.sh przecwiczony na atrapie dockera —
argv compose bez zmian. Instalacja unitow na wezlach i E2E: backlog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
112 lines
5.4 KiB
Markdown
112 lines
5.4 KiB
Markdown
# deploy-runner — host-side executor for `redeploy` actions
|
|
|
|
Runs on each node that should be able to execute an operator-approved
|
|
`redeploy`. It is the missing half of the self-healing loop: the control-plane
|
|
executor can generate and dispatch redeploys, but nothing on the node could ever
|
|
perform one.
|
|
|
|
## What was broken
|
|
|
|
The executor ran `scripts/deploy/deploy-node.sh <node> <service>` **inside its own
|
|
container**. That script ignores both arguments, and expects a repo at
|
|
`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it
|
|
exited at line 18 with `Error: Repository not found`. Behind that failure sat
|
|
three more: no `git`, no `docker` CLI in the image, and — had it ever got that
|
|
far — it would have deployed the **executor host's** entire service set, not the
|
|
action's target node/service. Every `healthcheck_failed → redeploy` dead-ended
|
|
(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15).
|
|
|
|
## How it works now
|
|
|
|
```
|
|
supervisor → actions/pending/<id>.json (unchanged)
|
|
operator → actions/approved/<id>.json (unchanged, HITL)
|
|
executor (vps) → actions/deploy/<node>/<id>.json ← dispatch only, no execution
|
|
deploy-runner ← rsync-pull (node initiates; VPS never connects to a node)
|
|
→ scripts/deploy/deploy-service.sh --force-recreate
|
|
→ events/<node>/evt-…-action_result-….json → rsync-push
|
|
executor → completed / failed (via _reconcile_running_actions)
|
|
```
|
|
|
|
Design points, and why:
|
|
|
|
- **Host-level, not a container.** Compose resolves relative bind mounts and the
|
|
project name against the filesystem of whatever runs it. In a container those
|
|
resolve to container paths that the daemon then interprets as host paths — a
|
|
silent way to produce broken mounts, or to land in a different Compose project
|
|
where `--remove-orphans` deletes the running stack. On the host it behaves
|
|
exactly like a human `deploy.sh`.
|
|
- **Independent of node-agent** (own rsync pull, own result push) so it can
|
|
redeploy `node-agent` itself — the solaria case, where node-agent has been
|
|
docker-blind for weeks.
|
|
- **Separate inbox** from `actions/dispatch/<node>/`: node-agent deletes and
|
|
failure-reports every file in its own inbox that is not `container_restart`.
|
|
- **No `git pull`.** A redeploy reconciles the node to the checkout it already
|
|
has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
|
|
- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain
|
|
`up -d` is a no-op when config is unchanged — exactly the unhealthy-container
|
|
case; building on the 4 GiB VPS mid-incident is an OOM risk.
|
|
|
|
Every action is validated before anything runs (`action.py`): type must be
|
|
`redeploy`, `node` must match this node, the service name must be a plain
|
|
kebab-case name, the service must be listed in `hosts/<node>/services.yaml`, and
|
|
its compose file must exist. Nothing from the action payload is ever executed —
|
|
only a validated service *name* reaches the deploy script. Actions are
|
|
idempotent (marker in `state/processed-deploy-actions/`), single-instance
|
|
(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the
|
|
executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the
|
|
control plane gives up).
|
|
|
|
## Install (per node)
|
|
|
|
Not deployed by `deploy.sh` — it is a host-level systemd unit, like
|
|
`jobs/documents-ingest/`. On the target node:
|
|
|
|
```bash
|
|
# 1. config
|
|
sudo mkdir -p /opt/homelab/config/deploy-runner
|
|
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
|
|
/opt/homelab/config/deploy-runner/env
|
|
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
|
|
sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS
|
|
|
|
# 2. units
|
|
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
|
|
/etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now homelab-deploy-runner.timer
|
|
|
|
# 3. verify (no action queued yet → one clean, empty run)
|
|
sudo systemctl start homelab-deploy-runner.service
|
|
journalctl -u homelab-deploy-runner.service -n 30 --no-pager
|
|
```
|
|
|
|
Requirements on the node: repo checkout at `REPO_PATH`, the service user in the
|
|
`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key
|
|
that reaches the VPS — the same one node-agent already uses for event shipping.
|
|
|
|
On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same
|
|
`/opt/homelab` mount, so there is nothing to pull and nothing to push.
|
|
|
|
## Operating it
|
|
|
|
- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy/<node>/` on the VPS.
|
|
- Per-action deploy output: `/opt/homelab/logs/deploy-runner/<action_id>-<ts>.log` on the node.
|
|
- Runner activity: `journalctl -u homelab-deploy-runner.service`.
|
|
- A redeploy for a service with its own `deploy-local.sh` (control-plane) is
|
|
reported as **failed** with an explanatory message — those need an operator
|
|
deploy, by design.
|
|
- To let an already-processed action run again, remove its marker:
|
|
`rm /opt/homelab/state/processed-deploy-actions/<action_id>.done`.
|
|
|
|
## Tests
|
|
|
|
`tests/` — validation and rejection cases, the event format contract against the
|
|
executor's real parser, the `docker compose` argv contract of
|
|
`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself
|
|
with a stubbed `docker`.
|
|
|
|
```bash
|
|
python3 -m pytest jobs/deploy-runner/tests -q
|
|
```
|