diff --git a/jobs/deploy-runner/README.md b/jobs/deploy-runner/README.md deleted file mode 100644 index 4ec0738..0000000 --- a/jobs/deploy-runner/README.md +++ /dev/null @@ -1,111 +0,0 @@ -# deploy-runner — host-side executor for `redeploy` actions - -Runs on each node that should be able to execute an operator-approved -`redeploy`. It is the missing half of the self-healing loop: the control-plane -executor can generate and dispatch redeploys, but nothing on the node could ever -perform one. - -## What was broken - -The executor ran `scripts/deploy/deploy-node.sh ` **inside its own -container**. That script ignores both arguments, and expects a repo at -`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it -exited at line 18 with `Error: Repository not found`. Behind that failure sat -three more: no `git`, no `docker` CLI in the image, and — had it ever got that -far — it would have deployed the **executor host's** entire service set, not the -action's target node/service. Every `healthcheck_failed → redeploy` dead-ended -(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15). - -## How it works now - -``` -supervisor → actions/pending/.json (unchanged) -operator → actions/approved/.json (unchanged, HITL) -executor (vps) → actions/deploy//.json ← dispatch only, no execution -deploy-runner ← rsync-pull (node initiates; VPS never connects to a node) - → scripts/deploy/deploy-service.sh --force-recreate - → events//evt-…-action_result-….json → rsync-push -executor → completed / failed (via _reconcile_running_actions) -``` - -Design points, and why: - -- **Host-level, not a container.** Compose resolves relative bind mounts and the - project name against the filesystem of whatever runs it. In a container those - resolve to container paths that the daemon then interprets as host paths — a - silent way to produce broken mounts, or to land in a different Compose project - where `--remove-orphans` deletes the running stack. On the host it behaves - exactly like a human `deploy.sh`. -- **Independent of node-agent** (own rsync pull, own result push) so it can - redeploy `node-agent` itself — the solaria case, where node-agent has been - docker-blind for weeks. -- **Separate inbox** from `actions/dispatch//`: node-agent deletes and - failure-reports every file in its own inbox that is not `container_restart`. -- **No `git pull`.** A redeploy reconciles the node to the checkout it already - has. Shipping new code stays a human `scripts/deploy/deploy.sh` action. -- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain - `up -d` is a no-op when config is unchanged — exactly the unhealthy-container - case; building on the 4 GiB VPS mid-incident is an OOM risk. - -Every action is validated before anything runs (`action.py`): type must be -`redeploy`, `node` must match this node, the service name must be a plain -kebab-case name, the service must be listed in `hosts//services.yaml`, and -its compose file must exist. Nothing from the action payload is ever executed — -only a validated service *name* reaches the deploy script. Actions are -idempotent (marker in `state/processed-deploy-actions/`), single-instance -(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the -executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the -control plane gives up). - -## Install (per node) - -Not deployed by `deploy.sh` — it is a host-level systemd unit, like -`jobs/documents-ingest/`. On the target node: - -```bash -# 1. config -sudo mkdir -p /opt/homelab/config/deploy-runner -sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \ - /opt/homelab/config/deploy-runner/env -sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env -sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS - -# 2. units -sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \ - /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now homelab-deploy-runner.timer - -# 3. verify (no action queued yet → one clean, empty run) -sudo systemctl start homelab-deploy-runner.service -journalctl -u homelab-deploy-runner.service -n 30 --no-pager -``` - -Requirements on the node: repo checkout at `REPO_PATH`, the service user in the -`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key -that reaches the VPS — the same one node-agent already uses for event shipping. - -On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same -`/opt/homelab` mount, so there is nothing to pull and nothing to push. - -## Operating it - -- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy//` on the VPS. -- Per-action deploy output: `/opt/homelab/logs/deploy-runner/-.log` on the node. -- Runner activity: `journalctl -u homelab-deploy-runner.service`. -- A redeploy for a service with its own `deploy-local.sh` (control-plane) is - reported as **failed** with an explanatory message — those need an operator - deploy, by design. -- To let an already-processed action run again, remove its marker: - `rm /opt/homelab/state/processed-deploy-actions/.done`. - -## Tests - -`tests/` — validation and rejection cases, the event format contract against the -executor's real parser, the `docker compose` argv contract of -`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself -with a stubbed `docker`. - -```bash -python3 -m pytest jobs/deploy-runner/tests -q -``` diff --git a/kb/decisions/deploy-runner-uzasadnienie.md b/kb/decisions/deploy-runner-uzasadnienie.md new file mode 100644 index 0000000..438d19f --- /dev/null +++ b/kb/decisions/deploy-runner-uzasadnienie.md @@ -0,0 +1,24 @@ +--- +okf: "0.1" +type: decision +visibility: private +status: active +updated: 2026-08-03 +links: + - ../services/job-deploy-runner.md + - ../runbooks/deploy-runner-install.md +--- + +# deploy-runner — co bylo zepsute (uzasadnienie zmiany toru redeployu) + +## What was broken + +The executor ran `scripts/deploy/deploy-node.sh ` **inside its own +container**. That script ignores both arguments, and expects a repo at +`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it +exited at line 18 with `Error: Repository not found`. Behind that failure sat +three more: no `git`, no `docker` CLI in the image, and — had it ever got that +far — it would have deployed the **executor host's** entire service set, not the +action's target node/service. Every `healthcheck_failed → redeploy` dead-ended +(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15). + diff --git a/kb/runbooks/deploy-runner-install.md b/kb/runbooks/deploy-runner-install.md new file mode 100644 index 0000000..dc614c2 --- /dev/null +++ b/kb/runbooks/deploy-runner-install.md @@ -0,0 +1,65 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-08-03 +links: + - ../services/job-deploy-runner.md + - ../decisions/deploy-runner-uzasadnienie.md +--- + +# deploy-runner — instalacja i obsluga + +## Install (per node) + +Not deployed by `deploy.sh` — it is a host-level systemd unit, like +`jobs/documents-ingest/`. On the target node: + +```bash +# 1. config +sudo mkdir -p /opt/homelab/config/deploy-runner +sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \ + /opt/homelab/config/deploy-runner/env +sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env +sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS + +# 2. units +sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \ + /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now homelab-deploy-runner.timer + +# 3. verify (no action queued yet → one clean, empty run) +sudo systemctl start homelab-deploy-runner.service +journalctl -u homelab-deploy-runner.service -n 30 --no-pager +``` + +Requirements on the node: repo checkout at `REPO_PATH`, the service user in the +`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key +that reaches the VPS — the same one node-agent already uses for event shipping. + +On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same +`/opt/homelab` mount, so there is nothing to pull and nothing to push. + +## Operating it + +- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy//` on the VPS. +- Per-action deploy output: `/opt/homelab/logs/deploy-runner/-.log` on the node. +- Runner activity: `journalctl -u homelab-deploy-runner.service`. +- A redeploy for a service with its own `deploy-local.sh` (control-plane) is + reported as **failed** with an explanatory message — those need an operator + deploy, by design. +- To let an already-processed action run again, remove its marker: + `rm /opt/homelab/state/processed-deploy-actions/.done`. + +## Tests + +`tests/` — validation and rejection cases, the event format contract against the +executor's real parser, the `docker compose` argv contract of +`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself +with a stubbed `docker`. + +```bash +python3 -m pytest jobs/deploy-runner/tests -q +``` diff --git a/kb/services/job-deploy-runner.md b/kb/services/job-deploy-runner.md new file mode 100644 index 0000000..d9f3b55 --- /dev/null +++ b/kb/services/job-deploy-runner.md @@ -0,0 +1,59 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-08-03 +links: + - ../decisions/deploy-runner-uzasadnienie.md + - ../runbooks/deploy-runner-install.md +--- + +# deploy-runner — host-side executor for `redeploy` actions + +Runs on each node that should be able to execute an operator-approved +`redeploy`. It is the missing half of the self-healing loop: the control-plane +executor can generate and dispatch redeploys, but nothing on the node could ever +perform one. + +## How it works now + +``` +supervisor → actions/pending/.json (unchanged) +operator → actions/approved/.json (unchanged, HITL) +executor (vps) → actions/deploy//.json ← dispatch only, no execution +deploy-runner ← rsync-pull (node initiates; VPS never connects to a node) + → scripts/deploy/deploy-service.sh --force-recreate + → events//evt-…-action_result-….json → rsync-push +executor → completed / failed (via _reconcile_running_actions) +``` + +Design points, and why: + +- **Host-level, not a container.** Compose resolves relative bind mounts and the + project name against the filesystem of whatever runs it. In a container those + resolve to container paths that the daemon then interprets as host paths — a + silent way to produce broken mounts, or to land in a different Compose project + where `--remove-orphans` deletes the running stack. On the host it behaves + exactly like a human `deploy.sh`. +- **Independent of node-agent** (own rsync pull, own result push) so it can + redeploy `node-agent` itself — the solaria case, where node-agent has been + docker-blind for weeks. +- **Separate inbox** from `actions/dispatch//`: node-agent deletes and + failure-reports every file in its own inbox that is not `container_restart`. +- **No `git pull`.** A redeploy reconciles the node to the checkout it already + has. Shipping new code stays a human `scripts/deploy/deploy.sh` action. +- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain + `up -d` is a no-op when config is unchanged — exactly the unhealthy-container + case; building on the 4 GiB VPS mid-incident is an OOM risk. + +Every action is validated before anything runs (`action.py`): type must be +`redeploy`, `node` must match this node, the service name must be a plain +kebab-case name, the service must be listed in `hosts//services.yaml`, and +its compose file must exist. Nothing from the action payload is ever executed — +only a validated service *name* reaches the deploy script. Actions are +idempotent (marker in `state/processed-deploy-actions/`), single-instance +(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the +executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the +control plane gives up). +