feat(kb): SPLIT deploy-runner -> service + decision + runbook
kb/services/job-deploy-runner.md (How it works now) kb/decisions/deploy-runner-uzasadnienie.md (What was broken) kb/runbooks/deploy-runner-install.md (Install per node, Operating it, Tests) Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
oskar
parent
05eaf31613
commit
280ed48d9a
|
|
@ -1,111 +0,0 @@
|
|||
# deploy-runner — host-side executor for `redeploy` actions
|
||||
|
||||
Runs on each node that should be able to execute an operator-approved
|
||||
`redeploy`. It is the missing half of the self-healing loop: the control-plane
|
||||
executor can generate and dispatch redeploys, but nothing on the node could ever
|
||||
perform one.
|
||||
|
||||
## What was broken
|
||||
|
||||
The executor ran `scripts/deploy/deploy-node.sh <node> <service>` **inside its own
|
||||
container**. That script ignores both arguments, and expects a repo at
|
||||
`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it
|
||||
exited at line 18 with `Error: Repository not found`. Behind that failure sat
|
||||
three more: no `git`, no `docker` CLI in the image, and — had it ever got that
|
||||
far — it would have deployed the **executor host's** entire service set, not the
|
||||
action's target node/service. Every `healthcheck_failed → redeploy` dead-ended
|
||||
(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15).
|
||||
|
||||
## How it works now
|
||||
|
||||
```
|
||||
supervisor → actions/pending/<id>.json (unchanged)
|
||||
operator → actions/approved/<id>.json (unchanged, HITL)
|
||||
executor (vps) → actions/deploy/<node>/<id>.json ← dispatch only, no execution
|
||||
deploy-runner ← rsync-pull (node initiates; VPS never connects to a node)
|
||||
→ scripts/deploy/deploy-service.sh --force-recreate
|
||||
→ events/<node>/evt-…-action_result-….json → rsync-push
|
||||
executor → completed / failed (via _reconcile_running_actions)
|
||||
```
|
||||
|
||||
Design points, and why:
|
||||
|
||||
- **Host-level, not a container.** Compose resolves relative bind mounts and the
|
||||
project name against the filesystem of whatever runs it. In a container those
|
||||
resolve to container paths that the daemon then interprets as host paths — a
|
||||
silent way to produce broken mounts, or to land in a different Compose project
|
||||
where `--remove-orphans` deletes the running stack. On the host it behaves
|
||||
exactly like a human `deploy.sh`.
|
||||
- **Independent of node-agent** (own rsync pull, own result push) so it can
|
||||
redeploy `node-agent` itself — the solaria case, where node-agent has been
|
||||
docker-blind for weeks.
|
||||
- **Separate inbox** from `actions/dispatch/<node>/`: node-agent deletes and
|
||||
failure-reports every file in its own inbox that is not `container_restart`.
|
||||
- **No `git pull`.** A redeploy reconciles the node to the checkout it already
|
||||
has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
|
||||
- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain
|
||||
`up -d` is a no-op when config is unchanged — exactly the unhealthy-container
|
||||
case; building on the 4 GiB VPS mid-incident is an OOM risk.
|
||||
|
||||
Every action is validated before anything runs (`action.py`): type must be
|
||||
`redeploy`, `node` must match this node, the service name must be a plain
|
||||
kebab-case name, the service must be listed in `hosts/<node>/services.yaml`, and
|
||||
its compose file must exist. Nothing from the action payload is ever executed —
|
||||
only a validated service *name* reaches the deploy script. Actions are
|
||||
idempotent (marker in `state/processed-deploy-actions/`), single-instance
|
||||
(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the
|
||||
executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the
|
||||
control plane gives up).
|
||||
|
||||
## Install (per node)
|
||||
|
||||
Not deployed by `deploy.sh` — it is a host-level systemd unit, like
|
||||
`jobs/documents-ingest/`. On the target node:
|
||||
|
||||
```bash
|
||||
# 1. config
|
||||
sudo mkdir -p /opt/homelab/config/deploy-runner
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
|
||||
/opt/homelab/config/deploy-runner/env
|
||||
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
|
||||
sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS
|
||||
|
||||
# 2. units
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
|
||||
/etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now homelab-deploy-runner.timer
|
||||
|
||||
# 3. verify (no action queued yet → one clean, empty run)
|
||||
sudo systemctl start homelab-deploy-runner.service
|
||||
journalctl -u homelab-deploy-runner.service -n 30 --no-pager
|
||||
```
|
||||
|
||||
Requirements on the node: repo checkout at `REPO_PATH`, the service user in the
|
||||
`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key
|
||||
that reaches the VPS — the same one node-agent already uses for event shipping.
|
||||
|
||||
On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same
|
||||
`/opt/homelab` mount, so there is nothing to pull and nothing to push.
|
||||
|
||||
## Operating it
|
||||
|
||||
- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy/<node>/` on the VPS.
|
||||
- Per-action deploy output: `/opt/homelab/logs/deploy-runner/<action_id>-<ts>.log` on the node.
|
||||
- Runner activity: `journalctl -u homelab-deploy-runner.service`.
|
||||
- A redeploy for a service with its own `deploy-local.sh` (control-plane) is
|
||||
reported as **failed** with an explanatory message — those need an operator
|
||||
deploy, by design.
|
||||
- To let an already-processed action run again, remove its marker:
|
||||
`rm /opt/homelab/state/processed-deploy-actions/<action_id>.done`.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/` — validation and rejection cases, the event format contract against the
|
||||
executor's real parser, the `docker compose` argv contract of
|
||||
`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself
|
||||
with a stubbed `docker`.
|
||||
|
||||
```bash
|
||||
python3 -m pytest jobs/deploy-runner/tests -q
|
||||
```
|
||||
24
kb/decisions/deploy-runner-uzasadnienie.md
Normal file
24
kb/decisions/deploy-runner-uzasadnienie.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: decision
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-08-03
|
||||
links:
|
||||
- ../services/job-deploy-runner.md
|
||||
- ../runbooks/deploy-runner-install.md
|
||||
---
|
||||
|
||||
# deploy-runner — co bylo zepsute (uzasadnienie zmiany toru redeployu)
|
||||
|
||||
## What was broken
|
||||
|
||||
The executor ran `scripts/deploy/deploy-node.sh <node> <service>` **inside its own
|
||||
container**. That script ignores both arguments, and expects a repo at
|
||||
`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it
|
||||
exited at line 18 with `Error: Repository not found`. Behind that failure sat
|
||||
three more: no `git`, no `docker` CLI in the image, and — had it ever got that
|
||||
far — it would have deployed the **executor host's** entire service set, not the
|
||||
action's target node/service. Every `healthcheck_failed → redeploy` dead-ended
|
||||
(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15).
|
||||
|
||||
65
kb/runbooks/deploy-runner-install.md
Normal file
65
kb/runbooks/deploy-runner-install.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-08-03
|
||||
links:
|
||||
- ../services/job-deploy-runner.md
|
||||
- ../decisions/deploy-runner-uzasadnienie.md
|
||||
---
|
||||
|
||||
# deploy-runner — instalacja i obsluga
|
||||
|
||||
## Install (per node)
|
||||
|
||||
Not deployed by `deploy.sh` — it is a host-level systemd unit, like
|
||||
`jobs/documents-ingest/`. On the target node:
|
||||
|
||||
```bash
|
||||
# 1. config
|
||||
sudo mkdir -p /opt/homelab/config/deploy-runner
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
|
||||
/opt/homelab/config/deploy-runner/env
|
||||
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
|
||||
sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS
|
||||
|
||||
# 2. units
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
|
||||
/etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now homelab-deploy-runner.timer
|
||||
|
||||
# 3. verify (no action queued yet → one clean, empty run)
|
||||
sudo systemctl start homelab-deploy-runner.service
|
||||
journalctl -u homelab-deploy-runner.service -n 30 --no-pager
|
||||
```
|
||||
|
||||
Requirements on the node: repo checkout at `REPO_PATH`, the service user in the
|
||||
`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key
|
||||
that reaches the VPS — the same one node-agent already uses for event shipping.
|
||||
|
||||
On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same
|
||||
`/opt/homelab` mount, so there is nothing to pull and nothing to push.
|
||||
|
||||
## Operating it
|
||||
|
||||
- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy/<node>/` on the VPS.
|
||||
- Per-action deploy output: `/opt/homelab/logs/deploy-runner/<action_id>-<ts>.log` on the node.
|
||||
- Runner activity: `journalctl -u homelab-deploy-runner.service`.
|
||||
- A redeploy for a service with its own `deploy-local.sh` (control-plane) is
|
||||
reported as **failed** with an explanatory message — those need an operator
|
||||
deploy, by design.
|
||||
- To let an already-processed action run again, remove its marker:
|
||||
`rm /opt/homelab/state/processed-deploy-actions/<action_id>.done`.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/` — validation and rejection cases, the event format contract against the
|
||||
executor's real parser, the `docker compose` argv contract of
|
||||
`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself
|
||||
with a stubbed `docker`.
|
||||
|
||||
```bash
|
||||
python3 -m pytest jobs/deploy-runner/tests -q
|
||||
```
|
||||
59
kb/services/job-deploy-runner.md
Normal file
59
kb/services/job-deploy-runner.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-08-03
|
||||
links:
|
||||
- ../decisions/deploy-runner-uzasadnienie.md
|
||||
- ../runbooks/deploy-runner-install.md
|
||||
---
|
||||
|
||||
# deploy-runner — host-side executor for `redeploy` actions
|
||||
|
||||
Runs on each node that should be able to execute an operator-approved
|
||||
`redeploy`. It is the missing half of the self-healing loop: the control-plane
|
||||
executor can generate and dispatch redeploys, but nothing on the node could ever
|
||||
perform one.
|
||||
|
||||
## How it works now
|
||||
|
||||
```
|
||||
supervisor → actions/pending/<id>.json (unchanged)
|
||||
operator → actions/approved/<id>.json (unchanged, HITL)
|
||||
executor (vps) → actions/deploy/<node>/<id>.json ← dispatch only, no execution
|
||||
deploy-runner ← rsync-pull (node initiates; VPS never connects to a node)
|
||||
→ scripts/deploy/deploy-service.sh --force-recreate
|
||||
→ events/<node>/evt-…-action_result-….json → rsync-push
|
||||
executor → completed / failed (via _reconcile_running_actions)
|
||||
```
|
||||
|
||||
Design points, and why:
|
||||
|
||||
- **Host-level, not a container.** Compose resolves relative bind mounts and the
|
||||
project name against the filesystem of whatever runs it. In a container those
|
||||
resolve to container paths that the daemon then interprets as host paths — a
|
||||
silent way to produce broken mounts, or to land in a different Compose project
|
||||
where `--remove-orphans` deletes the running stack. On the host it behaves
|
||||
exactly like a human `deploy.sh`.
|
||||
- **Independent of node-agent** (own rsync pull, own result push) so it can
|
||||
redeploy `node-agent` itself — the solaria case, where node-agent has been
|
||||
docker-blind for weeks.
|
||||
- **Separate inbox** from `actions/dispatch/<node>/`: node-agent deletes and
|
||||
failure-reports every file in its own inbox that is not `container_restart`.
|
||||
- **No `git pull`.** A redeploy reconciles the node to the checkout it already
|
||||
has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
|
||||
- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain
|
||||
`up -d` is a no-op when config is unchanged — exactly the unhealthy-container
|
||||
case; building on the 4 GiB VPS mid-incident is an OOM risk.
|
||||
|
||||
Every action is validated before anything runs (`action.py`): type must be
|
||||
`redeploy`, `node` must match this node, the service name must be a plain
|
||||
kebab-case name, the service must be listed in `hosts/<node>/services.yaml`, and
|
||||
its compose file must exist. Nothing from the action payload is ever executed —
|
||||
only a validated service *name* reaches the deploy script. Actions are
|
||||
idempotent (marker in `state/processed-deploy-actions/`), single-instance
|
||||
(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the
|
||||
executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the
|
||||
control plane gives up).
|
||||
|
||||
Loading…
Reference in a new issue