diff --git a/kb/audits/redeploy-2026-08-05.md b/kb/audits/redeploy-2026-08-05.md new file mode 100644 index 0000000..ae7c314 --- /dev/null +++ b/kb/audits/redeploy-2026-08-05.md @@ -0,0 +1,667 @@ +--- +okf: "0.1" +type: audit +visibility: private +status: active +updated: 2026-08-05 +as_of: 2026-08-05 +links: [] +--- + +# Recon — ścieżka remediacji `redeploy` (2026-08-05) + +Read-only recon. Ground truth: `kb/subsystems/recon-multiagent.md` (A, D14, D15) +and `kb/audits/czujniki-2026-07-30.md` — both formerly at +`docs/architecture/RECON-*`, migrated into `kb/` by `00de810` / `9f77a72`. +Source read at `task/recon-redeploy` cut from master @ `4ecbdbb`. Runtime +evidence collected 2026-08-05 ~09:40 UTC from vps and piha over ssh; solaria is +the host this recon ran on (local reads). + +--- + +## FINDING FIRST + +**The break this recon was commissioned to characterize was already repaired in +the repo — two days after the task premise was written — and the repair has +never been deployed. `redeploy` is still dead in production, but for a +completely different reason than stated.** + +The task brief describes master as: *"the executor calls deploy-node.sh with +args it ignores, on a path that doesn't exist inside the container."* That was +true of master until 2026-08-03. It is **not** true of master today: + +- `79bfe8c` (2026-08-03) `feat(control-plane): host-side deploy runner — fix + broken redeploy path` and `da151fc` (2026-08-03) `fix(control-plane): redeploy + wykonywalny — dispatch do host-side deploy-runnera` replaced the + `deploy-node.sh` call with a dispatch-file handoff to a new host-level systemd + job, `jobs/deploy-runner/`. +- Master's `executor.py:125-144` no longer references `deploy-node.sh` at all; + `REPO_ROOT` survives only as a decorative env var (`executor.py:30-33`). +- The design decision, install runbook and open runtime steps are already + written up: `kb/decisions/deploy-runner-uzasadnienie.md`, + `kb/runbooks/deploy-runner-install.md`, + `kb/decisions/backlog-deploy-runner-instalacja.md`. + +What is actually broken today is a **deploy gap in three places at once**: + +| Layer | Repo (master @ `4ecbdbb`) | Runtime (2026-08-05) | +|---|---|---| +| executor code | dispatches to `actions/deploy//` | container built **2026-07-22**, still runs the `deploy-node.sh` code | +| `actions/deploy/` inbox | created by `_ensure_dirs` (`executor.py:79`) | **does not exist** on vps | +| deploy-runner on nodes | `jobs/deploy-runner/` + systemd units | **not installed** on vps, piha or solaria | + +So the correct framing for the repair session is not "design a fix" — the fix is +designed, written, and 248-tests green. It is **"deploy the fix and prove it +end-to-end"**, plus resolving the four genuine design questions the repair left +open (see FIX SHAPE at the end), the sharpest of which is that **the only event +type that actually reaches a `redeploy` incident in live world state targets the +one service the new runner deliberately refuses to deploy.** + +Secondary finding, unrelated to redeploy but discovered en route: the executor +container has **no `ssh` binary** (verified below), so `disk_cleanup` — +the one action type that still shells out to ssh (`executor.py:392-400`) — is +dead by the same class of defect. It has never been exercised. + +--- + +## A. GENERATION + +### A1. `event_type → action_type` map as implemented on master @ `4ecbdbb` + +Unchanged from recon D15 except for the `healthcheck_failed` move in `fbf165f`, +which the brief asked me to verify. **Verified: `healthcheck_failed` routes to +`container_restart`, not `redeploy`.** + +`supervisor.py:52`: + +```python +CONTAINER_RESTART_TRIGGERS = {"containers_not_running", "healthcheck_failed"} +``` + +**Path 1 — world-state drift** (`supervisor.py:334-395`, dispatch at +`:401-477`). The supervisor reads `world/{services,incidents}.json`, never raw +events. `trigger_type` comes from the incident the observer opened +(`supervisor.py:317-327` ← `observer.py:868`): + +| Drift / incident `trigger_type` | Action | Where | +|---|---|---| +| `containers_not_running` | `container_restart` | `supervisor.py:52,410,423` | +| `healthcheck_failed` | `container_restart` | `supervisor.py:52` (added `fbf165f`, 2026-07-29) | +| `service_unhealthy` | `redeploy` | falls through `:447` | +| `deployment_failed` | `redeploy` | falls through `:447` | +| service absent from world state (`missing_service`, `trigger_type: None`) | `redeploy` | `supervisor.py:353-360` → `:447` | +| node `disk_pressure == high` | `disk_cleanup` | `supervisor.py:377-381,479+` | + +Only services listed in `hosts//services.yaml` are considered +(`supervisor.py:350`); dormant nodes are skipped (`supervisor.py:342,378`). + +**Path 2 — direct event-file routing** (`_process_ha_events`, `supervisor.py:395, +~630-660`) is unchanged and generates only `container_restart` (homeassistant, +shadow-downgraded to `alert_only`) and `alert_only`. **Path 2 never generates +`redeploy`.** + +So: **three producers of `redeploy` — `service_unhealthy`, `deployment_failed`, +`missing_service` — all on path 1.** Section D9 shows only one of them fires in +practice, and it fires at the wrong target. + +### A2. Fields on a `redeploy` action + +Built at `supervisor.py:453-467`, verbatim: + +```python +action = { + "action_id": action_id, # f"redeploy-{node}-{service}" (:413) + "timestamp": time.time(), + "type": "redeploy", + "node": node, + "service": service, + "risk_level": "guarded", + "confidence": 0.9, + "description": f"Redeploy {service} on {node} due to {drift['type']}", + "status": "pending", + "payload": { + "reason": drift["type"], + "svc_key": drift["svc_key"], + }, +} +``` + +Written to `actions/pending/.json` (`supervisor.py:469-471`). + +**There is no `container_name`, no path, no compose reference, and no args +field.** Contrast `container_restart` (`supervisor.py:428-446`), which carries +`container_name` resolved via `_get_container_name(service)`. A redeploy action +carries exactly two addressing facts: `node` and `service`. Everything else — +which compose file, which override, which repo — is resolved on the node at +execution time from `service` alone. + +Live example, the one non-chelsty pending redeploy on vps: + +```json +{ "action_id": "redeploy-vps-gokapi", "timestamp": 1783615889.1762547, + "type": "redeploy", "node": "vps", "service": "gokapi", + "risk_level": "guarded", "confidence": 0.9, + "description": "Redeploy gokapi on vps due to missing_service", + "status": "pending", + "payload": { "reason": "missing_service", "svc_key": "vps/gokapi" } } +``` + +--- + +## B. EXECUTION — why it's dead + +### B3. The redeploy handler + +**Two answers, because repo and runtime disagree.** + +**(a) Master @ `4ecbdbb` — `executor.py:125-144`.** No command is built at all. +The executor writes a dispatch file and returns: + +```python +if action_type == "redeploy": + if not node or not service: + success = False + error_msg = (f"redeploy requires both node and service " + f"(node={node!r}, service={service!r})") + else: + self._dispatch_redeploy(action_id, node, service) + return # stays in running/ until action_result +``` + +`_dispatch_redeploy` (`executor.py:238-267`) writes +`actions/deploy//.json` with `{action_id, type, node, service, +dispatched_at}`. Deliberately a **different inbox** from `container_restart`'s +`actions/dispatch//`, because node-agent deletes and failure-reports +anything in its own inbox that is not `container_restart` +(`executor.py:24-29` ← `node_agent.py:108,1011-1018`). + +**(b) The executor actually running on vps** — image created +`2026-07-22T16:13:29Z`, i.e. 12 days before the fix. Its `/app/src/executor.py` +lines 106-119, read out of the live container: + +```python +if action_type == "redeploy": + # Full service redeploy via the repo deploy script + cmd = [ + str(REPO_ROOT / "scripts" / "deploy" / "deploy-node.sh"), + node, + service + ] + logger.info(f"Running command: {' '.join(cmd)}") + result = subprocess.run(cmd, capture_output=True, text=True, cwd=str(REPO_ROOT)) +``` + +`REPO_ROOT` = `/repo` (`os.getenv("REPO_ROOT", "/repo")`, running line 24; +master `executor.py:33`). Synchronous: `returncode == 0` → completed, else +failed with `stderr or stdout`. + +### B4. Working directory / repo path, and does it exist in the container + +The repo **is** reachable inside the executor container — the claim "a path that +doesn't exist inside the container" needs one level of precision. Two different +paths are involved and only the second is missing. + +`docker inspect control-plane-executor` (vps): + +``` +Image=sha256:7603f09d… Created=2026-07-22T16:13:29.905995261Z +/opt/homelab -> /opt/homelab (rw) +/home/oskar/homelab-codex-ws -> /repo (rw) +/var/run/docker.sock -> /var/run/docker.sock (rw) +REPO_ROOT=/repo RUNTIME_PATH=/opt/homelab +``` + +Inside the container: + +``` +HOME=/home/homelab +/repo/scripts/deploy/deploy-node.sh ← EXISTS +ls: cannot access '/home/homelab/homelab-codex-ws': No such file or directory +git: MISSING docker: MISSING rsync: MISSING ssh: MISSING +bash: /usr/bin/bash python3: /usr/local/bin/python3 flock: /usr/bin/flock +``` + +So `cwd=/repo` is valid and the script is found and executed. It then fails on +**its own** internal path: `deploy-node.sh:8` hardcodes +`REPO_PATH="${HOME}/homelab-codex-ws"` = `/home/homelab/homelab-codex-ws`, which +does not exist, and `deploy-node.sh:18-21` exits 1. + +Reproduced read-only inside the live container with the exact argv the running +executor would use: + +``` +rc= 1 +stdout= --- Starting Deployment on 1CF38E89F150 --- +Error: Repository not found at /home/homelab/homelab-codex-ws +``` + +Note the hostname: `1CF38E89F150`, the container ID. Even with the repo present, +`deploy-node.sh:27-57` resolves the host directory by matching `os_hostname` +against `hosts/*/host.yaml` and would fail with *"No host directory found for +1cf38e89f150"* — the executor would have deployed **its own container's** +service set, never the action's `node`. And behind that, two more hard stops: +`git pull` (`deploy-node.sh:25`) and the compose invocation both need binaries +the image does not have. + +**Four independent blockers, in the order they would be hit:** missing +`${HOME}/homelab-codex-ws` → hostname resolves to a container ID → no `git` → no +`docker`. The image mount is *not* one of them. + +Note also `../..:/repo:ro` in `services/control-plane/docker-compose.yml:36,55,86` +— master mounts the repo read-only. The running container has it **rw**, another +marker that the container predates current master. + +### B5. `deploy-node.sh` arg parsing, and the call chain + +**`deploy-node.sh` has no argument parsing whatsoever.** There is no `getopts`, +no `case "$1"`, no `$1`/`$2` reference anywhere in its 115 lines. It is a +zero-argument script: it derives everything from `${HOME}` and `hostname` +(`:8,12,13`), and deploys the **whole service set** of whatever host it runs on +(`:60-78` reads `services.txt` / `services.yaml`, `:95-113` loops over all of +them). The `node` and `service` the old executor passed were silently discarded +by the shell. + +Per-service deploy exists, but in a **different script**: +`scripts/deploy/deploy-service.sh`, extracted from `deploy-node.sh` by the +2026-08-03 fix. It takes `--repo`, `--host-dir`, `--service` plus optional +`--build-if-needed` / `--force-recreate` / `--remove-orphans` +(`deploy-service.sh:51-62`), validates the service name against +`^[a-z0-9][a-z0-9._-]{0,63}$` (`:72`), and is the single shared compose +invocation for both callers — deliberately, because a project-name mismatch plus +`--remove-orphans` is what wiped the control-plane on vps on 2026-06-25 +(`deploy-service.sh:6-13`, `deploy-node.sh:86-92`). + +**Call chains, with the host and container each link runs on:** + +*Human deploy (SATURN-side dispatcher — confirms the solaria-gid-fix session's +model):* + +``` +operator @ SATURN: scripts/deploy/deploy.sh [host shell, saturn] + preflight: must be on master, clean tree (deploy.sh:69-82) + → ssh oskar@ 'cd ~/homelab-codex-ws && git pull + && ./scripts/deploy/deploy-node.sh' (deploy.sh:199-202) + → deploy-node.sh [host shell, target node] + → deploy-service.sh --build-if-needed --remove-orphans × every service + (deploy-node.sh:100-105) + (target == control-plane takes a different branch: + deploy-control-plane.sh --ssh, deploy.sh:194-196) +``` + +*Agent redeploy, master @ `4ecbdbb`:* + +``` +supervisor [container control-plane-supervisor, vps] + → /opt/homelab/actions/pending/redeploy--.json +operator (Telegram / operator-ui) → approved/ +executor [container control-plane-executor, vps] + → /opt/homelab/actions/deploy//.json (executor.py:238-267) +deploy-runner [HOST systemd oneshot on , NOT a container] + → rsync-pull inbox from vps (deploy-runner.sh pull_actions) + → python3 action.py validate (5 rules, action.py:78-125) + → deploy-service.sh --repo … --host-dir … --service … --force-recreate + → action.py emit-result → evt---action_result-.json + → rsync-push to vps events/ +executor._reconcile_running_actions() (executor.py:269-318) + → completed/ or failed/ (timeout REDEPLOY_TIMEOUT_SECS=900, executor.py:55) +``` + +The runner runs on the **host**, not in a container, on purpose: compose then +resolves relative bind mounts and the project name exactly as a human deploy +does (`deploy-runner.sh:26-31`). It also **never runs `git pull`** — a redeploy +reconciles the node to the checkout it already has; shipping code stays a human +`deploy.sh` (`deploy-runner.sh:32-33`). + +### B6. Concrete failure mode today + +**Nothing fires, and nothing ever has. No redeploy has ever been attempted, +approved, or executed — fleet-wide, ever.** + +Action pool on vps, 2026-08-05: + +``` +pending 18 approved 0 running 0 completed 0 failed 0 rejected 0 cancelled 18 +dispatch 1 deploy: ls: cannot access '/opt/homelab/actions/deploy': No such file or directory +``` + +`completed 0 / failed 0` is the whole story: **not one action has ever reached a +terminal state.** 18 redeploy actions exist on disk (16 cancelled, 2 pending); +none was ever approved, so the broken handler was never entered. + +The two live ones: + +``` +/opt/homelab/actions/pending/redeploy-vps-gokapi.json reason: missing_service +/opt/homelab/actions/pending/redeploy-chelsty-infra-ha-diag-agent.json reason: missing_service +``` + +The chelsty one targets a node dead since 2026-06-01 and is already flagged for +manual removal (`kb/subsystems/recon-multiagent.md`, "Runbook — stale chelsty +action"). + +The executor's **entire log since 2026-07-22** is 144 lines and contains three +action executions, all hand-injected tests, zero supervisor-generated actions, +zero redeploys, and nothing at all after 2026-07-23: + +``` +2026-07-22 16:13:41 - Starting executor loop +2026-07-22 16:32:41 - Executing action: test-restart-piha-node_exporter-2 +2026-07-22 16:32:41 - Dispatched container_restart … to node-agent on piha +2026-07-22 16:37:42 - ERROR - Action … timed out: Timed out after 300s waiting for node-agent on 'piha' +2026-07-23 11:54:05 - Executing action: test-e2e-1784807638 +2026-07-23 11:54:05 - ERROR - Failed to move test-e2e-1784807638 to running: Expecting value: line 1 column 38 + …(same pair repeating every 10 s for 11 minutes — malformed JSON never leaves approved/)… +2026-07-23 12:05:25 - Executing action: test-e2e-b +2026-07-23 12:05:25 - Dispatched container_restart test-e2e-b (container=node_exporter) to node-agent on piha +2026-07-23 12:05:56 - Action test-e2e-b completed +``` + +**Why nothing fires**, in the order the funnel closes: + +1. `redeploy` needs `service_unhealthy`, `deployment_failed`, or + `missing_service`. Post-`fbf165f`, the two high-volume signals + (`containers_not_running` 2,490; `healthcheck_failed` 3,376 on piha) both go + to `container_restart`. Live incident census on vps confirms it — + `world/incidents.json` holds 3 incidents total, `trigger_type: + {containers_not_running: 2, healthcheck_failed: 1}`. **Zero + redeploy-producing incidents are open.** +2. `missing_service` still fires (it needs no incident at all), which is exactly + what the 2 pending redeploys are. But it is dedup-suppressed forever by its + own pending file (`supervisor.py:418-421`) — `redeploy-vps-gokapi` has held + its ID since 2026-07-19. +3. Nothing gets approved. `approved 0`, and the executor reads only `approved/` + (`executor.py:94-95`). +4. Even on approval the running executor would fail as in B4 — but this has + never actually happened, so there is **no failed/ artefact, no stack trace, + and no "Running command: …deploy-node.sh" log line to point at.** The + evidence for the break is the reproduction in B4, not a production incident. + +--- + +## C. THE HEALTHY PATH FOR COMPARISON + +### C7. `container_restart` traced end to end + +Correction to the premise: the two `action_result` events of 2026-07-23 are +**manual e2e tests, not real remediations**. Payloads: + +```json +{"action_id": "test-restart-piha-node_exporter-2", "success": true, "error": "", "node": "piha"} +{"action_id": "test-e2e-b", "success": true, "error": "", "node": "piha"} +``` + +Both hand-crafted IDs. A supervisor-generated ID would be +`container-restart-piha-node_exporter` (`supervisor.py:411`). And only **one** +of the two round-tripped cleanly: `test-restart-piha-node_exporter-2` was +dispatched 2026-07-22 16:32 UTC, timed out at 300 s, and its `action_result` +only landed 2026-07-23 11:22 UTC — ~19 h later, after the executor had already +filed it failed. `test-e2e-b` completed in 31 s. **So the reference path has +exactly one clean end-to-end demonstration, and one latency failure, and has +never run from a real incident.** + +Path, verbatim: + +1. `executor._execute_action` moves `approved/.json` → `running/` + (`executor.py:104-112`), then `executor.py:146-157`: + `container_name = data.get("container_name") or service` → + `_dispatch_container_restart(...)` → `return` (stays in `running/`). +2. `_dispatch_container_restart` (`executor.py:209-236`) writes + `actions/dispatch//.json` = `{action_id, type, node, + service, container_name, dispatched_at}`. +3. node-agent on the node rsync-pulls its own subdir with + `--remove-source-files` (`node_agent.py:919-960`); on vps it reads the path + directly (`:933`). +4. `_execute_dispatched_action` (`node_agent.py:981-1051`) applies four gates — + idempotency (`:999`), node scoping (`:1003`), type whitelist + `ALLOWED_DISPATCH_ACTION_TYPES = {"container_restart"}` (`:108, :1011`), + self-restart guard `{"node-agent"}` (`:114, :1027`) — then + `self.docker_client.containers.get(container_name).restart()` (`:1043-1044`). +5. `_report_action_result` (`node_agent.py:1053+`) emits an `action_result` + event; the existing rsync ships it to vps. +6. `executor._reconcile_running_actions` (`:269-318`) matches it via + `_find_action_result` (`:320-356`) and moves the action to + `completed/`/`failed/`, or fails it after `ACTION_TIMEOUT_SECS=300`. + +### C7b. Structural differences between the two handlers + +On master they are now **deliberately near-identical** — steps 1, 2, 5 and 6 are +the same code, `DISPATCHED_ACTION_TYPES = {"container_restart", "redeploy"}` +(`executor.py:60`), and `_reconcile_running_actions` handles both with only the +timeout differing (`executor.py:299-302`). What remains different: + +| | `container_restart` | `redeploy` | +|---|---|---| +| Inbox | `actions/dispatch//` | `actions/deploy//` (must not share — `executor.py:24-29`) | +| Executor on the node | **node-agent**, a container, own docker socket | **deploy-runner**, host systemd oneshot + timer, no container | +| Deployed how | `deploy.sh` / compose, already everywhere | manual per-node systemd install, **nowhere yet** | +| Poll cadence | `CHECK_INTERVAL=60` | `OnUnitActiveSec=60s` (matched on purpose) | +| Timeout | `ACTION_TIMEOUT_SECS=300` | `REDEPLOY_TIMEOUT_SECS=900` | +| Payload | carries `container_name` | carries only `service` | +| Validation | 4 gates in `node_agent.py` | 5 gates in `action.py:78-125`, incl. *service must be in `hosts//services.yaml`* and *compose file must exist* | +| Idempotency | `_already_processed` marker | `state/processed-deploy-actions/.done` (`deploy-runner.sh:57,151-155`) | +| Concurrency | none needed | `flock` on `state/deploy-runner.lock` | +| Can act on node-agent itself | **no** (self-guard) | **yes** — deliberate; solaria's node-agent has been docker-blind for weeks (`deploy-runner.sh:26-31`) | +| Can act on control-plane / stability-agent | yes | **no** — `deploy-service.sh:93-96` exits 3, runner reports failure with an explanation | + +The last row is the important asymmetry: see D9. + +### C8. How the executor reaches nodes + +**It doesn't, and it cannot.** Verified inside the live container: `ssh: MISSING`, +`rsync: MISSING`, `git: MISSING`, `docker: MISSING`. There is no ssh client, no +key, no known_hosts. This is by design — `executor.py:200-207` and +`kb/phases/backlog.md` "Remediacja floty bez SSH": the VPS never initiates a +connection to a node. Every remote action is a **file the node comes and +collects**, over the ssh/rsync channel node-agent (and now deploy-runner) +already owns in the other direction. The executor's only outputs are writes into +the shared `/opt/homelab` mount. + +Two consequences: + +- **`disk_cleanup` is dead by the same defect.** `_execute_disk_cleanup` + (`executor.py:358-407`) builds `["ssh", *SSH_OPTIONS, f"{SSH_USER}@{node}", + …]` and calls `subprocess.run`. With no `ssh` binary this raises + `FileNotFoundError`, caught by `executor.py:175-177`, and the action fails with + `[Errno 2] No such file or directory: 'ssh'`. Never observed because + `failed/` is empty — no `disk_cleanup` was ever approved either. Out of scope + here; flagged as a follow-up. +- **Solaria's self-ssh problem is irrelevant to the executor.** `deploy.sh` runs + on SATURN and ssh's *to* the target, which is why it cannot deploy solaria + from solaria. The executor never ssh's anywhere; the connection is always + node → vps, initiated by the node. Solaria's node-agent already ships events + to vps over that channel daily, so the transport is proven. **Solaria's real + blocker is different and unrelated:** its node-agent has no docker socket + access (`group_add: "996"` fixed in repo @ `ddae57c`, still not deployed — + running container shows `GroupAdd=[999]`), which breaks `container_restart` + there. `redeploy` is unaffected, because deploy-runner runs on the host and + never touches node-agent. + +--- + +## D. WHAT REDEPLOY IS FOR + +### D9. When redeploy *should* fire vs `container_restart` + +The semantic split the brief proposes is the one the code already encodes +(`supervisor.py:36-52, 423-426, 447-452`): **`container_restart` = the container +exists and a restart plausibly heals it; `redeploy` = the container is absent, +or its image/compose/env no longer matches the repo, so it must be re-created +from the manifest.** `deploy-service.sh --force-recreate` is exactly that and +nothing more (`deploy-service.sh:111-130`: `docker compose -f … [-f override] +[--env-file] up -d --force-recreate`, **no `--build`**, `deploy-service.sh:113-118` +— a redeploy is a reconcile, not a code ship, and building on the 4 GiB vps +mid-incident is an OOM risk). + +Candidate event types for the redeploy side, and whether each actually reaches +an incident today: + +| Candidate | Opens an incident? | Live volume | Verdict | +|---|---|---|---| +| `missing_service` (drift, not an event) | **n/a — needs no incident**; `supervisor.py:353-360` raises it purely from desired-vs-actual | 2 pending actions right now | **The only redeploy producer that actually fires.** Semantically the perfect fit: the container is gone, a restart is impossible. | +| `service_unhealthy` | **yes** — `observer.py:771,792` | 34 events, vps only | **Fires, but is a trap — see below.** | +| `deployment_failed` | **yes** — `observer.py:838-842` `_handle_deployment_failure` | **0 events fleet-wide** | Only emitter is `scripts/lib/log.sh:35,43` (deploy-time shell logging). Never reaches the vps event store. Dead in practice. | +| `containers_not_running` | yes, `observer.py:771` | 2,490 piha / 3 solaria / 1 vps | Correctly on `container_restart` — container exists. | +| `healthcheck_failed` | yes, `observer.py:771` | 3,376 piha | Correctly on `container_restart` since `fbf165f`. Escalating repeat failures to redeploy is a candidate, but needs a retry policy that does not exist. | +| `container_restarting` / `container_state_unexpected` | **no** — observational only, `observer.py:793-814` | 2 solaria | Deliberately not remediated. | +| stability-agent's `containers_not_running` | **no** — `service=None` fails the `if service and service != "all"` guard at `observer.py:747`, so the incident branch is never reached | — | Known-broken producer (recon D15, `kb/audits/czujniki-2026-07-30.md`). | + +**The trap.** `service_unhealthy` has exactly one live emitter in the whole +fleet: `node_agent.py:1104-1115`, which probes +`http://localhost:18180/summary` and hardcodes `service="control-plane"`. So +every `service_unhealthy` incident that could ever produce a `redeploy` targets +`vps/control-plane` — and `control-plane` is one of only two services with its +own `deploy-local.sh` (`services/control-plane/`, `services/stability-agent/`), +which `deploy-service.sh:93-96` refuses with exit 3, which the runner turns into +a **failed** action: *"control-plane owns its deploy path +(services/control-plane/deploy-local.sh) — needs an operator deploy, not an +automated redeploy"* (`deploy-runner.sh:179-181`). That refusal is correct — a +containerised control plane redeploying itself mid-incident is not something to +automate — but it means **the fixed redeploy path, once deployed, will still +produce zero successful incident-driven redeploys.** Only `missing_service` +will ever succeed. + +Also worth stating plainly: **`redeploy` cannot repair a `service_unhealthy` +caused by bad code**, because the path deliberately does not `git pull` +(`deploy-runner.sh:32-33`) and deliberately does not `--build`. It repairs +*drift* — a container removed, a stale env file, a config change already in the +node's checkout — not *regression*. + +### D10. Preconditions for a working redeploy + +Per link in the chain, on master's architecture. Current status verified +2026-08-05. + +**On the vps (control plane):** + +| # | Precondition | Where | Status | +|---|---|---|---| +| 1 | executor running post-`da151fc` code | `executor.py:125-144` | ❌ container built 2026-07-22, runs `deploy-node.sh` code | +| 2 | `/opt/homelab/actions/deploy//` writable | `executor.py:79` (`_ensure_dirs`) | ❌ dir absent — created automatically by the new executor | +| 3 | repo checkout current | `/home/oskar/homelab-codex-ws` | ⚠️ at `efd8d2f`, one commit behind master `4ecbdbb`; **has** the fix (`jobs/deploy-runner/` dated Aug 3 17:28) | +| 4 | *not* needed: ssh, git, docker CLI in the executor image | — | ✅ by design | + +**On each target node:** + +| # | Precondition | Where | vps | piha | solaria | +|---|---|---|---|---|---| +| 5 | repo checkout at `REPO_PATH` | `deploy-runner.sh:73` | ✅ `efd8d2f` | ✅ `4ecbdbb` | (main checkout, not inspected) | +| 6 | `hosts//` exists in that checkout | `deploy-runner.sh:74` | ✅ | ✅ | ✅ | +| 7 | `scripts/deploy/deploy-service.sh` present + executable | `deploy-runner.sh:75-76` | ✅ Aug 3 | ✅ Aug 4 | — | +| 8 | `jobs/deploy-runner/` systemd unit + timer installed & enabled | `kb/runbooks/deploy-runner-install.md` | ❌ | ❌ | ❌ | +| 9 | `/opt/homelab/config/deploy-runner/env` with `NODE_NAME` (**empty `VPS_EVENTS_HOST` on vps**) | `env.example` | ❌ | ❌ | ❌ | +| 10 | service user in `docker` group | runbook | — | — | ✅ (`oskar` ∈ `docker` gid 996) | +| 11 | `python3` + PyYAML, `rsync`, `flock` | `action.py:60-64`, `deploy-runner.sh:70-76` | — | — | — | +| 12 | ssh key reaching vps (remote nodes only) | `deploy-runner.sh:62-66` | n/a | ✅ (node-agent already ships events) | ✅ | + +**Per-action, at execution time** (`action.py:78-125` — all five must hold or the +runner reports `failed` without deploying): + +| # | Rule | Note | +|---|---|---| +| 13 | `type == "redeploy"` | | +| 14 | `action["node"] == NODE_NAME` | defense in depth; inbox is already node-scoped | +| 15 | `action_id` / `service` match strict name patterns | no path traversal | +| 16 | **service listed in `hosts//services.yaml`** | repo desired state is the authority; a stale dispatch cannot pull an arbitrary stack | +| 17 | `services//docker-compose.yml` exists in the repo | | +| 18 | *(not a rule, an outcome)* service must **not** have `services//deploy-local.sh` | else exit 3 → failed-with-explanation | + +Checked against the one actionable pending action, `redeploy-vps-gokapi`: rule +16 ✅ (`hosts/vps/services.yaml:63`), rule 17 ✅ (`services/gokapi/docker-compose.yml`), +rule 18 ✅ (no `deploy-local.sh`). **`gokapi` is a valid end-to-end target the +moment the runner is installed on vps.** + +Also required for the loop to *close*: the result event must land in +`/opt/homelab/events//` on the vps in `node_agent.emit_event()`'s exact +format, because `executor._find_action_result` (`:320-356`) globs +`evt-*-action_result-*.json` and parses the unix timestamp out of the +**filename**. `action.py:emit_result` reproduces that byte-for-byte and adds +`payload.source = "deploy-runner"`. + +--- + +## FIX SHAPE — DECISIONS NEEDED + +The transport and script layers are decided and written. What is left is one +sequencing decision and four genuine design choices the 2026-08-03 work left +open. + +**1. Deploy vs re-derive.** The repair is `79bfe8c`/`da151fc` on master, 248 +tests green, unexercised at runtime. + +- *Deploy as written* — three runtime steps (`deploy.sh control-plane`; install + `jobs/deploy-runner/` on vps + piha + solaria per + `kb/runbooks/deploy-runner-install.md`; e2e on a benign piha service, then + `redeploy-vps-gokapi`). Fastest to a working loop; accepts a design nobody has + yet run. +- *Re-derive first* — treat the shipped design as a proposal, resolve 2–5 below, + then deploy once. Slower; avoids installing systemd units on three hosts twice. + +Note that 2–5 are answerable either way, but their answers are cheaper to apply +**before** the units are on three nodes. + +**2. Where the deploy actually runs: node-side runner vs saturn-side dispatcher.** +The shipped design is node-side (`deploy-runner` on each node, executor never +connects out). + +- *Node-side (as shipped)*: preserves "remediacja floty bez SSH"; works on + solaria despite its broken node-agent socket and despite the + self-ssh problem; every node needs a host-level systemd install and its own + repo checkout, and the install is not covered by `deploy.sh` — it drifts + silently. +- *Executor → ssh SATURN → `deploy.sh `*: reuses the human path exactly, + one install point instead of N, and inherits `deploy.sh`'s preflight gates. + But it inverts the no-SSH decision, puts a key to SATURN inside a + vps container, makes SATURN a runtime dependency of remediation (it is a + workstation, not a server), and `deploy.sh` is whole-node only — it cannot + express "redeploy just gokapi". +- *Executor → ssh direct to node → `deploy-service.sh`*: fewest moving parts, no + runner, no timer, no rsync. Requires an ssh client + fleet key in the executor + container — the exact thing the architecture forbids — and reintroduces the + self-ssh and reachability problems the pull model sidesteps. + +**3. Per-service vs whole-node redeploy.** Shipped: per-service, and the action +payload carries no node-wide option. + +- *Per-service*: minimal blast radius; matches the action's semantics + (`missing_service` names one service); a genuinely node-wide failure needs N + actions and N approvals. +- *Whole-node (i.e. `deploy-node.sh`, what the old executor accidentally + attempted)*: one action heals a node; but it touches every service including + healthy ones, runs `--build-if-needed`, and `--remove-orphans` is in play — the + 2026-06-25 control-plane wipe is precedent for how that ends. + +**4. How args flow — how much the action carries vs how much the node resolves.** +Shipped: the action carries `{node, service}` only; everything else (compose +path, override, env-file, force-recreate) is resolved on the node by +`deploy-service.sh` from repo state. + +- *Thin action (as shipped)*: the repo is the single authority; an old or forged + dispatch cannot smuggle a path or a flag; but the operator approving the action + cannot see or influence what will actually run, and `--build`/`--pull` are not + expressible even when they are what is needed. +- *Fat action*: supervisor stamps compose path, override, flags into the payload + — visible at approval time, tunable per incident; every field becomes attack + surface the runner must re-validate, and payloads can go stale against the + node's checkout. + +**5. What happens to `service_unhealthy` → `control-plane` (D9).** Once the +runner is installed, the only incident-driven redeploy producer will target the +one service the runner refuses. + +- *Leave it*: the failed action with its explanatory message **is** the alert; + costs one failed action per control-plane outage and burns the dedup ID. +- *Downgrade to `alert_only` at the supervisor*: honest — nothing automated can + fix it — but drops the drift out of the remediation view. +- *Give control-plane a self-redeploy path* (analogous to + `deploy-control-plane.sh`): closes the loop, and lets a broken control plane + attempt to repair itself with its own hands. Highest risk on the list. +- *Widen `service_unhealthy` beyond the hardcoded control-plane probe* + (`node_agent.py:1104-1115`): would give redeploy a real, non-degenerate input + set for the first time — but that is a monitoring change, not a redeploy + change, and it interacts with the `healthcheck_failed` routing decided in + `fbf165f`. + +**Out of scope here, found en route:** the executor's `disk_cleanup` handler +shells out to `ssh`, which does not exist in the image (C8) — same class of +defect as the original redeploy break, never observed because no `disk_cleanup` +was ever approved. Worth its own task.