homelab-codex-ws/kb/audits/redeploy-2026-08-05.md

okf	type	visibility	status	updated	as_of	links
0.1	audit	private	active	2026-08-05	2026-08-05

Recon — ścieżka remediacji redeploy (2026-08-05)

Read-only recon. Ground truth: kb/subsystems/recon-multiagent.md (A, D14, D15) and kb/audits/czujniki-2026-07-30.md — both formerly at docs/architecture/RECON-*, migrated into kb/ by 00de810 / 9f77a72. Source read at task/recon-redeploy cut from master @ 4ecbdbb. Runtime evidence collected 2026-08-05 ~09:40 UTC from vps and piha over ssh; solaria is the host this recon ran on (local reads).

---

FINDING FIRST

The break this recon was commissioned to characterize was already repaired in the repo — two days after the task premise was written — and the repair has never been deployed. redeploy is still dead in production, but for a completely different reason than stated.

The task brief describes master as: "the executor calls deploy-node.sh with args it ignores, on a path that doesn't exist inside the container." That was true of master until 2026-08-03. It is not true of master today:

• 79bfe8c (2026-08-03) feat(control-plane): host-side deploy runner — fix broken redeploy path and da151fc (2026-08-03) fix(control-plane): redeploy wykonywalny — dispatch do host-side deploy-runnera replaced the deploy-node.sh call with a dispatch-file handoff to a new host-level systemd job, jobs/deploy-runner/.
• Master's executor.py:125-144 no longer references deploy-node.sh at all; REPO_ROOT survives only as a decorative env var (executor.py:30-33).
• The design decision, install runbook and open runtime steps are already written up: kb/decisions/deploy-runner-uzasadnienie.md, kb/runbooks/deploy-runner-install.md, kb/decisions/backlog-deploy-runner-instalacja.md.

What is actually broken today is a deploy gap in three places at once:

Layer	Repo (master @ 4ecbdbb)	Runtime (2026-08-05)
executor code	dispatches to actions/deploy/<node>/	container built 2026-07-22, still runs the deploy-node.sh code
actions/deploy/ inbox	created by _ensure_dirs (executor.py:79)	does not exist on vps
deploy-runner on nodes	jobs/deploy-runner/ + systemd units	not installed on vps, piha or solaria

So the correct framing for the repair session is not "design a fix" — the fix is designed, written, and 248-tests green. It is "deploy the fix and prove it end-to-end", plus resolving the four genuine design questions the repair left open (see FIX SHAPE at the end), the sharpest of which is that the only event type that actually reaches a redeploy incident in live world state targets the one service the new runner deliberately refuses to deploy.

Secondary finding, unrelated to redeploy but discovered en route: the executor container has no ssh binary (verified below), so disk_cleanup — the one action type that still shells out to ssh (executor.py:392-400) — is dead by the same class of defect. It has never been exercised.

---

A. GENERATION

A1. event_type → action_type map as implemented on master @ 4ecbdbb

Unchanged from recon D15 except for the healthcheck_failed move in fbf165f, which the brief asked me to verify. Verified: healthcheck_failed routes to container_restart, not redeploy.

supervisor.py:52:

CONTAINER_RESTART_TRIGGERS = {"containers_not_running", "healthcheck_failed"}

Path 1 — world-state drift (supervisor.py:334-395, dispatch at :401-477). The supervisor reads world/{services,incidents}.json, never raw events. trigger_type comes from the incident the observer opened (supervisor.py:317-327 ← observer.py:868):

Drift / incident trigger_type	Action	Where
containers_not_running	container_restart	supervisor.py:52,410,423
healthcheck_failed	container_restart	supervisor.py:52 (added fbf165f, 2026-07-29)
service_unhealthy	redeploy	falls through :447
deployment_failed	redeploy	falls through :447
service absent from world state (missing_service, trigger_type: None)	redeploy	supervisor.py:353-360 → :447
node disk_pressure == high	disk_cleanup	supervisor.py:377-381,479+

Only services listed in hosts/<node>/services.yaml are considered (supervisor.py:350); dormant nodes are skipped (supervisor.py:342,378).

Path 2 — direct event-file routing (_process_ha_events, supervisor.py:395, ~630-660) is unchanged and generates only container_restart (homeassistant, shadow-downgraded to alert_only) and alert_only. Path 2 never generates redeploy.

So: three producers of redeploy — service_unhealthy, deployment_failed, missing_service — all on path 1. Section D9 shows only one of them fires in practice, and it fires at the wrong target.

A2. Fields on a redeploy action

Built at supervisor.py:453-467, verbatim:

action = {
    "action_id": action_id,                    # f"redeploy-{node}-{service}"  (:413)
    "timestamp": time.time(),
    "type": "redeploy",
    "node": node,
    "service": service,
    "risk_level": "guarded",
    "confidence": 0.9,
    "description": f"Redeploy {service} on {node} due to {drift['type']}",
    "status": "pending",
    "payload": {
        "reason": drift["type"],
        "svc_key": drift["svc_key"],
    },
}

Written to actions/pending/<action_id>.json (supervisor.py:469-471).

There is no container_name, no path, no compose reference, and no args field. Contrast container_restart (supervisor.py:428-446), which carries container_name resolved via _get_container_name(service). A redeploy action carries exactly two addressing facts: node and service. Everything else — which compose file, which override, which repo — is resolved on the node at execution time from service alone.

Live example, the one non-chelsty pending redeploy on vps:

{ "action_id": "redeploy-vps-gokapi", "timestamp": 1783615889.1762547,
  "type": "redeploy", "node": "vps", "service": "gokapi",
  "risk_level": "guarded", "confidence": 0.9,
  "description": "Redeploy gokapi on vps due to missing_service",
  "status": "pending",
  "payload": { "reason": "missing_service", "svc_key": "vps/gokapi" } }

---

B. EXECUTION — why it's dead

B3. The redeploy handler

Two answers, because repo and runtime disagree.

(a) Master @ 4ecbdbb — executor.py:125-144. No command is built at all. The executor writes a dispatch file and returns:

if action_type == "redeploy":
    if not node or not service:
        success = False
        error_msg = (f"redeploy requires both node and service "
                     f"(node={node!r}, service={service!r})")
    else:
        self._dispatch_redeploy(action_id, node, service)
        return                     # stays in running/ until action_result

_dispatch_redeploy (executor.py:238-267) writes actions/deploy/<node>/<action_id>.json with {action_id, type, node, service, dispatched_at}. Deliberately a different inbox from container_restart's actions/dispatch/<node>/, because node-agent deletes and failure-reports anything in its own inbox that is not container_restart (executor.py:24-29 ← node_agent.py:108,1011-1018).

(b) The executor actually running on vps — image created 2026-07-22T16:13:29Z, i.e. 12 days before the fix. Its /app/src/executor.py lines 106-119, read out of the live container:

if action_type == "redeploy":
    # Full service redeploy via the repo deploy script
    cmd = [
        str(REPO_ROOT / "scripts" / "deploy" / "deploy-node.sh"),
        node,
        service
    ]
    logger.info(f"Running command: {' '.join(cmd)}")
    result = subprocess.run(cmd, capture_output=True, text=True, cwd=str(REPO_ROOT))

REPO_ROOT = /repo (os.getenv("REPO_ROOT", "/repo"), running line 24; master executor.py:33). Synchronous: returncode == 0 → completed, else failed with stderr or stdout.

B4. Working directory / repo path, and does it exist in the container

The repo is reachable inside the executor container — the claim "a path that doesn't exist inside the container" needs one level of precision. Two different paths are involved and only the second is missing.

docker inspect control-plane-executor (vps):

Image=sha256:7603f09d… Created=2026-07-22T16:13:29.905995261Z
/opt/homelab                    -> /opt/homelab              (rw)
/home/oskar/homelab-codex-ws    -> /repo                     (rw)
/var/run/docker.sock            -> /var/run/docker.sock      (rw)
REPO_ROOT=/repo   RUNTIME_PATH=/opt/homelab

Inside the container:

HOME=/home/homelab
/repo/scripts/deploy/deploy-node.sh                      ← EXISTS
ls: cannot access '/home/homelab/homelab-codex-ws': No such file or directory
git: MISSING   docker: MISSING   rsync: MISSING   ssh: MISSING
bash: /usr/bin/bash   python3: /usr/local/bin/python3   flock: /usr/bin/flock

So cwd=/repo is valid and the script is found and executed. It then fails on its own internal path: deploy-node.sh:8 hardcodes REPO_PATH="${HOME}/homelab-codex-ws" = /home/homelab/homelab-codex-ws, which does not exist, and deploy-node.sh:18-21 exits 1.

Reproduced read-only inside the live container with the exact argv the running executor would use:

rc= 1
stdout= --- Starting Deployment on 1CF38E89F150 ---
Error: Repository not found at /home/homelab/homelab-codex-ws

Note the hostname: 1CF38E89F150, the container ID. Even with the repo present, deploy-node.sh:27-57 resolves the host directory by matching os_hostname against hosts/*/host.yaml and would fail with "No host directory found for 1cf38e89f150" — the executor would have deployed its own container's service set, never the action's node. And behind that, two more hard stops: git pull (deploy-node.sh:25) and the compose invocation both need binaries the image does not have.

Four independent blockers, in the order they would be hit: missing ${HOME}/homelab-codex-ws → hostname resolves to a container ID → no git → no docker. The image mount is not one of them.

Note also ../..:/repo:ro in services/control-plane/docker-compose.yml:36,55,86 — master mounts the repo read-only. The running container has it rw, another marker that the container predates current master.

B5. deploy-node.sh arg parsing, and the call chain

deploy-node.sh has no argument parsing whatsoever. There is no getopts, no case "$1", no $1/$2 reference anywhere in its 115 lines. It is a zero-argument script: it derives everything from ${HOME} and hostname (:8,12,13), and deploys the whole service set of whatever host it runs on (:60-78 reads services.txt / services.yaml, :95-113 loops over all of them). The node and service the old executor passed were silently discarded by the shell.

Per-service deploy exists, but in a different script: scripts/deploy/deploy-service.sh, extracted from deploy-node.sh by the 2026-08-03 fix. It takes --repo, --host-dir, --service plus optional --build-if-needed / --force-recreate / --remove-orphans (deploy-service.sh:51-62), validates the service name against ^[a-z0-9][a-z0-9._-]{0,63}$ (:72), and is the single shared compose invocation for both callers — deliberately, because a project-name mismatch plus --remove-orphans is what wiped the control-plane on vps on 2026-06-25 (deploy-service.sh:6-13, deploy-node.sh:86-92).

Call chains, with the host and container each link runs on:

Human deploy (SATURN-side dispatcher — confirms the solaria-gid-fix session's model):

operator @ SATURN:  scripts/deploy/deploy.sh <target>            [host shell, saturn]
  preflight: must be on master, clean tree (deploy.sh:69-82)
  → ssh oskar@<target> 'cd ~/homelab-codex-ws && git pull
                        && ./scripts/deploy/deploy-node.sh'      (deploy.sh:199-202)
      → deploy-node.sh                                           [host shell, target node]
          → deploy-service.sh --build-if-needed --remove-orphans  × every service
                                                                 (deploy-node.sh:100-105)
  (target == control-plane takes a different branch:
   deploy-control-plane.sh --ssh, deploy.sh:194-196)

Agent redeploy, master @ 4ecbdbb:

supervisor  [container control-plane-supervisor, vps]
  → /opt/homelab/actions/pending/redeploy-<node>-<svc>.json
operator (Telegram / operator-ui) → approved/
executor    [container control-plane-executor, vps]
  → /opt/homelab/actions/deploy/<node>/<action_id>.json          (executor.py:238-267)
deploy-runner  [HOST systemd oneshot on <node>, NOT a container]
  → rsync-pull inbox from vps (deploy-runner.sh pull_actions)
  → python3 action.py validate                                   (5 rules, action.py:78-125)
  → deploy-service.sh --repo … --host-dir … --service … --force-recreate
  → action.py emit-result → evt-<node>-<ts>-action_result-<svc>.json
  → rsync-push to vps events/
executor._reconcile_running_actions()                            (executor.py:269-318)
  → completed/ or failed/    (timeout REDEPLOY_TIMEOUT_SECS=900, executor.py:55)

The runner runs on the host, not in a container, on purpose: compose then resolves relative bind mounts and the project name exactly as a human deploy does (deploy-runner.sh:26-31). It also never runs git pull — a redeploy reconciles the node to the checkout it already has; shipping code stays a human deploy.sh (deploy-runner.sh:32-33).

B6. Concrete failure mode today

Nothing fires, and nothing ever has. No redeploy has ever been attempted, approved, or executed — fleet-wide, ever.

Action pool on vps, 2026-08-05:

pending 18   approved 0   running 0   completed 0   failed 0   rejected 0   cancelled 18
dispatch 1   deploy: ls: cannot access '/opt/homelab/actions/deploy': No such file or directory

completed 0 / failed 0 is the whole story: not one action has ever reached a terminal state. 18 redeploy actions exist on disk (16 cancelled, 2 pending); none was ever approved, so the broken handler was never entered.

The two live ones:

/opt/homelab/actions/pending/redeploy-vps-gokapi.json               reason: missing_service
/opt/homelab/actions/pending/redeploy-chelsty-infra-ha-diag-agent.json  reason: missing_service

The chelsty one targets a node dead since 2026-06-01 and is already flagged for manual removal (kb/subsystems/recon-multiagent.md, "Runbook — stale chelsty action").

The executor's entire log since 2026-07-22 is 144 lines and contains three action executions, all hand-injected tests, zero supervisor-generated actions, zero redeploys, and nothing at all after 2026-07-23:

2026-07-22 16:13:41 - Starting executor loop
2026-07-22 16:32:41 - Executing action: test-restart-piha-node_exporter-2
2026-07-22 16:32:41 - Dispatched container_restart … to node-agent on piha
2026-07-22 16:37:42 - ERROR - Action … timed out: Timed out after 300s waiting for node-agent on 'piha'
2026-07-23 11:54:05 - Executing action: test-e2e-1784807638
2026-07-23 11:54:05 - ERROR - Failed to move test-e2e-1784807638 to running: Expecting value: line 1 column 38
   …(same pair repeating every 10 s for 11 minutes — malformed JSON never leaves approved/)…
2026-07-23 12:05:25 - Executing action: test-e2e-b
2026-07-23 12:05:25 - Dispatched container_restart test-e2e-b (container=node_exporter) to node-agent on piha
2026-07-23 12:05:56 - Action test-e2e-b completed

Why nothing fires, in the order the funnel closes:

1. redeploy needs service_unhealthy, deployment_failed, or missing_service. Post-fbf165f, the two high-volume signals (containers_not_running 2,490; healthcheck_failed 3,376 on piha) both go to container_restart. Live incident census on vps confirms it — world/incidents.json holds 3 incidents total, trigger_type: {containers_not_running: 2, healthcheck_failed: 1}. Zero redeploy-producing incidents are open.
2. missing_service still fires (it needs no incident at all), which is exactly what the 2 pending redeploys are. But it is dedup-suppressed forever by its own pending file (supervisor.py:418-421) — redeploy-vps-gokapi has held its ID since 2026-07-19.
3. Nothing gets approved. approved 0, and the executor reads only approved/ (executor.py:94-95).
4. Even on approval the running executor would fail as in B4 — but this has never actually happened, so there is no failed/ artefact, no stack trace, and no "Running command: …deploy-node.sh" log line to point at. The evidence for the break is the reproduction in B4, not a production incident.

---

C. THE HEALTHY PATH FOR COMPARISON

C7. container_restart traced end to end

Correction to the premise: the two action_result events of 2026-07-23 are manual e2e tests, not real remediations. Payloads:

{"action_id": "test-restart-piha-node_exporter-2", "success": true, "error": "", "node": "piha"}
{"action_id": "test-e2e-b",                        "success": true, "error": "", "node": "piha"}

Both hand-crafted IDs. A supervisor-generated ID would be container-restart-piha-node_exporter (supervisor.py:411). And only one of the two round-tripped cleanly: test-restart-piha-node_exporter-2 was dispatched 2026-07-22 16:32 UTC, timed out at 300 s, and its action_result only landed 2026-07-23 11:22 UTC — ~19 h later, after the executor had already filed it failed. test-e2e-b completed in 31 s. So the reference path has exactly one clean end-to-end demonstration, and one latency failure, and has never run from a real incident.

Path, verbatim:

1. executor._execute_action moves approved/<id>.json → running/ (executor.py:104-112), then executor.py:146-157: container_name = data.get("container_name") or service → _dispatch_container_restart(...) → return (stays in running/).
2. _dispatch_container_restart (executor.py:209-236) writes actions/dispatch/<node>/<action_id>.json = {action_id, type, node, service, container_name, dispatched_at}.
3. node-agent on the node rsync-pulls its own subdir with --remove-source-files (node_agent.py:919-960); on vps it reads the path directly (:933).
4. _execute_dispatched_action (node_agent.py:981-1051) applies four gates — idempotency (:999), node scoping (:1003), type whitelist ALLOWED_DISPATCH_ACTION_TYPES = {"container_restart"} (:108, :1011), self-restart guard {"node-agent"} (:114, :1027) — then self.docker_client.containers.get(container_name).restart() (:1043-1044).
5. _report_action_result (node_agent.py:1053+) emits an action_result event; the existing rsync ships it to vps.
6. executor._reconcile_running_actions (:269-318) matches it via _find_action_result (:320-356) and moves the action to completed//failed/, or fails it after ACTION_TIMEOUT_SECS=300.

C7b. Structural differences between the two handlers

On master they are now deliberately near-identical — steps 1, 2, 5 and 6 are the same code, DISPATCHED_ACTION_TYPES = {"container_restart", "redeploy"} (executor.py:60), and _reconcile_running_actions handles both with only the timeout differing (executor.py:299-302). What remains different:

	container_restart	redeploy
Inbox	actions/dispatch/<node>/	actions/deploy/<node>/ (must not share — executor.py:24-29)
Executor on the node	node-agent, a container, own docker socket	deploy-runner, host systemd oneshot + timer, no container
Deployed how	deploy.sh / compose, already everywhere	manual per-node systemd install, nowhere yet
Poll cadence	CHECK_INTERVAL=60	OnUnitActiveSec=60s (matched on purpose)
Timeout	ACTION_TIMEOUT_SECS=300	REDEPLOY_TIMEOUT_SECS=900
Payload	carries container_name	carries only service
Validation	4 gates in node_agent.py	5 gates in action.py:78-125, incl. service must be in hosts/<node>/services.yaml and compose file must exist
Idempotency	_already_processed marker	state/processed-deploy-actions/<id>.done (deploy-runner.sh:57,151-155)
Concurrency	none needed	flock on state/deploy-runner.lock
Can act on node-agent itself	no (self-guard)	yes — deliberate; solaria's node-agent has been docker-blind for weeks (deploy-runner.sh:26-31)
Can act on control-plane / stability-agent	yes	no — deploy-service.sh:93-96 exits 3, runner reports failure with an explanation

The last row is the important asymmetry: see D9.

C8. How the executor reaches nodes

It doesn't, and it cannot. Verified inside the live container: ssh: MISSING, rsync: MISSING, git: MISSING, docker: MISSING. There is no ssh client, no key, no known_hosts. This is by design — executor.py:200-207 and kb/phases/backlog.md "Remediacja floty bez SSH": the VPS never initiates a connection to a node. Every remote action is a file the node comes and collects, over the ssh/rsync channel node-agent (and now deploy-runner) already owns in the other direction. The executor's only outputs are writes into the shared /opt/homelab mount.

Two consequences:

• disk_cleanup is dead by the same defect. _execute_disk_cleanup (executor.py:358-407) builds ["ssh", *SSH_OPTIONS, f"{SSH_USER}@{node}", …] and calls subprocess.run. With no ssh binary this raises FileNotFoundError, caught by executor.py:175-177, and the action fails with [Errno 2] No such file or directory: 'ssh'. Never observed because failed/ is empty — no disk_cleanup was ever approved either. Out of scope here; flagged as a follow-up.
• Solaria's self-ssh problem is irrelevant to the executor. deploy.sh runs on SATURN and ssh's to the target, which is why it cannot deploy solaria from solaria. The executor never ssh's anywhere; the connection is always node → vps, initiated by the node. Solaria's node-agent already ships events to vps over that channel daily, so the transport is proven. Solaria's real blocker is different and unrelated: its node-agent has no docker socket access (group_add: "996" fixed in repo @ ddae57c, still not deployed — running container shows GroupAdd=[999]), which breaks container_restart there. redeploy is unaffected, because deploy-runner runs on the host and never touches node-agent.

---

D. WHAT REDEPLOY IS FOR

D9. When redeploy should fire vs container_restart

The semantic split the brief proposes is the one the code already encodes (supervisor.py:36-52, 423-426, 447-452): container_restart = the container exists and a restart plausibly heals it; redeploy = the container is absent, or its image/compose/env no longer matches the repo, so it must be re-created from the manifest. deploy-service.sh --force-recreate is exactly that and nothing more (deploy-service.sh:111-130: docker compose -f … [-f override] [--env-file] up -d --force-recreate, no --build, deploy-service.sh:113-118 — a redeploy is a reconcile, not a code ship, and building on the 4 GiB vps mid-incident is an OOM risk).

Candidate event types for the redeploy side, and whether each actually reaches an incident today:

Candidate	Opens an incident?	Live volume	Verdict
missing_service (drift, not an event)	n/a — needs no incident; supervisor.py:353-360 raises it purely from desired-vs-actual	2 pending actions right now	The only redeploy producer that actually fires. Semantically the perfect fit: the container is gone, a restart is impossible.
service_unhealthy	yes — observer.py:771,792	34 events, vps only	Fires, but is a trap — see below.
deployment_failed	yes — observer.py:838-842 _handle_deployment_failure	0 events fleet-wide	Only emitter is scripts/lib/log.sh:35,43 (deploy-time shell logging). Never reaches the vps event store. Dead in practice.
containers_not_running	yes, observer.py:771	2,490 piha / 3 solaria / 1 vps	Correctly on container_restart — container exists.
healthcheck_failed	yes, observer.py:771	3,376 piha	Correctly on container_restart since fbf165f. Escalating repeat failures to redeploy is a candidate, but needs a retry policy that does not exist.
container_restarting / container_state_unexpected	no — observational only, observer.py:793-814	2 solaria	Deliberately not remediated.
stability-agent's containers_not_running	no — service=None fails the if service and service != "all" guard at observer.py:747, so the incident branch is never reached	—	Known-broken producer (recon D15, kb/audits/czujniki-2026-07-30.md).

The trap. service_unhealthy has exactly one live emitter in the whole fleet: node_agent.py:1104-1115, which probes http://localhost:18180/summary and hardcodes service="control-plane". So every service_unhealthy incident that could ever produce a redeploy targets vps/control-plane — and control-plane is one of only two services with its own deploy-local.sh (services/control-plane/, services/stability-agent/), which deploy-service.sh:93-96 refuses with exit 3, which the runner turns into a failed action: "control-plane owns its deploy path (services/control-plane/deploy-local.sh) — needs an operator deploy, not an automated redeploy" (deploy-runner.sh:179-181). That refusal is correct — a containerised control plane redeploying itself mid-incident is not something to automate — but it means the fixed redeploy path, once deployed, will still produce zero successful incident-driven redeploys. Only missing_service will ever succeed.

Also worth stating plainly: redeploy cannot repair a service_unhealthy caused by bad code, because the path deliberately does not git pull (deploy-runner.sh:32-33) and deliberately does not --build. It repairs drift — a container removed, a stale env file, a config change already in the node's checkout — not regression.

D10. Preconditions for a working redeploy

Per link in the chain, on master's architecture. Current status verified 2026-08-05.

On the vps (control plane):

#	Precondition	Where	Status
1	executor running post-da151fc code	executor.py:125-144	❌ container built 2026-07-22, runs deploy-node.sh code
2	/opt/homelab/actions/deploy/<node>/ writable	executor.py:79 (_ensure_dirs)	❌ dir absent — created automatically by the new executor
3	repo checkout current	/home/oskar/homelab-codex-ws	⚠️ at efd8d2f, one commit behind master 4ecbdbb; has the fix (jobs/deploy-runner/ dated Aug 3 17:28)
4	not needed: ssh, git, docker CLI in the executor image	—	✅ by design

On each target node:

#	Precondition	Where	vps	piha	solaria
5	repo checkout at REPO_PATH	deploy-runner.sh:73	✅ efd8d2f	✅ 4ecbdbb	(main checkout, not inspected)
6	hosts/<node>/ exists in that checkout	deploy-runner.sh:74	✅	✅	✅
7	scripts/deploy/deploy-service.sh present + executable	deploy-runner.sh:75-76	✅ Aug 3	✅ Aug 4	—
8	jobs/deploy-runner/ systemd unit + timer installed & enabled	kb/runbooks/deploy-runner-install.md	❌	❌	❌
9	/opt/homelab/config/deploy-runner/env with NODE_NAME (empty VPS_EVENTS_HOST on vps)	env.example	❌	❌	❌
10	service user in docker group	runbook	—	—	✅ (oskar ∈ docker gid 996)
11	python3 + PyYAML, rsync, flock	action.py:60-64, deploy-runner.sh:70-76	—	—	—
12	ssh key reaching vps (remote nodes only)	deploy-runner.sh:62-66	n/a	✅ (node-agent already ships events)	✅

Per-action, at execution time (action.py:78-125 — all five must hold or the runner reports failed without deploying):

#	Rule	Note
13	type == "redeploy"
14	action["node"] == NODE_NAME	defense in depth; inbox is already node-scoped
15	action_id / service match strict name patterns	no path traversal
16	service listed in hosts/<node>/services.yaml	repo desired state is the authority; a stale dispatch cannot pull an arbitrary stack
17	services/<service>/docker-compose.yml exists in the repo
18	(not a rule, an outcome) service must not have services/<svc>/deploy-local.sh	else exit 3 → failed-with-explanation

Checked against the one actionable pending action, redeploy-vps-gokapi: rule 16 ✅ (hosts/vps/services.yaml:63), rule 17 ✅ (services/gokapi/docker-compose.yml), rule 18 ✅ (no deploy-local.sh). gokapi is a valid end-to-end target the moment the runner is installed on vps.

Also required for the loop to close: the result event must land in /opt/homelab/events/<node>/ on the vps in node_agent.emit_event()'s exact format, because executor._find_action_result (:320-356) globs evt-*-action_result-*.json and parses the unix timestamp out of the filename. action.py:emit_result reproduces that byte-for-byte and adds payload.source = "deploy-runner".

---

FIX SHAPE — DECISIONS NEEDED

The transport and script layers are decided and written. What is left is one sequencing decision and four genuine design choices the 2026-08-03 work left open.

1. Deploy vs re-derive. The repair is 79bfe8c/da151fc on master, 248 tests green, unexercised at runtime.

• Deploy as written — three runtime steps (deploy.sh control-plane; install jobs/deploy-runner/ on vps + piha + solaria per kb/runbooks/deploy-runner-install.md; e2e on a benign piha service, then redeploy-vps-gokapi). Fastest to a working loop; accepts a design nobody has yet run.
• Re-derive first — treat the shipped design as a proposal, resolve 2–5 below, then deploy once. Slower; avoids installing systemd units on three hosts twice.

Note that 2–5 are answerable either way, but their answers are cheaper to apply before the units are on three nodes.

2. Where the deploy actually runs: node-side runner vs saturn-side dispatcher. The shipped design is node-side (deploy-runner on each node, executor never connects out).

• Node-side (as shipped): preserves "remediacja floty bez SSH"; works on solaria despite its broken node-agent socket and despite the self-ssh problem; every node needs a host-level systemd install and its own repo checkout, and the install is not covered by deploy.sh — it drifts silently.
• Executor → ssh SATURN → deploy.sh <node>: reuses the human path exactly, one install point instead of N, and inherits deploy.sh's preflight gates. But it inverts the no-SSH decision, puts a key to SATURN inside a vps container, makes SATURN a runtime dependency of remediation (it is a workstation, not a server), and deploy.sh is whole-node only — it cannot express "redeploy just gokapi".
• Executor → ssh direct to node → deploy-service.sh: fewest moving parts, no runner, no timer, no rsync. Requires an ssh client + fleet key in the executor container — the exact thing the architecture forbids — and reintroduces the self-ssh and reachability problems the pull model sidesteps.

3. Per-service vs whole-node redeploy. Shipped: per-service, and the action payload carries no node-wide option.

• Per-service: minimal blast radius; matches the action's semantics (missing_service names one service); a genuinely node-wide failure needs N actions and N approvals.
• Whole-node (i.e. deploy-node.sh, what the old executor accidentally attempted): one action heals a node; but it touches every service including healthy ones, runs --build-if-needed, and --remove-orphans is in play — the 2026-06-25 control-plane wipe is precedent for how that ends.

4. How args flow — how much the action carries vs how much the node resolves. Shipped: the action carries {node, service} only; everything else (compose path, override, env-file, force-recreate) is resolved on the node by deploy-service.sh from repo state.

• Thin action (as shipped): the repo is the single authority; an old or forged dispatch cannot smuggle a path or a flag; but the operator approving the action cannot see or influence what will actually run, and --build/--pull are not expressible even when they are what is needed.
• Fat action: supervisor stamps compose path, override, flags into the payload — visible at approval time, tunable per incident; every field becomes attack surface the runner must re-validate, and payloads can go stale against the node's checkout.

5. What happens to service_unhealthy → control-plane (D9). Once the runner is installed, the only incident-driven redeploy producer will target the one service the runner refuses.

• Leave it: the failed action with its explanatory message is the alert; costs one failed action per control-plane outage and burns the dedup ID.
• Downgrade to alert_only at the supervisor: honest — nothing automated can fix it — but drops the drift out of the remediation view.
• Give control-plane a self-redeploy path (analogous to deploy-control-plane.sh): closes the loop, and lets a broken control plane attempt to repair itself with its own hands. Highest risk on the list.
• Widen service_unhealthy beyond the hardcoded control-plane probe (node_agent.py:1104-1115): would give redeploy a real, non-degenerate input set for the first time — but that is a monitoring change, not a redeploy change, and it interacts with the healthcheck_failed routing decided in fbf165f.

Out of scope here, found en route: the executor's disk_cleanup handler shells out to ssh, which does not exist in the image (C8) — same class of defect as the original redeploy break, never observed because no disk_cleanup was ever approved. Worth its own task.