diff --git a/CLAUDE.md b/CLAUDE.md index 0e2932d..1029ec9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,9 +90,21 @@ The platform uses a multi-agent model with **human-in-the-loop** for destructive Agent → /opt/homelab/actions/pending/.json → Telegram notification → Operator approves → /opt/homelab/actions/approved/.json - → Executor runs → completed / failed + → Executor dispatches to the target node → completed / failed ``` +The executor never connects to a node (deliberate — see docs/backlog.md +"Remediacja floty bez SSH"). It writes a dispatch file that the node collects: + +| Action type | Inbox | Executed on the node by | +|---|---|---| +| `container_restart` | `actions/dispatch//` | node-agent (own docker socket) | +| `redeploy` | `actions/deploy//` | deploy-runner, host-level systemd (`jobs/deploy-runner/`) | + +Both report back with an `action_result` event, which the executor turns into +`completed` / `failed`. `disk_cleanup` and `alert_only` still resolve inside the +executor. + Agents must never execute destructive actions (restarts, deploys, config changes) without a corresponding approved action file. ## Event System @@ -109,7 +121,7 @@ Normalized event types: `deployment_started/completed/failed`, `service_unhealth |---|---|---|---| | `containers_not_running` | stability-agent | `container_restart` | dedup via stable ID | | `healthcheck_failed` | node-agent | `container_restart` | dedup via stable ID | -| `service_unhealthy` / other | stability-agent | `redeploy` (broken until etap 2 executor fix) | dedup via stable ID | +| `service_unhealthy` / other | stability-agent | `redeploy` → dispatched to the node's deploy-runner (`jobs/deploy-runner/`) | dedup via stable ID | | `disk_pressure` (high) | stability-agent | `disk_cleanup` | dedup via stable ID | | `ha_websocket_dead` | ha-diag-agent | `container_restart` (homeassistant) | 30 min after completion | | `ha_websocket_recovered` | ha-diag-agent | cancels matching restart | — | diff --git a/docs/backlog.md b/docs/backlog.md index 829bcbb..4d5fb79 100644 --- a/docs/backlog.md +++ b/docs/backlog.md @@ -4,6 +4,60 @@ Centralny tracker tech-długu i znanych usterek. Wpisy ze sesji — dodawaj z da --- +## deploy-runner: instalacja na węzłach + E2E redeployu (2026-08-03) + +**Data**: 2026-08-03 +**Źródło**: sesja `task/redeploy-fix` — recon D14/D15 + OPEN QUESTION 4 (nieopróżniona +kolejka akcji). +**Było**: `redeploy` nie mógł się wykonać nigdy. Executor odpalał +`scripts/deploy/deploy-node.sh ` **wewnątrz swojego kontenera** — +skrypt ignoruje oba argumenty i wymaga repo w `${HOME}/homelab-codex-ws` +(w kontenerze `HOME=/home/homelab`, katalog nie istnieje) → `exit 1` w 18. linii. +Za tym stały jeszcze trzy blokady: brak `git`, brak klienta `docker` w obrazie oraz +— gdyby przeszedł — deploy **całego zestawu usług hosta executora**, nie węzła +z akcji. Stąd `healthcheck_failed` przekierowany 2026-07-29 na `container_restart` +i 18 pending / 0 completed. +**Zrobione w repo (ta sesja)**: `scripts/deploy/deploy-service.sh` (deploy jednej +usługi, wspólny z `deploy-node.sh` — ta sama inwokacja compose, więc ta sama nazwa +projektu), `jobs/deploy-runner/` (systemd na hoście: rsync-pull akcji, walidacja, +deploy, `action_result` z powrotem), executor dispatchuje `redeploy` do +`actions/deploy//` i rozlicza je jak `container_restart` +(`REDEPLOY_TIMEOUT_SECS=900`). 248 testów zielonych. +**Do zrobienia (runtime, wymaga operatora)**: +1. Deploy control-plane na VPS (nowy executor + `../..:/repo:ro`). +2. Instalacja `jobs/deploy-runner/` na vps, piha, solaria — patrz README + („Install (per node)"). Na VPS `VPS_EVENTS_HOST` musi zostać puste. +3. E2E na benignej usłudze na PIHA (wzorzec `test-e2e-b` z 2026-07-23), potem + opróżnienie kolejki 18 pending — w tym `redeploy-vps-gokapi`. +4. SOLARIA: `group_add: "996"` dla node-agenta wciąż niewdrożony (recon 641-649) — + dopóki nie wejdzie, `container_restart` tam nie działa (redeploy działa, bo nie + idzie przez node-agenta). + +--- + +## disk_cleanup w executorze jest zepsuty tak samo jak był redeploy (2026-08-03) + +**Data**: 2026-08-03 +**Źródło**: sesja `task/redeploy-fix` (znalezione przy okazji, poza zakresem zadania). +**Problem**: `executor._execute_disk_cleanup()` odpala `ssh oskar@ …`, a obraz +control-plane (`python:3.11-slim` + `pip install pyyaml`) **nie ma klienta ssh** — +`subprocess.run(["ssh", …])` leci `FileNotFoundError`, akcja ląduje w `failed/`. +To ta sama klasa błędu co redeploy i sprzeczne z decyzją „bez SSH w executorze". +**Fix**: przenieść `disk_cleanup` na model dispatch (node-agent albo deploy-runner — +runner ma już hosta i uprawnienia) albo usunąć typ akcji. Do decyzji przy okazji +opróżniania kolejki. + +--- + +## `scripts/deploy/deploy-host.sh` to pusty plik (2026-08-03) + +**Data**: 2026-08-03 +**Źródło**: sesja `task/redeploy-fix`. +**Problem**: 0-bajtowy, wykonywalny stub w `scripts/deploy/` — wygląda jak +entrypoint, nie robi nic. Kandydat do skasowania (jak `deploy-role.sh` w etapie 0). + +--- + ## Cutover HA "ken": kontener piha to legacy, prawdziwy dom to RPi4/HAOS (2026-07-22) **Data**: 2026-07-22 diff --git a/jobs/deploy-runner/README.md b/jobs/deploy-runner/README.md new file mode 100644 index 0000000..4ec0738 --- /dev/null +++ b/jobs/deploy-runner/README.md @@ -0,0 +1,111 @@ +# deploy-runner — host-side executor for `redeploy` actions + +Runs on each node that should be able to execute an operator-approved +`redeploy`. It is the missing half of the self-healing loop: the control-plane +executor can generate and dispatch redeploys, but nothing on the node could ever +perform one. + +## What was broken + +The executor ran `scripts/deploy/deploy-node.sh ` **inside its own +container**. That script ignores both arguments, and expects a repo at +`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it +exited at line 18 with `Error: Repository not found`. Behind that failure sat +three more: no `git`, no `docker` CLI in the image, and — had it ever got that +far — it would have deployed the **executor host's** entire service set, not the +action's target node/service. Every `healthcheck_failed → redeploy` dead-ended +(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15). + +## How it works now + +``` +supervisor → actions/pending/.json (unchanged) +operator → actions/approved/.json (unchanged, HITL) +executor (vps) → actions/deploy//.json ← dispatch only, no execution +deploy-runner ← rsync-pull (node initiates; VPS never connects to a node) + → scripts/deploy/deploy-service.sh --force-recreate + → events//evt-…-action_result-….json → rsync-push +executor → completed / failed (via _reconcile_running_actions) +``` + +Design points, and why: + +- **Host-level, not a container.** Compose resolves relative bind mounts and the + project name against the filesystem of whatever runs it. In a container those + resolve to container paths that the daemon then interprets as host paths — a + silent way to produce broken mounts, or to land in a different Compose project + where `--remove-orphans` deletes the running stack. On the host it behaves + exactly like a human `deploy.sh`. +- **Independent of node-agent** (own rsync pull, own result push) so it can + redeploy `node-agent` itself — the solaria case, where node-agent has been + docker-blind for weeks. +- **Separate inbox** from `actions/dispatch//`: node-agent deletes and + failure-reports every file in its own inbox that is not `container_restart`. +- **No `git pull`.** A redeploy reconciles the node to the checkout it already + has. Shipping new code stays a human `scripts/deploy/deploy.sh` action. +- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain + `up -d` is a no-op when config is unchanged — exactly the unhealthy-container + case; building on the 4 GiB VPS mid-incident is an OOM risk. + +Every action is validated before anything runs (`action.py`): type must be +`redeploy`, `node` must match this node, the service name must be a plain +kebab-case name, the service must be listed in `hosts//services.yaml`, and +its compose file must exist. Nothing from the action payload is ever executed — +only a validated service *name* reaches the deploy script. Actions are +idempotent (marker in `state/processed-deploy-actions/`), single-instance +(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the +executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the +control plane gives up). + +## Install (per node) + +Not deployed by `deploy.sh` — it is a host-level systemd unit, like +`jobs/documents-ingest/`. On the target node: + +```bash +# 1. config +sudo mkdir -p /opt/homelab/config/deploy-runner +sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \ + /opt/homelab/config/deploy-runner/env +sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env +sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS + +# 2. units +sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \ + /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now homelab-deploy-runner.timer + +# 3. verify (no action queued yet → one clean, empty run) +sudo systemctl start homelab-deploy-runner.service +journalctl -u homelab-deploy-runner.service -n 30 --no-pager +``` + +Requirements on the node: repo checkout at `REPO_PATH`, the service user in the +`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key +that reaches the VPS — the same one node-agent already uses for event shipping. + +On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same +`/opt/homelab` mount, so there is nothing to pull and nothing to push. + +## Operating it + +- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy//` on the VPS. +- Per-action deploy output: `/opt/homelab/logs/deploy-runner/-.log` on the node. +- Runner activity: `journalctl -u homelab-deploy-runner.service`. +- A redeploy for a service with its own `deploy-local.sh` (control-plane) is + reported as **failed** with an explanatory message — those need an operator + deploy, by design. +- To let an already-processed action run again, remove its marker: + `rm /opt/homelab/state/processed-deploy-actions/.done`. + +## Tests + +`tests/` — validation and rejection cases, the event format contract against the +executor's real parser, the `docker compose` argv contract of +`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself +with a stubbed `docker`. + +```bash +python3 -m pytest jobs/deploy-runner/tests -q +``` diff --git a/jobs/deploy-runner/action.py b/jobs/deploy-runner/action.py new file mode 100755 index 0000000..c0d932e --- /dev/null +++ b/jobs/deploy-runner/action.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +"""Validation and result-reporting helper for the host-side deploy runner. + +The runner (deploy-runner.sh) is bash — it does flock, rsync and timeout, which +bash does well. Everything that needs to parse untrusted JSON, read +hosts//services.yaml or write an event in the exact node-agent format +lives here, where it is testable (jobs/deploy-runner/tests/). + +Two subcommands, both designed to be consumed from bash: + + validate --file --node --repo + Prints shell assignments for `eval`: + OK='true'|'false' + ACTION_ID='...' (best effort — set even when OK=false, so the + runner can still report a result for it) + SERVICE='...' + ERROR='...' + Always exits 0 unless the arguments themselves are unusable; the verdict + is in OK. Nothing from the action file is ever executed or interpolated + into a command — only a validated service NAME is handed back. + + emit-result --events-dir --node --action-id --service + --success true|false [--error ] + Writes evt---action_result-.json into the node's event + directory, byte-compatible with node_agent.emit_event() so the control + plane executor's _find_action_result() picks it up unchanged. Prints the + path it wrote. + +Validation rules (all must hold): + 1. the file is JSON with type == "redeploy" + 2. action["node"] == this node — defense in depth; the inbox is already + node-scoped, but a mis-delivered file must never deploy here + 3. action_id and service match strict name patterns (no path traversal) + 4. the service is listed in hosts//services.yaml — the repo's desired + state is the authority on what this node may run, so a rogue or stale + dispatch cannot pull an arbitrary stack onto the node + 5. services//docker-compose.yml exists in the repo +""" +from __future__ import annotations + +import argparse +import json +import re +import shlex +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +ACTION_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") +SERVICE_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$") +ACTION_TYPE = "redeploy" + + +class Invalid(Exception): + """Validation failure carrying a human-readable reason.""" + + +def _load_desired_services(repo: Path, node: str) -> set[str]: + """Service names listed in hosts//services.yaml.""" + try: + import yaml + except ImportError as exc: # pragma: no cover - environment problem + raise Invalid(f"PyYAML not available on this node: {exc}") from exc + + path = repo / "hosts" / node / "services.yaml" + if not path.is_file(): + raise Invalid(f"no desired-state manifest for node '{node}' ({path})") + try: + data = yaml.safe_load(path.read_text()) or {} + except Exception as exc: + raise Invalid(f"unreadable {path}: {exc}") from exc + + services = data.get("services") or {} + if isinstance(services, dict): + return {str(k) for k in services.keys()} + if isinstance(services, list): + return {str(s) for s in services} + raise Invalid(f"unexpected 'services' shape in {path}: {type(services).__name__}") + + +def validate(action_file: Path, node: str, repo: Path) -> tuple[str, str]: + """Return (action_id, service) for a valid redeploy action, else raise Invalid. + + action_id is also returned inside the Invalid path via the caller's + best-effort parse, so a rejected action can still be reported back. + """ + try: + action = json.loads(action_file.read_text()) + except Exception as exc: + raise Invalid(f"unparseable action file: {exc}") from exc + if not isinstance(action, dict): + raise Invalid("action file is not a JSON object") + + action_id = str(action.get("action_id") or action_file.stem) + if not ACTION_ID_RE.match(action_id): + raise Invalid(f"invalid action_id: {action_id!r}") + + action_type = action.get("type") + if action_type != ACTION_TYPE: + raise Invalid( + f"action type {action_type!r} is not executable by the deploy runner " + f"(only {ACTION_TYPE!r})" + ) + + target_node = action.get("node") + if target_node != node: + raise Invalid(f"action addressed to node {target_node!r}, not {node!r} — refused") + + service = action.get("service") + if not isinstance(service, str) or not SERVICE_RE.match(service): + raise Invalid(f"invalid service name: {service!r}") + + desired = _load_desired_services(repo, node) + if service not in desired: + raise Invalid( + f"service '{service}' is not in hosts/{node}/services.yaml — " + f"refusing to deploy a service this node is not declared to run" + ) + + compose = repo / "services" / service / "docker-compose.yml" + if not compose.is_file(): + raise Invalid(f"no compose file in repo for '{service}' ({compose})") + + return action_id, service + + +def _best_effort_action_id(action_file: Path) -> str: + try: + data = json.loads(action_file.read_text()) + candidate = str(data.get("action_id") or action_file.stem) + except Exception: + candidate = action_file.stem + return candidate if ACTION_ID_RE.match(candidate) else "" + + +def _emit_assignments(ok: bool, action_id: str, service: str, error: str) -> str: + return "\n".join( + f"{key}={shlex.quote(value)}" + for key, value in ( + ("OK", "true" if ok else "false"), + ("ACTION_ID", action_id), + ("SERVICE", service), + ("ERROR", error), + ) + ) + + +def emit_result(events_dir: Path, node: str, action_id: str, service: str, + success: bool, error: str) -> Path: + """Write an action_result event in node_agent.emit_event()'s exact format. + + The control-plane executor globs evt-*-action_result-*.json in the node's + event directory and matches payload.action_id, comparing the unix timestamp + embedded in the filename against the action's started_at — so both the name + and the payload shape matter here (see executor._find_action_result). + """ + ts = int(time.time()) + slug = re.sub(r"[^a-z0-9]", "-", (service or "node").lower())[:32].strip("-") + event_id = f"evt-{node}-{ts}-action_result-{slug}" + event = { + "id": event_id, + "timestamp": ts, + "date": datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z"), + "type": "action_result", + "severity": "info" if success else "high", + "node": node, + "service": service or "", + "message": f"Action {action_id} {'succeeded' if success else 'failed'}", + "payload": { + "action_id": action_id, + "success": success, + "error": error, + "node": node, + "source": "deploy-runner", + }, + } + events_dir.mkdir(parents=True, exist_ok=True) + path = events_dir / f"{event_id}.json" + path.write_text(json.dumps(event, indent=2)) + return path + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + + p_val = sub.add_parser("validate") + p_val.add_argument("--file", required=True, type=Path) + p_val.add_argument("--node", required=True) + p_val.add_argument("--repo", required=True, type=Path) + + p_emit = sub.add_parser("emit-result") + p_emit.add_argument("--events-dir", required=True, type=Path) + p_emit.add_argument("--node", required=True) + p_emit.add_argument("--action-id", required=True) + p_emit.add_argument("--service", default="") + p_emit.add_argument("--success", required=True, choices=["true", "false"]) + p_emit.add_argument("--error", default="") + + args = parser.parse_args(argv) + + if args.command == "validate": + try: + action_id, service = validate(args.file, args.node, args.repo) + except Invalid as exc: + print(_emit_assignments(False, _best_effort_action_id(args.file), "", str(exc))) + return 0 + print(_emit_assignments(True, action_id, service, "")) + return 0 + + path = emit_result( + args.events_dir, args.node, args.action_id, args.service, + args.success == "true", args.error, + ) + print(path) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/jobs/deploy-runner/deploy-runner.sh b/jobs/deploy-runner/deploy-runner.sh new file mode 100755 index 0000000..de7e0aa --- /dev/null +++ b/jobs/deploy-runner/deploy-runner.sh @@ -0,0 +1,200 @@ +#!/usr/bin/env bash +# jobs/deploy-runner/deploy-runner.sh — host-side executor for `redeploy` actions. +# +# WHY THIS EXISTS (docs/architecture/RECON-multiagent-2026-07-27.md D14/D15): +# the control-plane executor could never carry out a redeploy — it ran +# deploy-node.sh inside its own container, a script that ignores its arguments +# and expects a repo at ${HOME}/homelab-codex-ws that does not exist there (nor +# does git or the docker CLI). Every healthcheck_failed → redeploy action +# dead-ended, which is the single biggest reason the self-healing loop had +# 18 pending / 0 completed actions. +# +# The fix keeps the architecture decision from docs/backlog.md ("Remediacja +# floty bez SSH"): the VPS never initiates a connection to a node. The executor +# only WRITES an action file; this runner, on the target node, pulls it, runs +# the deploy locally, and reports the outcome back through the existing event +# pipeline. +# +# executor (vps) → /opt/homelab/actions/deploy//.json +# → this runner rsync-pulls it (--remove-source-files: collected once) +# → validates it (jobs/deploy-runner/action.py) +# → scripts/deploy/deploy-service.sh --force-recreate (same compose +# invocation as a human deploy — identical project name) +# → writes an action_result event and rsync-pushes it back to the VPS +# → executor._reconcile_running_actions() → completed / failed +# +# It runs on the HOST, not in a container: compose then resolves relative bind +# mounts and the project name exactly as it does for a human `deploy.sh`, and +# no agent container needs a docker CLI. It is deliberately independent of +# node-agent (own rsync, own result push) so it can redeploy node-agent itself — +# the solaria case, where node-agent has been docker-blind for weeks. +# +# It NEVER runs `git pull`: a redeploy reconciles the node to the checkout it +# already has. Shipping new code stays a human `scripts/deploy/deploy.sh` action. +# +# Config comes from the environment (systemd EnvironmentFile) — see env.example. +# Install: see README.md. + +# NOT set -e: one malformed or failing action must be reported and skipped, not +# abort the whole run. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +HELPER="${SCRIPT_DIR}/action.py" + +NODE_NAME="${NODE_NAME:-}" +REPO_PATH="${REPO_PATH:-${HOME}/homelab-codex-ws}" +RUNTIME_PATH="${RUNTIME_PATH:-/opt/homelab}" +VPS_EVENTS_HOST="${VPS_EVENTS_HOST:-}" +VPS_EVENTS_USER="${VPS_EVENTS_USER:-oskar}" +VPS_EVENTS_PATH="${VPS_EVENTS_PATH:-/opt/homelab/events}" +VPS_DEPLOY_PATH="${VPS_DEPLOY_PATH:-/opt/homelab/actions/deploy}" +DEPLOY_TIMEOUT_SECS="${DEPLOY_TIMEOUT_SECS:-600}" + +INBOX="${RUNTIME_PATH}/actions/deploy/${NODE_NAME}" +EVENTS_DIR="${RUNTIME_PATH}/events/${NODE_NAME}" +STATE_DIR="${RUNTIME_PATH}/state" +MARKER_DIR="${STATE_DIR}/processed-deploy-actions" +LOG_DIR="${RUNTIME_PATH}/logs/deploy-runner" +HOST_DIR="${REPO_PATH}/hosts/${NODE_NAME}" + +# Same ssh settings node-agent uses for its rsync channel: no ~/.ssh/config, no +# host-key prompts, batch mode so a missing key fails fast instead of hanging. +SSH_CMD="ssh -F /dev/null -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o BatchMode=yes" +if [[ -n "${SSH_KEY:-}" ]]; then + SSH_CMD="${SSH_CMD} -i ${SSH_KEY}" +fi + +log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [deploy-runner] $*"; } +die() { log "ERROR: $*"; exit 1; } + +[[ -n "$NODE_NAME" ]] || die "NODE_NAME is not set (see env.example)" +[[ -d "$REPO_PATH" ]] || die "repo not found at ${REPO_PATH} (set REPO_PATH)" +[[ -d "$HOST_DIR" ]] || die "no hosts/${NODE_NAME}/ directory in ${REPO_PATH}" +[[ -x "${REPO_PATH}/scripts/deploy/deploy-service.sh" ]] \ + || die "scripts/deploy/deploy-service.sh missing or not executable in ${REPO_PATH}" +command -v python3 >/dev/null || die "python3 not found" + +mkdir -p "$INBOX" "$EVENTS_DIR" "$MARKER_DIR" "$LOG_DIR" || die "cannot create runtime dirs under ${RUNTIME_PATH}" + +# Single instance: the timer fires every 60s but a deploy may take minutes. +exec 9>"${STATE_DIR}/deploy-runner.lock" || die "cannot open lock file" +if ! flock -n 9; then + log "another deploy-runner run is in progress — exiting" + exit 0 +fi + +# ── transport ──────────────────────────────────────────────────────────────── + +# Remote nodes fetch their inbox from the VPS. On the VPS itself the executor +# writes straight into the shared /opt/homelab mount, so there is nothing to +# pull and nothing to push — the executor reads the result event in place. +is_remote() { [[ -n "$VPS_EVENTS_HOST" ]]; } + +pull_actions() { + is_remote || return 0 + local rc=0 + rsync -az --remove-source-files \ + --omit-dir-times --no-perms --no-owner --no-group \ + -e "$SSH_CMD" \ + "${VPS_EVENTS_USER}@${VPS_EVENTS_HOST}:${VPS_DEPLOY_PATH}/${NODE_NAME}/" \ + "${INBOX}/" || rc=$? + # 23/24 = "partial transfer"/"vanished source files": what an empty or + # concurrently-drained remote inbox looks like. Not worth logging each run. + if [[ $rc -ne 0 && $rc -ne 23 && $rc -ne 24 ]]; then + log "WARN: dispatch pull failed (rsync rc=${rc})" + fi +} + +push_event() { + local event_file="$1" + is_remote || return 0 + local rc=0 + rsync -az --remove-source-files \ + --omit-dir-times --no-perms --no-owner --no-group \ + -e "$SSH_CMD" \ + "$event_file" \ + "${VPS_EVENTS_USER}@${VPS_EVENTS_HOST}:${VPS_EVENTS_PATH}/${NODE_NAME}/" || rc=$? + if [[ $rc -ne 0 && $rc -ne 23 && $rc -ne 24 ]]; then + # Left in place on failure: node-agent's own event shipping will pick it + # up on its next cycle, so a result is delayed, never lost. + log "WARN: result push failed (rsync rc=${rc}) — leaving ${event_file} for node-agent to ship" + fi +} + +report() { + local action_id="$1" service="$2" success="$3" error="$4" + local event_file + event_file=$(python3 "$HELPER" emit-result \ + --events-dir "$EVENTS_DIR" \ + --node "$NODE_NAME" \ + --action-id "$action_id" \ + --service "$service" \ + --success "$success" \ + --error "$error") || { log "ERROR: could not write result event for ${action_id}"; return 1; } + log "reported ${action_id}: success=${success}${error:+ error=${error}}" + push_event "$event_file" +} + +# ── main ───────────────────────────────────────────────────────────────────── + +pull_actions + +shopt -s nullglob +for action_file in "${INBOX}"/*.json; do + OK="false"; ACTION_ID=""; SERVICE=""; ERROR="" + eval "$(python3 "$HELPER" validate --file "$action_file" --node "$NODE_NAME" --repo "$REPO_PATH")" + + if [[ "$OK" != "true" ]]; then + log "rejected $(basename "$action_file"): ${ERROR}" + if [[ -n "$ACTION_ID" ]]; then + report "$ACTION_ID" "$SERVICE" "false" "$ERROR" + fi + rm -f "$action_file" + continue + fi + + marker="${MARKER_DIR}/${ACTION_ID}.done" + if [[ -e "$marker" ]]; then + log "action ${ACTION_ID} already processed — skipping (idempotency)" + rm -f "$action_file" + continue + fi + + log "deploying ${SERVICE} for action ${ACTION_ID}" + action_log="${LOG_DIR}/${ACTION_ID}-$(date -u +%Y%m%dT%H%M%SZ).log" + rc=0 + timeout "$DEPLOY_TIMEOUT_SECS" \ + "${REPO_PATH}/scripts/deploy/deploy-service.sh" \ + --repo "$REPO_PATH" \ + --host-dir "$HOST_DIR" \ + --service "$SERVICE" \ + --force-recreate \ + >"$action_log" 2>&1 || rc=$? + + case "$rc" in + 0) + report "$ACTION_ID" "$SERVICE" "true" "" + ;; + 3) + report "$ACTION_ID" "$SERVICE" "false" \ + "${SERVICE} owns its deploy path (services/${SERVICE}/deploy-local.sh) — needs an operator deploy, not an automated redeploy" + ;; + 124) + report "$ACTION_ID" "$SERVICE" "false" \ + "deploy timed out after ${DEPLOY_TIMEOUT_SECS}s (log: ${action_log})" + ;; + *) + # Last 20 lines are the useful part of a compose failure; the full + # output stays in the per-action log on the node. + err=$(tail -n 20 "$action_log" 2>/dev/null | tr -d '\000' | tail -c 2000) + report "$ACTION_ID" "$SERVICE" "false" \ + "deploy-service.sh exited ${rc} (log: ${action_log}): ${err}" + ;; + esac + + touch "$marker" || log "WARN: could not write idempotency marker ${marker}" + rm -f "$action_file" +done + +exit 0 diff --git a/jobs/deploy-runner/env.example b/jobs/deploy-runner/env.example new file mode 100644 index 0000000..50f1644 --- /dev/null +++ b/jobs/deploy-runner/env.example @@ -0,0 +1,29 @@ +# deploy-runner runtime config — copy to /opt/homelab/config/deploy-runner/env +# on each node and fill in. Read by homelab-deploy-runner.service +# (EnvironmentFile), so: KEY=value, no quotes, no shell expansion. + +# Canonical topology node name — MUST match hosts// and the `node` field +# in dispatched actions. Not the OS hostname. +NODE_NAME=piha + +# Repo checkout on this node (the runner never pulls it; deploys what is there). +REPO_PATH=/home/oskar/homelab-codex-ws + +RUNTIME_PATH=/opt/homelab + +# VPS Tailscale IP — where the control plane writes dispatched actions and +# expects result events. LEAVE EMPTY ON THE VPS ITSELF: there the executor +# writes into the same /opt/homelab mount and no rsync hop is needed. +VPS_EVENTS_HOST=100.95.58.48 +VPS_EVENTS_USER=oskar +VPS_EVENTS_PATH=/opt/homelab/events +VPS_DEPLOY_PATH=/opt/homelab/actions/deploy + +# Per-action wall-clock ceiling. Must stay below the executor's +# REDEPLOY_TIMEOUT_SECS (default 900) so a stuck deploy is reported by the node +# rather than silently timed out by the control plane. +DEPLOY_TIMEOUT_SECS=600 + +# Optional: explicit ssh key for the rsync channel. Omit when the service user's +# default key (~/.ssh/id_*) already reaches the VPS — same key node-agent uses. +# SSH_KEY=/home/oskar/.ssh/id_ed25519 diff --git a/jobs/deploy-runner/systemd/homelab-deploy-runner.service b/jobs/deploy-runner/systemd/homelab-deploy-runner.service new file mode 100644 index 0000000..ce192ae --- /dev/null +++ b/jobs/deploy-runner/systemd/homelab-deploy-runner.service @@ -0,0 +1,25 @@ +# homelab-deploy-runner.service — host-side executor for control-plane `redeploy` +# actions (docs/architecture/RECON-multiagent-2026-07-27.md D14/D15). +# +# Host-level and NOT a container, deliberately: it runs `docker compose` the way +# a human deploy does, so relative bind mounts and the compose project name +# resolve identically. A containerised runner would resolve them against its own +# filesystem and hand the daemon host paths that do not exist. +# +# Install: see ../README.md (copy this + the timer to /etc/systemd/system/, +# create /opt/homelab/config/deploy-runner/env from env.example, daemon-reload, +# enable the TIMER — not this unit). +[Unit] +Description=Homelab deploy runner (executes approved redeploy actions on this node) +After=network-online.target docker.service +Wants=network-online.target + +[Service] +Type=oneshot +User=oskar +# Runtime config: NODE_NAME, REPO_PATH, VPS_EVENTS_HOST, ... (see env.example). +EnvironmentFile=/opt/homelab/config/deploy-runner/env +ExecStart=/home/oskar/homelab-codex-ws/jobs/deploy-runner/deploy-runner.sh +# Slightly above the runner's own per-action DEPLOY_TIMEOUT_SECS (default 600) +# so the script always gets to report a result before systemd intervenes. +TimeoutStartSec=900 diff --git a/jobs/deploy-runner/systemd/homelab-deploy-runner.timer b/jobs/deploy-runner/systemd/homelab-deploy-runner.timer new file mode 100644 index 0000000..2a6f5dd --- /dev/null +++ b/jobs/deploy-runner/systemd/homelab-deploy-runner.timer @@ -0,0 +1,16 @@ +# homelab-deploy-runner.timer — poll for dispatched redeploy actions. +# +# 60s matches node-agent's CHECK_INTERVAL, so remediation latency is the same on +# both paths and well inside the executor's REDEPLOY_TIMEOUT_SECS (900). +# Overlapping runs are prevented by the runner's own flock, not by the timer. +[Unit] +Description=Poll for dispatched redeploy actions every minute + +[Timer] +OnBootSec=2min +OnUnitActiveSec=60s +AccuracySec=15s +Unit=homelab-deploy-runner.service + +[Install] +WantedBy=timers.target diff --git a/jobs/deploy-runner/tests/test_action.py b/jobs/deploy-runner/tests/test_action.py new file mode 100644 index 0000000..cb8a29f --- /dev/null +++ b/jobs/deploy-runner/tests/test_action.py @@ -0,0 +1,238 @@ +"""Tests for the deploy runner's validation / result-reporting helper. + +The validator is the only gate between a JSON file that arrived over the +network and `docker compose up` running on the node, so its rejection cases +matter more than its happy path. The emit-result tests pin the event format +against the control-plane executor's actual parser — if either side drifts, +redeploy actions would hang in running/ until they time out. +""" +from __future__ import annotations + +import json +import shlex +import sys +import time +from pathlib import Path + +import pytest + +_JOB_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(_JOB_DIR)) + +import action as action_mod # noqa: E402 +from action import Invalid, emit_result, validate # noqa: E402 + + +@pytest.fixture +def repo(tmp_path: Path) -> Path: + """Minimal repo: node piha declares vikunja, which has a compose file.""" + repo = tmp_path / "repo" + (repo / "hosts" / "piha").mkdir(parents=True) + (repo / "hosts" / "piha" / "services.yaml").write_text( + "host: piha\nservices:\n vikunja:\n role: task-tracker\n node-agent:\n role: monitor\n" + ) + (repo / "services" / "vikunja").mkdir(parents=True) + (repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n") + (repo / "services" / "gokapi").mkdir(parents=True) + (repo / "services" / "gokapi" / "docker-compose.yml").write_text("services: {}\n") + return repo + + +def _action_file(tmp_path: Path, **overrides) -> Path: + action = { + "action_id": "redeploy-piha-vikunja", + "type": "redeploy", + "node": "piha", + "service": "vikunja", + } + action.update(overrides) + path = tmp_path / "redeploy-piha-vikunja.json" + path.write_text(json.dumps(action)) + return path + + +# --------------------------------------------------------------------------- +# validate +# --------------------------------------------------------------------------- + +def test_valid_action_returns_id_and_service(tmp_path, repo): + action_id, service = validate(_action_file(tmp_path), "piha", repo) + assert action_id == "redeploy-piha-vikunja" + assert service == "vikunja" + + +def test_action_for_another_node_is_refused(tmp_path, repo): + path = _action_file(tmp_path, node="solaria") + with pytest.raises(Invalid, match="addressed to node"): + validate(path, "piha", repo) + + +def test_non_redeploy_type_is_refused(tmp_path, repo): + """The runner executes redeploys only — container_restart stays node-agent's + job, and anything else must never reach a shell.""" + path = _action_file(tmp_path, type="container_restart") + with pytest.raises(Invalid, match="not executable by the deploy runner"): + validate(path, "piha", repo) + + +def test_service_not_in_desired_state_is_refused(tmp_path, repo): + """gokapi has a compose file in the repo but is not in hosts/piha — + desired state is the authority on what this node may run.""" + path = _action_file(tmp_path, service="gokapi") + with pytest.raises(Invalid, match="not in hosts/piha/services.yaml"): + validate(path, "piha", repo) + + +@pytest.mark.parametrize("bad", ["../../etc/passwd", "/absolute", "Vikunja", "", "a b", "x" * 65]) +def test_malformed_service_names_are_refused(tmp_path, repo, bad): + path = _action_file(tmp_path, service=bad) + with pytest.raises(Invalid, match="invalid service name"): + validate(path, "piha", repo) + + +def test_service_without_compose_file_is_refused(tmp_path, repo): + (repo / "services" / "vikunja" / "docker-compose.yml").unlink() + with pytest.raises(Invalid, match="no compose file"): + validate(_action_file(tmp_path), "piha", repo) + + +def test_missing_manifest_for_own_node_is_refused(tmp_path, repo): + (repo / "hosts" / "piha" / "services.yaml").unlink() + with pytest.raises(Invalid, match="no desired-state manifest"): + validate(_action_file(tmp_path), "piha", repo) + + +def test_services_as_list_is_supported(tmp_path, repo): + (repo / "hosts" / "piha" / "services.yaml").write_text("host: piha\nservices:\n - vikunja\n") + _, service = validate(_action_file(tmp_path), "piha", repo) + assert service == "vikunja" + + +def test_unparseable_file_is_refused(tmp_path, repo): + path = tmp_path / "broken.json" + path.write_text("{not json") + with pytest.raises(Invalid, match="unparseable"): + validate(path, "piha", repo) + + +def test_invalid_action_id_is_refused(tmp_path, repo): + path = _action_file(tmp_path, action_id="../escape") + with pytest.raises(Invalid, match="invalid action_id"): + validate(path, "piha", repo) + + +# --------------------------------------------------------------------------- +# validate via the CLI (shell-assignment contract used by deploy-runner.sh) +# --------------------------------------------------------------------------- + +def _eval_cli(argv) -> dict: + """Parse the KEY='value' lines the bash runner evaluates.""" + import io + import contextlib + + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + rc = action_mod.main(argv) + assert rc == 0 + out = {} + for line in buf.getvalue().splitlines(): + key, _, raw = line.partition("=") + out[key] = shlex.split(raw)[0] if raw and raw != "''" else "" + return out + + +def test_cli_valid_action_emits_ok_true(tmp_path, repo): + path = _action_file(tmp_path) + result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)]) + assert result["OK"] == "true" + assert result["SERVICE"] == "vikunja" + assert result["ACTION_ID"] == "redeploy-piha-vikunja" + assert result["ERROR"] == "" + + +def test_cli_rejected_action_still_reports_action_id(tmp_path, repo): + """So the runner can report a failure result instead of silently dropping + the action — a dropped action would sit in running/ until it times out.""" + path = _action_file(tmp_path, service="gokapi") + result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)]) + assert result["OK"] == "false" + assert result["ACTION_ID"] == "redeploy-piha-vikunja" + assert "not in hosts/piha/services.yaml" in result["ERROR"] + + +def test_cli_quotes_hostile_error_content(tmp_path, repo): + """A crafted service name must not be able to inject shell syntax through + the eval'd ERROR assignment.""" + path = _action_file(tmp_path, service="'; touch /tmp/pwned; '") + result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)]) + assert result["OK"] == "false" + assert "touch /tmp/pwned" in result["ERROR"] # inert data, not a command + + +# --------------------------------------------------------------------------- +# emit-result — format compatibility with the control-plane executor +# --------------------------------------------------------------------------- + +def test_emit_result_shape(tmp_path): + events = tmp_path / "events" / "piha" + path = emit_result(events, "piha", "redeploy-piha-vikunja", "vikunja", True, "") + event = json.loads(path.read_text()) + assert event["type"] == "action_result" + assert event["node"] == "piha" + assert event["severity"] == "info" + assert event["payload"]["action_id"] == "redeploy-piha-vikunja" + assert event["payload"]["success"] is True + assert event["payload"]["source"] == "deploy-runner" + + +def test_emit_result_failure_carries_error(tmp_path): + events = tmp_path / "events" / "piha" + path = emit_result(events, "piha", "act-1", "vikunja", False, "compose exited 1") + event = json.loads(path.read_text()) + assert event["severity"] == "high" + assert event["payload"]["success"] is False + assert event["payload"]["error"] == "compose exited 1" + + +def test_executor_finds_the_emitted_result(tmp_path, monkeypatch): + """The real consumer: control-plane executor._find_action_result(). This is + the contract that makes a redeploy resolve instead of timing out.""" + executor_src = _JOB_DIR.parents[1] / "services" / "control-plane" / "src" + sys.path.insert(0, str(executor_src)) + import executor as executor_mod + + events_root = tmp_path / "events" + monkeypatch.setattr(executor_mod, "EVENTS_DIR", events_root) + monkeypatch.setattr(executor_mod, "ACTIONS_DIR", tmp_path / "actions") + monkeypatch.setattr(executor_mod, "DISPATCH_DIR", tmp_path / "actions" / "dispatch") + monkeypatch.setattr(executor_mod, "DEPLOY_DISPATCH_DIR", tmp_path / "actions" / "deploy") + + started_at = time.time() + emit_result(events_root / "piha", "piha", "redeploy-piha-vikunja", "vikunja", True, "") + + payload = executor_mod.Executor()._find_action_result( + "piha", "redeploy-piha-vikunja", started_at + ) + assert payload is not None + assert payload["success"] is True + + +def test_executor_ignores_result_older_than_the_action(tmp_path, monkeypatch): + """Action IDs are deterministic and repeat, so a result from a previous run + must not resolve the current one.""" + executor_src = _JOB_DIR.parents[1] / "services" / "control-plane" / "src" + sys.path.insert(0, str(executor_src)) + import executor as executor_mod + + events_root = tmp_path / "events" + monkeypatch.setattr(executor_mod, "EVENTS_DIR", events_root) + monkeypatch.setattr(executor_mod, "ACTIONS_DIR", tmp_path / "actions") + monkeypatch.setattr(executor_mod, "DISPATCH_DIR", tmp_path / "actions" / "dispatch") + monkeypatch.setattr(executor_mod, "DEPLOY_DISPATCH_DIR", tmp_path / "actions" / "deploy") + + emit_result(events_root / "piha", "piha", "redeploy-piha-vikunja", "vikunja", True, "") + started_at = time.time() + 60 # action started after the event was written + + assert executor_mod.Executor()._find_action_result( + "piha", "redeploy-piha-vikunja", started_at + ) is None diff --git a/jobs/deploy-runner/tests/test_deploy_service.py b/jobs/deploy-runner/tests/test_deploy_service.py new file mode 100644 index 0000000..341e955 --- /dev/null +++ b/jobs/deploy-runner/tests/test_deploy_service.py @@ -0,0 +1,150 @@ +"""Contract tests for scripts/deploy/deploy-service.sh. + +Why these live here: the deploy runner and the human whole-node deploy both go +through this script, and what must stay identical between them is the argv it +hands to `docker compose`. docker compose derives the project name from the +directory of the FIRST -f file — a caller that assembles those arguments +differently lands in a different project, where `--remove-orphans` tears down +the running stack (the 2026-06-25 control-plane wipe on VPS). + +Each test runs the real script with a stub `docker` on PATH that records its +argv, so nothing is executed against a daemon. +""" +from __future__ import annotations + +import os +import subprocess +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[3] +SCRIPT = REPO / "scripts" / "deploy" / "deploy-service.sh" + + +@pytest.fixture +def env(tmp_path: Path): + """A fake repo + host dir + a `docker` stub that logs its arguments.""" + repo = tmp_path / "repo" + (repo / "services" / "vikunja").mkdir(parents=True) + (repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n") + host_dir = repo / "hosts" / "piha" + host_dir.mkdir(parents=True) + + bindir = tmp_path / "bin" + bindir.mkdir() + argv_log = tmp_path / "docker-argv.txt" + stub = bindir / "docker" + stub.write_text(f'#!/usr/bin/env bash\nprintf "%s\\n" "$@" > {argv_log}\nexit 0\n') + stub.chmod(0o755) + + return {"repo": repo, "host_dir": host_dir, "bindir": bindir, "argv_log": argv_log} + + +def run(env, *args, service="vikunja"): + proc_env = dict(os.environ, PATH=f"{env['bindir']}:{os.environ['PATH']}") + return subprocess.run( + [str(SCRIPT), "--repo", str(env["repo"]), "--host-dir", str(env["host_dir"]), + "--service", service, *args], + capture_output=True, text=True, env=proc_env, + ) + + +def argv(env) -> list[str]: + return env["argv_log"].read_text().splitlines() + + +def test_base_invocation(env): + result = run(env) + assert result.returncode == 0, result.stderr + assert argv(env) == [ + "compose", "-f", str(env["repo"] / "services" / "vikunja" / "docker-compose.yml"), + "up", "-d", + ] + + +def test_project_name_invariant(env): + """No -p and no --project-directory: the project name must come from the + compose file's own directory, exactly as it does for a human deploy.""" + run(env, "--force-recreate") + assert "-p" not in argv(env) + assert "--project-directory" not in argv(env) + assert argv(env)[1] == "-f" # first -f is the service's own compose file + + +def test_override_is_second_f(env): + override = env["host_dir"] / "runtime" / "vikunja" + override.mkdir(parents=True) + (override / "docker-compose.override.yml").write_text("services: {}\n") + + run(env) + args = argv(env) + assert args[1:5] == [ + "-f", str(env["repo"] / "services" / "vikunja" / "docker-compose.yml"), + "-f", str(override / "docker-compose.override.yml"), + ] + + +def test_env_file_is_passed_when_present(env): + (env["repo"] / "services" / "vikunja" / ".env").write_text("FOO=bar\n") + run(env) + args = argv(env) + assert "--env-file" in args + assert args[args.index("--env-file") + 1] == str(env["repo"] / "services" / "vikunja" / ".env") + + +def test_force_recreate_flag(env): + """The remediation path needs it: plain `up -d` is a no-op when config is + unchanged, i.e. exactly the unhealthy-container case a redeploy targets.""" + run(env, "--force-recreate") + assert "--force-recreate" in argv(env) + + +def test_no_remove_orphans_by_default(env): + """The agent path must never be able to delete containers on a project-name + mismatch — only the human whole-node deploy opts in.""" + run(env, "--force-recreate") + assert "--remove-orphans" not in argv(env) + + +def test_remove_orphans_when_requested(env): + run(env, "--remove-orphans") + assert "--remove-orphans" in argv(env) + + +def test_build_if_needed_only_builds_with_dockerfile(env): + run(env, "--build-if-needed") + assert "--build" not in argv(env) + + (env["repo"] / "services" / "vikunja" / "Dockerfile").write_text("FROM scratch\n") + run(env, "--build-if-needed") + assert "--build" in argv(env) + + +def test_redeploy_path_never_builds(env): + """The runner does not pass --build-if-needed: building on the 4 GiB VPS + mid-incident is an OOM risk, and a redeploy is a reconcile, not a code ship.""" + (env["repo"] / "services" / "vikunja" / "Dockerfile").write_text("FROM scratch\n") + run(env, "--force-recreate") + assert "--build" not in argv(env) + + +def test_service_with_own_deploy_path_exits_3(env): + (env["repo"] / "services" / "vikunja" / "deploy-local.sh").write_text("#!/bin/sh\n") + result = run(env) + assert result.returncode == 3 + assert not env["argv_log"].exists() # docker never invoked + + +@pytest.mark.parametrize("bad", ["../../etc", "/abs", "Vikunja", "a b"]) +def test_invalid_service_name_exits_2(env, bad): + result = run(env, service=bad) + assert result.returncode == 2 + assert not env["argv_log"].exists() + + +def test_missing_compose_file_exits_2(env): + (env["repo"] / "services" / "vikunja" / "docker-compose.yml").unlink() + result = run(env) + assert result.returncode == 2 + assert not env["argv_log"].exists() diff --git a/jobs/deploy-runner/tests/test_runner_integration.py b/jobs/deploy-runner/tests/test_runner_integration.py new file mode 100644 index 0000000..c7c51ad --- /dev/null +++ b/jobs/deploy-runner/tests/test_runner_integration.py @@ -0,0 +1,163 @@ +"""End-to-end test of deploy-runner.sh on a fake node. + +Runs the real bash runner against temp directories with a stubbed `docker`, in +local (VPS) mode so no rsync/ssh is involved. Covers the full loop the control +plane depends on: dispatch file in → validate → deploy → action_result event +out → idempotency marker → inbox drained. +""" +from __future__ import annotations + +import json +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[3] +RUNNER = REPO / "jobs" / "deploy-runner" / "deploy-runner.sh" + + +@pytest.fixture +def node(tmp_path: Path): + """A fake node: repo checkout, /opt/homelab-alike runtime, docker stub.""" + repo = tmp_path / "repo" + (repo / "scripts" / "deploy").mkdir(parents=True) + shutil.copy(REPO / "scripts" / "deploy" / "deploy-service.sh", + repo / "scripts" / "deploy" / "deploy-service.sh") + os.chmod(repo / "scripts" / "deploy" / "deploy-service.sh", 0o755) + + (repo / "hosts" / "piha").mkdir(parents=True) + (repo / "hosts" / "piha" / "services.yaml").write_text( + "host: piha\nservices:\n vikunja:\n role: task-tracker\n" + ) + (repo / "services" / "vikunja").mkdir(parents=True) + (repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n") + + runtime = tmp_path / "runtime" + (runtime / "actions" / "deploy" / "piha").mkdir(parents=True) + + bindir = tmp_path / "bin" + bindir.mkdir() + argv_log = tmp_path / "docker-argv.txt" + (bindir / "docker").write_text( + f'#!/usr/bin/env bash\nprintf "%s " "$@" >> {argv_log}\necho >> {argv_log}\n' + f'exit ${{DOCKER_STUB_EXIT:-0}}\n' + ) + os.chmod(bindir / "docker", 0o755) + + return {"repo": repo, "runtime": runtime, "bindir": bindir, "argv_log": argv_log, + "inbox": runtime / "actions" / "deploy" / "piha", + "events": runtime / "events" / "piha"} + + +def dispatch(fake_node, action_id="redeploy-piha-vikunja", **overrides): + action = {"action_id": action_id, "type": "redeploy", "node": "piha", + "service": "vikunja", "dispatched_at": 0} + action.update(overrides) + (fake_node["inbox"] / f"{action_id}.json").write_text(json.dumps(action)) + + +def run_runner(fake_node, **extra_env): + env = dict(os.environ) + env.update({ + "PATH": f"{fake_node['bindir']}:{os.environ['PATH']}", + "NODE_NAME": "piha", + "REPO_PATH": str(fake_node["repo"]), + "RUNTIME_PATH": str(fake_node["runtime"]), + "VPS_EVENTS_HOST": "", # local mode: executor shares the same filesystem + }) + env.update(extra_env) + return subprocess.run([str(RUNNER)], capture_output=True, text=True, env=env) + + +def results(node) -> list[dict]: + if not node["events"].exists(): + return [] + return [json.loads(p.read_text()) + for p in sorted(node["events"].glob("evt-*-action_result-*.json"))] + + +def test_successful_redeploy_reports_and_drains(node): + dispatch(node) + proc = run_runner(node) + assert proc.returncode == 0, proc.stdout + proc.stderr + + assert "compose -f" in node["argv_log"].read_text() + assert "--force-recreate" in node["argv_log"].read_text() + + events = results(node) + assert len(events) == 1 + assert events[0]["payload"] == { + "action_id": "redeploy-piha-vikunja", "success": True, "error": "", + "node": "piha", "source": "deploy-runner", + } + + assert list(node["inbox"].glob("*.json")) == [] + assert (node["runtime"] / "state" / "processed-deploy-actions" + / "redeploy-piha-vikunja.done").exists() + + +def test_failed_deploy_reports_compose_output(node): + dispatch(node) + proc = run_runner(node, DOCKER_STUB_EXIT="1") + assert proc.returncode == 0 # the runner itself succeeds; the action fails + + event = results(node)[0] + assert event["payload"]["success"] is False + assert "deploy-service.sh exited 1" in event["payload"]["error"] + assert event["severity"] == "high" + # Full output kept on the node for forensics + assert list((node["runtime"] / "logs" / "deploy-runner").glob("*.log")) + + +def test_rejected_action_is_reported_not_silently_dropped(node): + """An action for a service this node does not declare must come back as a + failure — otherwise it sits in running/ until the executor times it out.""" + dispatch(node, service="gokapi") + run_runner(node) + + event = results(node)[0] + assert event["payload"]["success"] is False + assert "not in hosts/piha/services.yaml" in event["payload"]["error"] + assert not node["argv_log"].exists() # docker never invoked + assert list(node["inbox"].glob("*.json")) == [] + + +def test_action_for_another_node_is_refused(node): + dispatch(node, node="solaria") + run_runner(node) + + event = results(node)[0] + assert "addressed to node" in event["payload"]["error"] + assert not node["argv_log"].exists() + + +def test_container_restart_is_not_executed_by_the_runner(node): + """node-agent's job. The runner refuses anything but redeploy.""" + dispatch(node, type="container_restart") + run_runner(node) + + event = results(node)[0] + assert "not executable by the deploy runner" in event["payload"]["error"] + assert not node["argv_log"].exists() + + +def test_already_processed_action_is_not_redeployed(node): + dispatch(node) + run_runner(node) + first = len(results(node)) + + dispatch(node) # same action_id arrives again + run_runner(node) + + assert len(results(node)) == first # no second result + assert len(node["argv_log"].read_text().splitlines()) == 1 # docker ran once + assert list(node["inbox"].glob("*.json")) == [] + + +def test_missing_node_name_fails_loudly(node): + proc = run_runner(node, NODE_NAME="") + assert proc.returncode == 1 + assert "NODE_NAME is not set" in proc.stdout + proc.stderr diff --git a/scripts/deploy/deploy-node.sh b/scripts/deploy/deploy-node.sh index c76e143..b15f710 100755 --- a/scripts/deploy/deploy-node.sh +++ b/scripts/deploy/deploy-node.sh @@ -6,7 +6,9 @@ set -e # Configuration REPO_PATH="${HOME}/homelab-codex-ws" -RUNTIME_PATH="/opt/homelab" +# Env-overridable so the script can be exercised against a scratch runtime dir +# without writing into the node's real /opt/homelab. +RUNTIME_PATH="${RUNTIME_PATH:-/opt/homelab}" HOSTNAME=$(hostname | tr '[:lower:]' '[:upper:]') CURRENT_OS_HOST=$(hostname) @@ -81,57 +83,33 @@ if [ ${#SERVICES[@]} -eq 0 ]; then fi # 3. Deploy Services +# +# The per-service compose invocation lives in deploy-service.sh, shared with the +# agent-driven single-service redeploy (services/deploy-runner/). Both callers +# MUST assemble the -f arguments identically: docker compose derives the project +# name from the first -f file's directory, and a mismatch plus --remove-orphans +# is what wiped the control-plane on VPS on 2026-06-25. Services with their own +# deploy-local.sh are skipped by deploy-service.sh (exit 3). +DEPLOY_SERVICE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/deploy-service.sh" + for service in "${SERVICES[@]}"; do - # Services that ship their own deploy-local.sh have a dedicated, orchestrated - # deploy path (e.g. control-plane via deploy-control-plane.sh → deploy-local.sh). - # Deploying them in this loop too would invoke `docker compose up` with a - # different Compose project name than their dedicated path uses (this loop runs - # from ${REPO_PATH}; deploy-local.sh runs from services//), so `up - # -d --remove-orphans` would Recreate and tear down the already-running - # containers — exactly what wiped the control-plane on VPS on 2026-06-25. - # Their own deploy path owns them; skip here. - if [ -f "${REPO_PATH}/services/${service}/deploy-local.sh" ]; then - echo "Skipping ${service}: ma własną ścieżkę deployu (services/${service}/deploy-local.sh), pomijam" - continue - fi - - echo "Deploying service: ${service}..." - - COMPOSE_FILE="${REPO_PATH}/services/${service}/docker-compose.yml" - - if [ ! -f "$COMPOSE_FILE" ]; then - echo "Warning: Compose file not found for ${service} at ${COMPOSE_FILE}" - continue - fi - TARGET_DIR="${RUNTIME_PATH}/services/${service}" mkdir -p "$TARGET_DIR" - OVERRIDE_FILE="${HOST_DIR}/runtime/${service}/docker-compose.override.yml" + rc=0 + "$DEPLOY_SERVICE" \ + --repo "$REPO_PATH" \ + --host-dir "$HOST_DIR" \ + --service "$service" \ + --build-if-needed \ + --remove-orphans || rc=$? - COMPOSE_CMD="docker compose -f ${COMPOSE_FILE}" - if [ -f "$OVERRIDE_FILE" ]; then - echo "Using override file for ${service}" - COMPOSE_CMD="${COMPOSE_CMD} -f ${OVERRIDE_FILE}" - fi - - ENV_FILE="${REPO_PATH}/services/${service}/.env" - if [ -f "$ENV_FILE" ]; then - COMPOSE_CMD="${COMPOSE_CMD} --env-file ${ENV_FILE}" - fi - - # Services with their own Dockerfile need a rebuild on every deploy — - # otherwise `up -d` sees "container running, image tag unchanged" and skips - # rebuilding even when src/ changed, silently leaving the old code running - # (backlog c858dbc). Services with a prebuilt image (no Dockerfile) don't - # need --build. - if [ -f "${REPO_PATH}/services/${service}/Dockerfile" ]; then - echo "Building ${service} (has Dockerfile)" - $COMPOSE_CMD up -d --build --remove-orphans - else - echo "Using prebuilt image for ${service}" - $COMPOSE_CMD up -d --remove-orphans - fi + case "$rc" in + 0) ;; # deployed + 3) ;; # owns its deploy path — already reported, skip + 2) echo "Warning: skipping ${service} (validation failed, see above)" ;; + *) exit "$rc" ;; # real deploy failure — preserve set -e semantics + esac done echo "--- Deployment Complete ---" diff --git a/scripts/deploy/deploy-service.sh b/scripts/deploy/deploy-service.sh new file mode 100755 index 0000000..62ce2df --- /dev/null +++ b/scripts/deploy/deploy-service.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# scripts/deploy/deploy-service.sh — deploy exactly ONE service on the local node. +# +# Extracted from deploy-node.sh so that the human whole-node deploy and the +# agent-driven single-service redeploy (services/deploy-runner/) share ONE +# compose invocation. That sharing is the whole point: docker compose derives +# the PROJECT NAME from the directory of the first -f file (verified: +# `docker compose -f services/node-agent/docker-compose.yml config` → name +# "node-agent", independent of cwd; adding --project-directory changes it). +# A caller that assembles the -f arguments differently lands in a DIFFERENT +# project, where `--remove-orphans` tears down the already-running stack — +# exactly what wiped the control-plane on VPS on 2026-06-25. Every caller goes +# through here. +# +# Usage: +# deploy-service.sh --repo --host-dir path> +# --service +# [--build-if-needed] [--force-recreate] [--remove-orphans] +# +# Flags: +# --build-if-needed add --build when the service ships its own Dockerfile +# (whole-node deploy path — see the rationale below) +# --force-recreate recreate containers even when config is unchanged +# (remediation path: plain `up -d` is a no-op for an +# unhealthy-but-unchanged service, i.e. it would "succeed" +# without doing anything) +# --remove-orphans human whole-node deploy only; never used by the agent +# path, where a project-name mismatch must not be able to +# delete containers +# +# Exit codes: +# 0 deployed +# 2 usage / validation error +# 3 service owns its deploy path (services//deploy-local.sh) — skipped +# * docker compose exit code + +set -euo pipefail + +REPO_PATH="" +HOST_DIR="" +SERVICE="" +BUILD_IF_NEEDED=false +FORCE_RECREATE=false +REMOVE_ORPHANS=false + +usage() { + sed -n '2,30p' "${BASH_SOURCE[0]}" >&2 + exit 2 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --repo) REPO_PATH="${2:-}"; shift 2 ;; + --host-dir) HOST_DIR="${2:-}"; shift 2 ;; + --service) SERVICE="${2:-}"; shift 2 ;; + --build-if-needed) BUILD_IF_NEEDED=true; shift ;; + --force-recreate) FORCE_RECREATE=true; shift ;; + --remove-orphans) REMOVE_ORPHANS=true; shift ;; + -h|--help) usage ;; + *) echo "deploy-service.sh: unknown argument: $1" >&2; usage ;; + esac +done + +[[ -n "$REPO_PATH" && -n "$HOST_DIR" && -n "$SERVICE" ]] || { + echo "deploy-service.sh: --repo, --host-dir and --service are required" >&2 + usage +} + +# Service names are used to build filesystem paths. Reject anything that is not +# a plain kebab-case service name so a malformed (or maliciously crafted) +# dispatched action can never escape services/ via ../. +if [[ ! "$SERVICE" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]]; then + echo "deploy-service.sh: invalid service name: '${SERVICE}'" >&2 + exit 2 +fi + +[[ -d "$REPO_PATH" ]] || { echo "deploy-service.sh: repo not found: ${REPO_PATH}" >&2; exit 2; } +[[ -d "$HOST_DIR" ]] || { echo "deploy-service.sh: host dir not found: ${HOST_DIR}" >&2; exit 2; } + +SERVICE_DIR="${REPO_PATH}/services/${SERVICE}" +COMPOSE_FILE="${SERVICE_DIR}/docker-compose.yml" + +if [[ ! -f "$COMPOSE_FILE" ]]; then + echo "deploy-service.sh: compose file not found for ${SERVICE} at ${COMPOSE_FILE}" >&2 + exit 2 +fi + +# Services that ship their own deploy-local.sh have a dedicated, orchestrated +# deploy path (e.g. control-plane via deploy-control-plane.sh → deploy-local.sh) +# which does host-level preparation (dirs, ownership, sudo) this script cannot +# do. Deploying them from here would also run `up` against a stack whose own +# path expects to own it. Their own deploy path owns them; skip. +if [[ -f "${SERVICE_DIR}/deploy-local.sh" ]]; then + echo "Skipping ${SERVICE}: ma własną ścieżkę deployu (services/${SERVICE}/deploy-local.sh), pomijam" + exit 3 +fi + +COMPOSE_ARGS=(docker compose -f "$COMPOSE_FILE") + +OVERRIDE_FILE="${HOST_DIR}/runtime/${SERVICE}/docker-compose.override.yml" +if [[ -f "$OVERRIDE_FILE" ]]; then + echo "Using override file for ${SERVICE}" + COMPOSE_ARGS+=(-f "$OVERRIDE_FILE") +fi + +ENV_FILE="${SERVICE_DIR}/.env" +if [[ -f "$ENV_FILE" ]]; then + COMPOSE_ARGS+=(--env-file "$ENV_FILE") +fi + +COMPOSE_ARGS+=(up -d) + +# Services with their own Dockerfile need a rebuild on a whole-node deploy — +# otherwise `up -d` sees "container running, image tag unchanged" and skips +# rebuilding even when src/ changed, silently leaving the old code running +# (backlog c858dbc). The agent redeploy path deliberately does NOT pass +# --build-if-needed: a redeploy is a reconcile, not a code ship, and building +# on the 4 GiB VPS mid-incident is an OOM risk. +if [[ "$BUILD_IF_NEEDED" == "true" && -f "${SERVICE_DIR}/Dockerfile" ]]; then + echo "Building ${SERVICE} (has Dockerfile)" + COMPOSE_ARGS+=(--build) +fi + +if [[ "$FORCE_RECREATE" == "true" ]]; then + COMPOSE_ARGS+=(--force-recreate) +fi + +if [[ "$REMOVE_ORPHANS" == "true" ]]; then + COMPOSE_ARGS+=(--remove-orphans) +fi + +echo "Deploying service: ${SERVICE}..." +echo "+ ${COMPOSE_ARGS[*]}" +"${COMPOSE_ARGS[@]}" diff --git a/services/control-plane/docker-compose.yml b/services/control-plane/docker-compose.yml index 0cde0ea..6a9dc5c 100644 --- a/services/control-plane/docker-compose.yml +++ b/services/control-plane/docker-compose.yml @@ -79,7 +79,11 @@ services: command: python src/executor.py volumes: - /opt/homelab:/opt/homelab - - ../..:/repo + # Read-only since the redeploy fix: the executor used to run + # scripts/deploy/deploy-node.sh out of this mount (it never worked — see + # jobs/deploy-runner/README.md). Deploys now happen on the node itself, so + # nothing here needs write access to the checkout. + - ../..:/repo:ro - /var/run/docker.sock:/var/run/docker.sock restart: unless-stopped environment: diff --git a/services/control-plane/src/executor.py b/services/control-plane/src/executor.py index 710c34e..3618d9d 100644 --- a/services/control-plane/src/executor.py +++ b/services/control-plane/src/executor.py @@ -21,6 +21,15 @@ RUNTIME_PATH = os.getenv("RUNTIME_PATH", "/opt/homelab") ACTIONS_DIR = Path(RUNTIME_PATH) / "actions" EVENTS_DIR = Path(RUNTIME_PATH) / "events" DISPATCH_DIR = ACTIONS_DIR / "dispatch" +# Separate inbox for redeploy actions, consumed by the host-side deploy runner +# (jobs/deploy-runner/) rather than by node-agent. It must NOT share +# DISPATCH_DIR: node-agent's process_dispatched_actions() deletes and +# failure-reports every file it finds in its inbox, so a redeploy landing there +# would be killed before the runner ever saw it. +DEPLOY_DISPATCH_DIR = ACTIONS_DIR / "deploy" +# The executor no longer reads the repo at all (the old redeploy path ran a +# script out of it). Kept only so an operator can still see which checkout the +# container is wired to; nothing in this module resolves paths against it. REPO_ROOT = Path(os.getenv("REPO_ROOT", "/repo")) # SSH configuration @@ -41,6 +50,15 @@ SSH_OPTIONS = [ # code change. ACTION_TIMEOUT_SECS = int(os.getenv("ACTION_TIMEOUT_SECS", "300")) +# Redeploy gets its own, longer budget: the deploy runner polls once a minute +# and a `docker compose up` may pull images before it can report anything. +REDEPLOY_TIMEOUT_SECS = int(os.getenv("REDEPLOY_TIMEOUT_SECS", "900")) + +# Action types the executor hands to an on-node agent instead of resolving +# synchronously. They stay in running/ until an action_result event arrives (or +# the type's timeout expires) — see _reconcile_running_actions. +DISPATCHED_ACTION_TYPES = {"container_restart", "redeploy"} + # Matches evt----.json, same convention as # node_agent.py / observer.py / operator_ui.py. _EVENT_TS_RE = re.compile(r"-(\d{9,11})-") @@ -58,6 +76,7 @@ class Executor: for s in ["approved", "running", "completed", "failed", "rejected"]: (ACTIONS_DIR / s).mkdir(parents=True, exist_ok=True) DISPATCH_DIR.mkdir(parents=True, exist_ok=True) + DEPLOY_DISPATCH_DIR.mkdir(parents=True, exist_ok=True) def process_actions(self): # Update heartbeat @@ -104,19 +123,25 @@ class Executor: service = data.get("service") if action_type == "redeploy": - # Full service redeploy via the repo deploy script - cmd = [ - str(REPO_ROOT / "scripts" / "deploy" / "deploy-node.sh"), - node, - service - ] - logger.info(f"Running command: {' '.join(cmd)}") - result = subprocess.run(cmd, capture_output=True, text=True, cwd=str(REPO_ROOT)) - if result.returncode == 0: - success = True - else: + # Single-service redeploy, dispatched to the host-side deploy + # runner on the target node (jobs/deploy-runner/). Same pull + # architecture as container_restart — the executor never runs a + # deploy itself. It used to try: it ran deploy-node.sh inside + # this container, a script that ignores its arguments and needs + # a repo at ${HOME}/homelab-codex-ws (plus git and the docker + # CLI) that does not exist here, so every redeploy failed at the + # first line (recon D14/D15). + if not node or not service: success = False - error_msg = result.stderr or result.stdout + error_msg = ( + f"redeploy requires both node and service " + f"(node={node!r}, service={service!r})" + ) + else: + self._dispatch_redeploy(action_id, node, service) + # Stays in "running" until the runner reports an + # action_result — same as container_restart below. + return elif action_type == "container_restart": # No SSH from the executor (see CLAUDE.md / docs/backlog.md): the @@ -210,12 +235,47 @@ class Executor: except Exception as e: logger.error(f"Failed to dispatch {action_id} to {node}: {e}") - def _reconcile_running_actions(self): - """Resolve container_restart actions previously dispatched to a node-agent. + def _dispatch_redeploy(self, action_id, node, service): + """Write a redeploy action for the deploy runner on `node` to pick up. - Other action types (redeploy/disk_cleanup/alert_only) resolve - synchronously inside _execute_action and never linger in running/, so - they are not touched here. + Deliberately a different inbox from container_restart: node-agent + consumes actions/dispatch// and rejects (and deletes) anything + that is not container_restart, so redeploys get their own + actions/deploy// tree. The runner on the node pulls it over the + same rsync/ssh channel node-agent already uses, deploys locally with + scripts/deploy/deploy-service.sh, and reports back as an action_result + event — the executor still never initiates a connection to a node. + + Does not resolve the action itself; _reconcile_running_actions() does. + """ + inbox = DEPLOY_DISPATCH_DIR / node + inbox.mkdir(parents=True, exist_ok=True) + payload = { + "action_id": action_id, + "type": "redeploy", + "node": node, + "service": service, + "dispatched_at": time.time(), + } + try: + _atomic_write_json(inbox / f"{action_id}.json", payload) + logger.info( + f"Dispatched redeploy {action_id} (service={service}) " + f"to deploy runner on {node}" + ) + except Exception as e: + logger.error(f"Failed to dispatch {action_id} to {node}: {e}") + + def _reconcile_running_actions(self): + """Resolve actions previously dispatched to an on-node agent. + + Covers both dispatched types: container_restart (executed by node-agent) + and redeploy (executed by the host-side deploy runner). Both report the + outcome the same way — an action_result event carrying the action_id — + so the resolution logic is identical; only the timeout differs. + + Synchronous types (disk_cleanup/alert_only) resolve inside + _execute_action and never linger in running/, so they are not touched. """ running_dir = ACTIONS_DIR / "running" if not running_dir.exists(): @@ -229,24 +289,29 @@ class Executor: logger.error(f"Failed to read running action {action_file.name}: {e}") continue - if data.get("type") != "container_restart": + action_type = data.get("type") + if action_type not in DISPATCHED_ACTION_TYPES: continue action_id = data.get("action_id") or action_file.stem node = data.get("node") started_at = data.get("started_at") or 0 + executor_name = "deploy runner" if action_type == "redeploy" else "node-agent" + timeout_secs = ( + REDEPLOY_TIMEOUT_SECS if action_type == "redeploy" else ACTION_TIMEOUT_SECS + ) result = self._find_action_result(node, action_id, started_at) if result is not None: success = bool(result.get("success")) - error_msg = "" if success else (result.get("error") or "node-agent reported failure") + error_msg = "" if success else (result.get("error") or f"{executor_name} reported failure") self._finalize_action(action_id, action_file, data, success, error_msg) continue - if started_at and (time.time() - started_at) > ACTION_TIMEOUT_SECS: + if started_at and (time.time() - started_at) > timeout_secs: error_msg = ( - f"Timed out after {ACTION_TIMEOUT_SECS}s waiting for node-agent " - f"on '{node}' to report a result for container_restart " + f"Timed out after {timeout_secs}s waiting for {executor_name} " + f"on '{node}' to report a result for {action_type} " f"(action_id={action_id})" ) logger.error(f"Action {action_id} timed out: {error_msg}") diff --git a/services/control-plane/src/supervisor.py b/services/control-plane/src/supervisor.py index 93e066d..4ca5303 100644 --- a/services/control-plane/src/supervisor.py +++ b/services/control-plane/src/supervisor.py @@ -37,13 +37,14 @@ except Exception: # rather than a full redeploy: the container is present but not running, or # running with a failing health check — a restart plausibly heals both. # healthcheck_failed added 2026-07-29 (recon D14/D15): it used to route to -# redeploy, which is broken as wired (executor calls deploy-node.sh with -# arguments it ignores, at a path that does not exist in the container), so -# 3376 healthcheck_failed events dead-ended. redeploy returns to the map only -# once it actually works (etap 2). Everything not listed here -# (service_unhealthy, deployment_failed, missing_service) still falls through -# to redeploy — theoretical until etap 2 fixes the executor, kept as-is so the -# drift stays visible in pending actions. +# redeploy, which was broken as wired (the executor ran deploy-node.sh with +# arguments it ignores, from a repo path that does not exist in its container), +# so 3376 healthcheck_failed events dead-ended. Redeploy works as of 2026-08-03 +# (executor dispatch → jobs/deploy-runner/ on the node), but healthcheck_failed +# deliberately STAYS here: restart-first is the cheaper, lower-risk remediation +# for a container that is up with a failing probe. Everything not listed here +# (service_unhealthy, deployment_failed, missing_service) falls through to +# redeploy, which now actually executes. # mqtt_unreachable was removed 2026-07-28: the observer never creates incidents # with that trigger_type, so the branch was dead code (recon # docs/architecture/RECON-multiagent-2026-07-27.md, D15). stability-agent still @@ -446,10 +447,9 @@ class Supervisor: else: # Full redeploy: container is running but service is broken, # or the cause is unknown / not a simple restart candidate. - # NOTE: the redeploy action type is currently theoretical — the - # executor's redeploy path is broken until etap 2 (see - # CONTAINER_RESTART_TRIGGERS comment). Generated anyway so the - # drift is visible to the operator in pending actions. + # Executed on the target node by jobs/deploy-runner/ once an + # operator approves — `docker compose up -d --force-recreate` for + # this one service, from the checkout the node already has. action = { "action_id": action_id, "timestamp": time.time(), diff --git a/services/control-plane/tests/test_executor_dispatch.py b/services/control-plane/tests/test_executor_dispatch.py index f1ef3b4..9b3011d 100644 --- a/services/control-plane/tests/test_executor_dispatch.py +++ b/services/control-plane/tests/test_executor_dispatch.py @@ -31,6 +31,9 @@ def _setup_executor(tmp_path: Path, monkeypatch) -> Executor: monkeypatch.setattr(executor_module, "ACTIONS_DIR", actions) monkeypatch.setattr(executor_module, "EVENTS_DIR", events) monkeypatch.setattr(executor_module, "DISPATCH_DIR", actions / "dispatch") + # Must be patched too: _ensure_dirs() creates it, and the module default + # points at the real /opt/homelab. + monkeypatch.setattr(executor_module, "DEPLOY_DISPATCH_DIR", actions / "deploy") monkeypatch.setattr(executor_module, "REPO_ROOT", repo) return Executor() @@ -212,17 +215,19 @@ def test_reconcile_does_not_time_out_before_deadline(tmp_path, monkeypatch): # Other action types are untouched by the reconcile loop # --------------------------------------------------------------------------- -def test_reconcile_ignores_non_container_restart_running_actions(tmp_path, monkeypatch): +def test_reconcile_ignores_synchronous_running_actions(tmp_path, monkeypatch): + """Only dispatched types (container_restart, redeploy) are reconciled here. + disk_cleanup/alert_only resolve inside _execute_action and never linger.""" ex = _setup_executor(tmp_path, monkeypatch) - running_path = tmp_path / "actions" / "running" / "redeploy-x.json" + running_path = tmp_path / "actions" / "running" / "disk-cleanup-x.json" running_path.write_text(json.dumps({ - "action_id": "redeploy-x", "type": "redeploy", "node": "piha", + "action_id": "disk-cleanup-x", "type": "disk_cleanup", "node": "piha", "started_at": time.time() - 999_999, })) ex._reconcile_running_actions() - assert running_path.exists() # untouched: redeploy resolves synchronously elsewhere + assert running_path.exists() # untouched def test_alert_only_action_resolves_synchronously_not_via_dispatch(tmp_path, monkeypatch): diff --git a/services/control-plane/tests/test_executor_redeploy.py b/services/control-plane/tests/test_executor_redeploy.py new file mode 100644 index 0000000..a583835 --- /dev/null +++ b/services/control-plane/tests/test_executor_redeploy.py @@ -0,0 +1,189 @@ +"""Tests for the Executor redeploy path (recon D14/D15 — etap 2). + +Before this, `redeploy` ran scripts/deploy/deploy-node.sh inside the executor +container: a script that ignores its arguments, expects a repo at +${HOME}/homelab-codex-ws (absent there, along with git and the docker CLI), and +would have deployed the EXECUTOR's own host's full service set rather than the +action's target. Every redeploy failed, which is why healthcheck_failed had to +be rerouted to container_restart and the action queue never drained. + +Now redeploy follows the same pull architecture as container_restart: the +executor writes a dispatch file, the host-side deploy runner on the target node +(jobs/deploy-runner/) executes it and reports back an action_result event. +""" +from __future__ import annotations + +import json +import sys +import time +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent / "src")) +import executor as executor_module +from executor import Executor + + +def _setup_executor(tmp_path: Path, monkeypatch) -> Executor: + actions = tmp_path / "actions" + events = tmp_path / "events" + for d in (actions, events): + d.mkdir(parents=True, exist_ok=True) + monkeypatch.setattr(executor_module, "ACTIONS_DIR", actions) + monkeypatch.setattr(executor_module, "EVENTS_DIR", events) + monkeypatch.setattr(executor_module, "DISPATCH_DIR", actions / "dispatch") + monkeypatch.setattr(executor_module, "DEPLOY_DISPATCH_DIR", actions / "deploy") + return Executor() + + +def _write_approved(tmp_path, action_id, node="piha", service="vikunja"): + action = { + "action_id": action_id, + "type": "redeploy", + "node": node, + "service": service, + "status": "approved", + "timestamp": time.time(), + } + path = tmp_path / "actions" / "approved" / f"{action_id}.json" + path.write_text(json.dumps(action)) + return path + + +def _write_action_result_event(tmp_path, node, action_id, success, error="", ts=None): + ts = int(ts if ts is not None else time.time()) + node_dir = tmp_path / "events" / node + node_dir.mkdir(parents=True, exist_ok=True) + event_id = f"evt-{node}-{ts}-action_result-{action_id}" + (node_dir / f"{event_id}.json").write_text(json.dumps({ + "id": event_id, + "timestamp": ts, + "type": "action_result", + "node": node, + "payload": {"action_id": action_id, "success": success, "error": error, + "node": node, "source": "deploy-runner"}, + })) + + +def _exists(tmp_path, state, action_id): + return (tmp_path / "actions" / state / f"{action_id}.json").exists() + + +def _read(tmp_path, state, action_id): + return json.loads((tmp_path / "actions" / state / f"{action_id}.json").read_text()) + + +# --------------------------------------------------------------------------- +# Dispatch +# --------------------------------------------------------------------------- + +def test_redeploy_is_dispatched_not_executed_locally(tmp_path, monkeypatch): + ex = _setup_executor(tmp_path, monkeypatch) + + def fail(*args, **kwargs): + raise AssertionError("executor must never run a deploy itself") + + monkeypatch.setattr(executor_module.subprocess, "run", fail) + + ex._execute_action(_write_approved(tmp_path, "redeploy-piha-vikunja")) + + dispatch_file = tmp_path / "actions" / "deploy" / "piha" / "redeploy-piha-vikunja.json" + assert dispatch_file.exists() + payload = json.loads(dispatch_file.read_text()) + assert payload["type"] == "redeploy" + assert payload["node"] == "piha" + assert payload["service"] == "vikunja" + assert "dispatched_at" in payload + + +def test_redeploy_inbox_is_separate_from_node_agent_dispatch(tmp_path, monkeypatch): + """node-agent deletes and failure-reports anything in its own inbox that is + not container_restart, so a redeploy must never be written there.""" + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "redeploy-piha-vikunja")) + + node_agent_inbox = tmp_path / "actions" / "dispatch" + assert not list(node_agent_inbox.glob("**/*.json")) + + +def test_redeploy_stays_running_until_reported(tmp_path, monkeypatch): + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "rd-1")) + + assert _exists(tmp_path, "running", "rd-1") + assert not _exists(tmp_path, "completed", "rd-1") + assert not _exists(tmp_path, "failed", "rd-1") + + +@pytest.mark.parametrize("node,service", [("", "vikunja"), ("piha", ""), ("", "")]) +def test_redeploy_without_node_or_service_fails_immediately(tmp_path, monkeypatch, node, service): + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "rd-bad", node=node, service=service)) + + assert _exists(tmp_path, "failed", "rd-bad") + assert "requires both node and service" in _read(tmp_path, "failed", "rd-bad")["error"] + assert not list((tmp_path / "actions" / "deploy").glob("**/*.json")) + + +# --------------------------------------------------------------------------- +# Reconcile +# --------------------------------------------------------------------------- + +def test_redeploy_completes_on_success_result(tmp_path, monkeypatch): + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "rd-2")) + + _write_action_result_event(tmp_path, "piha", "rd-2", success=True) + ex._reconcile_running_actions() + + assert _exists(tmp_path, "completed", "rd-2") + assert not _exists(tmp_path, "running", "rd-2") + + +def test_redeploy_fails_on_failure_result(tmp_path, monkeypatch): + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "rd-3")) + + _write_action_result_event(tmp_path, "piha", "rd-3", success=False, + error="deploy-service.sh exited 1") + ex._reconcile_running_actions() + + assert _read(tmp_path, "failed", "rd-3")["error"] == "deploy-service.sh exited 1" + + +def test_redeploy_uses_its_own_longer_timeout(tmp_path, monkeypatch): + """A redeploy must not be timed out on the container_restart budget: the + runner polls once a minute and may pull images before reporting.""" + monkeypatch.setattr(executor_module, "ACTION_TIMEOUT_SECS", 300) + monkeypatch.setattr(executor_module, "REDEPLOY_TIMEOUT_SECS", 900) + ex = _setup_executor(tmp_path, monkeypatch) + ex._execute_action(_write_approved(tmp_path, "rd-4")) + + running_path = tmp_path / "actions" / "running" / "rd-4.json" + data = json.loads(running_path.read_text()) + data["started_at"] = time.time() - 600 # past 300s, inside 900s + running_path.write_text(json.dumps(data)) + + ex._reconcile_running_actions() + assert _exists(tmp_path, "running", "rd-4") + + data["started_at"] = time.time() - 1000 # past 900s + running_path.write_text(json.dumps(data)) + + ex._reconcile_running_actions() + assert _exists(tmp_path, "failed", "rd-4") + error = _read(tmp_path, "failed", "rd-4")["error"] + assert "Timed out after 900s" in error + assert "deploy runner" in error + + +def test_stale_result_from_previous_run_does_not_resolve(tmp_path, monkeypatch): + ex = _setup_executor(tmp_path, monkeypatch) + _write_action_result_event(tmp_path, "piha", "rd-5", success=True, + ts=int(time.time()) - 86_400) + + ex._execute_action(_write_approved(tmp_path, "rd-5")) + ex._reconcile_running_actions() + + assert _exists(tmp_path, "running", "rd-5")