Compare commits
2 commits
4c457a8006
...
79bfe8ceef
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
79bfe8ceef | ||
|
|
da151fc8d3 |
16
CLAUDE.md
16
CLAUDE.md
|
|
@ -90,9 +90,21 @@ The platform uses a multi-agent model with **human-in-the-loop** for destructive
|
|||
Agent → /opt/homelab/actions/pending/<id>.json
|
||||
→ Telegram notification → Operator approves
|
||||
→ /opt/homelab/actions/approved/<id>.json
|
||||
→ Executor runs → completed / failed
|
||||
→ Executor dispatches to the target node → completed / failed
|
||||
```
|
||||
|
||||
The executor never connects to a node (deliberate — see docs/backlog.md
|
||||
"Remediacja floty bez SSH"). It writes a dispatch file that the node collects:
|
||||
|
||||
| Action type | Inbox | Executed on the node by |
|
||||
|---|---|---|
|
||||
| `container_restart` | `actions/dispatch/<node>/` | node-agent (own docker socket) |
|
||||
| `redeploy` | `actions/deploy/<node>/` | deploy-runner, host-level systemd (`jobs/deploy-runner/`) |
|
||||
|
||||
Both report back with an `action_result` event, which the executor turns into
|
||||
`completed` / `failed`. `disk_cleanup` and `alert_only` still resolve inside the
|
||||
executor.
|
||||
|
||||
Agents must never execute destructive actions (restarts, deploys, config changes) without a corresponding approved action file.
|
||||
|
||||
## Event System
|
||||
|
|
@ -109,7 +121,7 @@ Normalized event types: `deployment_started/completed/failed`, `service_unhealth
|
|||
|---|---|---|---|
|
||||
| `containers_not_running` | stability-agent | `container_restart` | dedup via stable ID |
|
||||
| `healthcheck_failed` | node-agent | `container_restart` | dedup via stable ID |
|
||||
| `service_unhealthy` / other | stability-agent | `redeploy` (broken until etap 2 executor fix) | dedup via stable ID |
|
||||
| `service_unhealthy` / other | stability-agent | `redeploy` → dispatched to the node's deploy-runner (`jobs/deploy-runner/`) | dedup via stable ID |
|
||||
| `disk_pressure` (high) | stability-agent | `disk_cleanup` | dedup via stable ID |
|
||||
| `ha_websocket_dead` | ha-diag-agent | `container_restart` (homeassistant) | 30 min after completion |
|
||||
| `ha_websocket_recovered` | ha-diag-agent | cancels matching restart | — |
|
||||
|
|
|
|||
|
|
@ -4,6 +4,60 @@ Centralny tracker tech-długu i znanych usterek. Wpisy ze sesji — dodawaj z da
|
|||
|
||||
---
|
||||
|
||||
## deploy-runner: instalacja na węzłach + E2E redeployu (2026-08-03)
|
||||
|
||||
**Data**: 2026-08-03
|
||||
**Źródło**: sesja `task/redeploy-fix` — recon D14/D15 + OPEN QUESTION 4 (nieopróżniona
|
||||
kolejka akcji).
|
||||
**Było**: `redeploy` nie mógł się wykonać nigdy. Executor odpalał
|
||||
`scripts/deploy/deploy-node.sh <node> <service>` **wewnątrz swojego kontenera** —
|
||||
skrypt ignoruje oba argumenty i wymaga repo w `${HOME}/homelab-codex-ws`
|
||||
(w kontenerze `HOME=/home/homelab`, katalog nie istnieje) → `exit 1` w 18. linii.
|
||||
Za tym stały jeszcze trzy blokady: brak `git`, brak klienta `docker` w obrazie oraz
|
||||
— gdyby przeszedł — deploy **całego zestawu usług hosta executora**, nie węzła
|
||||
z akcji. Stąd `healthcheck_failed` przekierowany 2026-07-29 na `container_restart`
|
||||
i 18 pending / 0 completed.
|
||||
**Zrobione w repo (ta sesja)**: `scripts/deploy/deploy-service.sh` (deploy jednej
|
||||
usługi, wspólny z `deploy-node.sh` — ta sama inwokacja compose, więc ta sama nazwa
|
||||
projektu), `jobs/deploy-runner/` (systemd na hoście: rsync-pull akcji, walidacja,
|
||||
deploy, `action_result` z powrotem), executor dispatchuje `redeploy` do
|
||||
`actions/deploy/<node>/` i rozlicza je jak `container_restart`
|
||||
(`REDEPLOY_TIMEOUT_SECS=900`). 248 testów zielonych.
|
||||
**Do zrobienia (runtime, wymaga operatora)**:
|
||||
1. Deploy control-plane na VPS (nowy executor + `../..:/repo:ro`).
|
||||
2. Instalacja `jobs/deploy-runner/` na vps, piha, solaria — patrz README
|
||||
(„Install (per node)"). Na VPS `VPS_EVENTS_HOST` musi zostać puste.
|
||||
3. E2E na benignej usłudze na PIHA (wzorzec `test-e2e-b` z 2026-07-23), potem
|
||||
opróżnienie kolejki 18 pending — w tym `redeploy-vps-gokapi`.
|
||||
4. SOLARIA: `group_add: "996"` dla node-agenta wciąż niewdrożony (recon 641-649) —
|
||||
dopóki nie wejdzie, `container_restart` tam nie działa (redeploy działa, bo nie
|
||||
idzie przez node-agenta).
|
||||
|
||||
---
|
||||
|
||||
## disk_cleanup w executorze jest zepsuty tak samo jak był redeploy (2026-08-03)
|
||||
|
||||
**Data**: 2026-08-03
|
||||
**Źródło**: sesja `task/redeploy-fix` (znalezione przy okazji, poza zakresem zadania).
|
||||
**Problem**: `executor._execute_disk_cleanup()` odpala `ssh oskar@<node> …`, a obraz
|
||||
control-plane (`python:3.11-slim` + `pip install pyyaml`) **nie ma klienta ssh** —
|
||||
`subprocess.run(["ssh", …])` leci `FileNotFoundError`, akcja ląduje w `failed/`.
|
||||
To ta sama klasa błędu co redeploy i sprzeczne z decyzją „bez SSH w executorze".
|
||||
**Fix**: przenieść `disk_cleanup` na model dispatch (node-agent albo deploy-runner —
|
||||
runner ma już hosta i uprawnienia) albo usunąć typ akcji. Do decyzji przy okazji
|
||||
opróżniania kolejki.
|
||||
|
||||
---
|
||||
|
||||
## `scripts/deploy/deploy-host.sh` to pusty plik (2026-08-03)
|
||||
|
||||
**Data**: 2026-08-03
|
||||
**Źródło**: sesja `task/redeploy-fix`.
|
||||
**Problem**: 0-bajtowy, wykonywalny stub w `scripts/deploy/` — wygląda jak
|
||||
entrypoint, nie robi nic. Kandydat do skasowania (jak `deploy-role.sh` w etapie 0).
|
||||
|
||||
---
|
||||
|
||||
## Cutover HA "ken": kontener piha to legacy, prawdziwy dom to RPi4/HAOS (2026-07-22)
|
||||
|
||||
**Data**: 2026-07-22
|
||||
|
|
|
|||
111
jobs/deploy-runner/README.md
Normal file
111
jobs/deploy-runner/README.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
# deploy-runner — host-side executor for `redeploy` actions
|
||||
|
||||
Runs on each node that should be able to execute an operator-approved
|
||||
`redeploy`. It is the missing half of the self-healing loop: the control-plane
|
||||
executor can generate and dispatch redeploys, but nothing on the node could ever
|
||||
perform one.
|
||||
|
||||
## What was broken
|
||||
|
||||
The executor ran `scripts/deploy/deploy-node.sh <node> <service>` **inside its own
|
||||
container**. That script ignores both arguments, and expects a repo at
|
||||
`${HOME}/homelab-codex-ws` — inside the container `HOME=/home/homelab`, so it
|
||||
exited at line 18 with `Error: Repository not found`. Behind that failure sat
|
||||
three more: no `git`, no `docker` CLI in the image, and — had it ever got that
|
||||
far — it would have deployed the **executor host's** entire service set, not the
|
||||
action's target node/service. Every `healthcheck_failed → redeploy` dead-ended
|
||||
(recon `docs/architecture/RECON-multiagent-2026-07-27.md`, D14/D15).
|
||||
|
||||
## How it works now
|
||||
|
||||
```
|
||||
supervisor → actions/pending/<id>.json (unchanged)
|
||||
operator → actions/approved/<id>.json (unchanged, HITL)
|
||||
executor (vps) → actions/deploy/<node>/<id>.json ← dispatch only, no execution
|
||||
deploy-runner ← rsync-pull (node initiates; VPS never connects to a node)
|
||||
→ scripts/deploy/deploy-service.sh --force-recreate
|
||||
→ events/<node>/evt-…-action_result-….json → rsync-push
|
||||
executor → completed / failed (via _reconcile_running_actions)
|
||||
```
|
||||
|
||||
Design points, and why:
|
||||
|
||||
- **Host-level, not a container.** Compose resolves relative bind mounts and the
|
||||
project name against the filesystem of whatever runs it. In a container those
|
||||
resolve to container paths that the daemon then interprets as host paths — a
|
||||
silent way to produce broken mounts, or to land in a different Compose project
|
||||
where `--remove-orphans` deletes the running stack. On the host it behaves
|
||||
exactly like a human `deploy.sh`.
|
||||
- **Independent of node-agent** (own rsync pull, own result push) so it can
|
||||
redeploy `node-agent` itself — the solaria case, where node-agent has been
|
||||
docker-blind for weeks.
|
||||
- **Separate inbox** from `actions/dispatch/<node>/`: node-agent deletes and
|
||||
failure-reports every file in its own inbox that is not `container_restart`.
|
||||
- **No `git pull`.** A redeploy reconciles the node to the checkout it already
|
||||
has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
|
||||
- **No `--build`, no `--remove-orphans`, always `--force-recreate`.** Plain
|
||||
`up -d` is a no-op when config is unchanged — exactly the unhealthy-container
|
||||
case; building on the 4 GiB VPS mid-incident is an OOM risk.
|
||||
|
||||
Every action is validated before anything runs (`action.py`): type must be
|
||||
`redeploy`, `node` must match this node, the service name must be a plain
|
||||
kebab-case name, the service must be listed in `hosts/<node>/services.yaml`, and
|
||||
its compose file must exist. Nothing from the action payload is ever executed —
|
||||
only a validated service *name* reaches the deploy script. Actions are
|
||||
idempotent (marker in `state/processed-deploy-actions/`), single-instance
|
||||
(`flock`), and time-boxed (`DEPLOY_TIMEOUT_SECS`, default 600 s, below the
|
||||
executor's `REDEPLOY_TIMEOUT_SECS` of 900 s so the node reports before the
|
||||
control plane gives up).
|
||||
|
||||
## Install (per node)
|
||||
|
||||
Not deployed by `deploy.sh` — it is a host-level systemd unit, like
|
||||
`jobs/documents-ingest/`. On the target node:
|
||||
|
||||
```bash
|
||||
# 1. config
|
||||
sudo mkdir -p /opt/homelab/config/deploy-runner
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/env.example \
|
||||
/opt/homelab/config/deploy-runner/env
|
||||
sudo chown oskar:oskar /opt/homelab/config/deploy-runner/env
|
||||
sudoedit /opt/homelab/config/deploy-runner/env # set NODE_NAME; clear VPS_EVENTS_HOST on the VPS
|
||||
|
||||
# 2. units
|
||||
sudo cp ~/homelab-codex-ws/jobs/deploy-runner/systemd/homelab-deploy-runner.{service,timer} \
|
||||
/etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now homelab-deploy-runner.timer
|
||||
|
||||
# 3. verify (no action queued yet → one clean, empty run)
|
||||
sudo systemctl start homelab-deploy-runner.service
|
||||
journalctl -u homelab-deploy-runner.service -n 30 --no-pager
|
||||
```
|
||||
|
||||
Requirements on the node: repo checkout at `REPO_PATH`, the service user in the
|
||||
`docker` group, `python3` + PyYAML, `rsync`, and (remote nodes only) an ssh key
|
||||
that reaches the VPS — the same one node-agent already uses for event shipping.
|
||||
|
||||
On the **VPS** leave `VPS_EVENTS_HOST` empty: the executor writes into the same
|
||||
`/opt/homelab` mount, so there is nothing to pull and nothing to push.
|
||||
|
||||
## Operating it
|
||||
|
||||
- Dispatched but not yet collected: `ls /opt/homelab/actions/deploy/<node>/` on the VPS.
|
||||
- Per-action deploy output: `/opt/homelab/logs/deploy-runner/<action_id>-<ts>.log` on the node.
|
||||
- Runner activity: `journalctl -u homelab-deploy-runner.service`.
|
||||
- A redeploy for a service with its own `deploy-local.sh` (control-plane) is
|
||||
reported as **failed** with an explanatory message — those need an operator
|
||||
deploy, by design.
|
||||
- To let an already-processed action run again, remove its marker:
|
||||
`rm /opt/homelab/state/processed-deploy-actions/<action_id>.done`.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/` — validation and rejection cases, the event format contract against the
|
||||
executor's real parser, the `docker compose` argv contract of
|
||||
`scripts/deploy/deploy-service.sh`, and an end-to-end run of the runner itself
|
||||
with a stubbed `docker`.
|
||||
|
||||
```bash
|
||||
python3 -m pytest jobs/deploy-runner/tests -q
|
||||
```
|
||||
221
jobs/deploy-runner/action.py
Executable file
221
jobs/deploy-runner/action.py
Executable file
|
|
@ -0,0 +1,221 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Validation and result-reporting helper for the host-side deploy runner.
|
||||
|
||||
The runner (deploy-runner.sh) is bash — it does flock, rsync and timeout, which
|
||||
bash does well. Everything that needs to parse untrusted JSON, read
|
||||
hosts/<node>/services.yaml or write an event in the exact node-agent format
|
||||
lives here, where it is testable (jobs/deploy-runner/tests/).
|
||||
|
||||
Two subcommands, both designed to be consumed from bash:
|
||||
|
||||
validate --file <action.json> --node <name> --repo <path>
|
||||
Prints shell assignments for `eval`:
|
||||
OK='true'|'false'
|
||||
ACTION_ID='...' (best effort — set even when OK=false, so the
|
||||
runner can still report a result for it)
|
||||
SERVICE='...'
|
||||
ERROR='...'
|
||||
Always exits 0 unless the arguments themselves are unusable; the verdict
|
||||
is in OK. Nothing from the action file is ever executed or interpolated
|
||||
into a command — only a validated service NAME is handed back.
|
||||
|
||||
emit-result --events-dir <dir> --node <n> --action-id <id> --service <svc>
|
||||
--success true|false [--error <msg>]
|
||||
Writes evt-<node>-<ts>-action_result-<slug>.json into the node's event
|
||||
directory, byte-compatible with node_agent.emit_event() so the control
|
||||
plane executor's _find_action_result() picks it up unchanged. Prints the
|
||||
path it wrote.
|
||||
|
||||
Validation rules (all must hold):
|
||||
1. the file is JSON with type == "redeploy"
|
||||
2. action["node"] == this node — defense in depth; the inbox is already
|
||||
node-scoped, but a mis-delivered file must never deploy here
|
||||
3. action_id and service match strict name patterns (no path traversal)
|
||||
4. the service is listed in hosts/<node>/services.yaml — the repo's desired
|
||||
state is the authority on what this node may run, so a rogue or stale
|
||||
dispatch cannot pull an arbitrary stack onto the node
|
||||
5. services/<service>/docker-compose.yml exists in the repo
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shlex
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
ACTION_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
|
||||
SERVICE_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$")
|
||||
ACTION_TYPE = "redeploy"
|
||||
|
||||
|
||||
class Invalid(Exception):
|
||||
"""Validation failure carrying a human-readable reason."""
|
||||
|
||||
|
||||
def _load_desired_services(repo: Path, node: str) -> set[str]:
|
||||
"""Service names listed in hosts/<node>/services.yaml."""
|
||||
try:
|
||||
import yaml
|
||||
except ImportError as exc: # pragma: no cover - environment problem
|
||||
raise Invalid(f"PyYAML not available on this node: {exc}") from exc
|
||||
|
||||
path = repo / "hosts" / node / "services.yaml"
|
||||
if not path.is_file():
|
||||
raise Invalid(f"no desired-state manifest for node '{node}' ({path})")
|
||||
try:
|
||||
data = yaml.safe_load(path.read_text()) or {}
|
||||
except Exception as exc:
|
||||
raise Invalid(f"unreadable {path}: {exc}") from exc
|
||||
|
||||
services = data.get("services") or {}
|
||||
if isinstance(services, dict):
|
||||
return {str(k) for k in services.keys()}
|
||||
if isinstance(services, list):
|
||||
return {str(s) for s in services}
|
||||
raise Invalid(f"unexpected 'services' shape in {path}: {type(services).__name__}")
|
||||
|
||||
|
||||
def validate(action_file: Path, node: str, repo: Path) -> tuple[str, str]:
|
||||
"""Return (action_id, service) for a valid redeploy action, else raise Invalid.
|
||||
|
||||
action_id is also returned inside the Invalid path via the caller's
|
||||
best-effort parse, so a rejected action can still be reported back.
|
||||
"""
|
||||
try:
|
||||
action = json.loads(action_file.read_text())
|
||||
except Exception as exc:
|
||||
raise Invalid(f"unparseable action file: {exc}") from exc
|
||||
if not isinstance(action, dict):
|
||||
raise Invalid("action file is not a JSON object")
|
||||
|
||||
action_id = str(action.get("action_id") or action_file.stem)
|
||||
if not ACTION_ID_RE.match(action_id):
|
||||
raise Invalid(f"invalid action_id: {action_id!r}")
|
||||
|
||||
action_type = action.get("type")
|
||||
if action_type != ACTION_TYPE:
|
||||
raise Invalid(
|
||||
f"action type {action_type!r} is not executable by the deploy runner "
|
||||
f"(only {ACTION_TYPE!r})"
|
||||
)
|
||||
|
||||
target_node = action.get("node")
|
||||
if target_node != node:
|
||||
raise Invalid(f"action addressed to node {target_node!r}, not {node!r} — refused")
|
||||
|
||||
service = action.get("service")
|
||||
if not isinstance(service, str) or not SERVICE_RE.match(service):
|
||||
raise Invalid(f"invalid service name: {service!r}")
|
||||
|
||||
desired = _load_desired_services(repo, node)
|
||||
if service not in desired:
|
||||
raise Invalid(
|
||||
f"service '{service}' is not in hosts/{node}/services.yaml — "
|
||||
f"refusing to deploy a service this node is not declared to run"
|
||||
)
|
||||
|
||||
compose = repo / "services" / service / "docker-compose.yml"
|
||||
if not compose.is_file():
|
||||
raise Invalid(f"no compose file in repo for '{service}' ({compose})")
|
||||
|
||||
return action_id, service
|
||||
|
||||
|
||||
def _best_effort_action_id(action_file: Path) -> str:
|
||||
try:
|
||||
data = json.loads(action_file.read_text())
|
||||
candidate = str(data.get("action_id") or action_file.stem)
|
||||
except Exception:
|
||||
candidate = action_file.stem
|
||||
return candidate if ACTION_ID_RE.match(candidate) else ""
|
||||
|
||||
|
||||
def _emit_assignments(ok: bool, action_id: str, service: str, error: str) -> str:
|
||||
return "\n".join(
|
||||
f"{key}={shlex.quote(value)}"
|
||||
for key, value in (
|
||||
("OK", "true" if ok else "false"),
|
||||
("ACTION_ID", action_id),
|
||||
("SERVICE", service),
|
||||
("ERROR", error),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def emit_result(events_dir: Path, node: str, action_id: str, service: str,
|
||||
success: bool, error: str) -> Path:
|
||||
"""Write an action_result event in node_agent.emit_event()'s exact format.
|
||||
|
||||
The control-plane executor globs evt-*-action_result-*.json in the node's
|
||||
event directory and matches payload.action_id, comparing the unix timestamp
|
||||
embedded in the filename against the action's started_at — so both the name
|
||||
and the payload shape matter here (see executor._find_action_result).
|
||||
"""
|
||||
ts = int(time.time())
|
||||
slug = re.sub(r"[^a-z0-9]", "-", (service or "node").lower())[:32].strip("-")
|
||||
event_id = f"evt-{node}-{ts}-action_result-{slug}"
|
||||
event = {
|
||||
"id": event_id,
|
||||
"timestamp": ts,
|
||||
"date": datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z"),
|
||||
"type": "action_result",
|
||||
"severity": "info" if success else "high",
|
||||
"node": node,
|
||||
"service": service or "",
|
||||
"message": f"Action {action_id} {'succeeded' if success else 'failed'}",
|
||||
"payload": {
|
||||
"action_id": action_id,
|
||||
"success": success,
|
||||
"error": error,
|
||||
"node": node,
|
||||
"source": "deploy-runner",
|
||||
},
|
||||
}
|
||||
events_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = events_dir / f"{event_id}.json"
|
||||
path.write_text(json.dumps(event, indent=2))
|
||||
return path
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_val = sub.add_parser("validate")
|
||||
p_val.add_argument("--file", required=True, type=Path)
|
||||
p_val.add_argument("--node", required=True)
|
||||
p_val.add_argument("--repo", required=True, type=Path)
|
||||
|
||||
p_emit = sub.add_parser("emit-result")
|
||||
p_emit.add_argument("--events-dir", required=True, type=Path)
|
||||
p_emit.add_argument("--node", required=True)
|
||||
p_emit.add_argument("--action-id", required=True)
|
||||
p_emit.add_argument("--service", default="")
|
||||
p_emit.add_argument("--success", required=True, choices=["true", "false"])
|
||||
p_emit.add_argument("--error", default="")
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if args.command == "validate":
|
||||
try:
|
||||
action_id, service = validate(args.file, args.node, args.repo)
|
||||
except Invalid as exc:
|
||||
print(_emit_assignments(False, _best_effort_action_id(args.file), "", str(exc)))
|
||||
return 0
|
||||
print(_emit_assignments(True, action_id, service, ""))
|
||||
return 0
|
||||
|
||||
path = emit_result(
|
||||
args.events_dir, args.node, args.action_id, args.service,
|
||||
args.success == "true", args.error,
|
||||
)
|
||||
print(path)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
200
jobs/deploy-runner/deploy-runner.sh
Executable file
200
jobs/deploy-runner/deploy-runner.sh
Executable file
|
|
@ -0,0 +1,200 @@
|
|||
#!/usr/bin/env bash
|
||||
# jobs/deploy-runner/deploy-runner.sh — host-side executor for `redeploy` actions.
|
||||
#
|
||||
# WHY THIS EXISTS (docs/architecture/RECON-multiagent-2026-07-27.md D14/D15):
|
||||
# the control-plane executor could never carry out a redeploy — it ran
|
||||
# deploy-node.sh inside its own container, a script that ignores its arguments
|
||||
# and expects a repo at ${HOME}/homelab-codex-ws that does not exist there (nor
|
||||
# does git or the docker CLI). Every healthcheck_failed → redeploy action
|
||||
# dead-ended, which is the single biggest reason the self-healing loop had
|
||||
# 18 pending / 0 completed actions.
|
||||
#
|
||||
# The fix keeps the architecture decision from docs/backlog.md ("Remediacja
|
||||
# floty bez SSH"): the VPS never initiates a connection to a node. The executor
|
||||
# only WRITES an action file; this runner, on the target node, pulls it, runs
|
||||
# the deploy locally, and reports the outcome back through the existing event
|
||||
# pipeline.
|
||||
#
|
||||
# executor (vps) → /opt/homelab/actions/deploy/<node>/<action_id>.json
|
||||
# → this runner rsync-pulls it (--remove-source-files: collected once)
|
||||
# → validates it (jobs/deploy-runner/action.py)
|
||||
# → scripts/deploy/deploy-service.sh --force-recreate (same compose
|
||||
# invocation as a human deploy — identical project name)
|
||||
# → writes an action_result event and rsync-pushes it back to the VPS
|
||||
# → executor._reconcile_running_actions() → completed / failed
|
||||
#
|
||||
# It runs on the HOST, not in a container: compose then resolves relative bind
|
||||
# mounts and the project name exactly as it does for a human `deploy.sh`, and
|
||||
# no agent container needs a docker CLI. It is deliberately independent of
|
||||
# node-agent (own rsync, own result push) so it can redeploy node-agent itself —
|
||||
# the solaria case, where node-agent has been docker-blind for weeks.
|
||||
#
|
||||
# It NEVER runs `git pull`: a redeploy reconciles the node to the checkout it
|
||||
# already has. Shipping new code stays a human `scripts/deploy/deploy.sh` action.
|
||||
#
|
||||
# Config comes from the environment (systemd EnvironmentFile) — see env.example.
|
||||
# Install: see README.md.
|
||||
|
||||
# NOT set -e: one malformed or failing action must be reported and skipped, not
|
||||
# abort the whole run.
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
HELPER="${SCRIPT_DIR}/action.py"
|
||||
|
||||
NODE_NAME="${NODE_NAME:-}"
|
||||
REPO_PATH="${REPO_PATH:-${HOME}/homelab-codex-ws}"
|
||||
RUNTIME_PATH="${RUNTIME_PATH:-/opt/homelab}"
|
||||
VPS_EVENTS_HOST="${VPS_EVENTS_HOST:-}"
|
||||
VPS_EVENTS_USER="${VPS_EVENTS_USER:-oskar}"
|
||||
VPS_EVENTS_PATH="${VPS_EVENTS_PATH:-/opt/homelab/events}"
|
||||
VPS_DEPLOY_PATH="${VPS_DEPLOY_PATH:-/opt/homelab/actions/deploy}"
|
||||
DEPLOY_TIMEOUT_SECS="${DEPLOY_TIMEOUT_SECS:-600}"
|
||||
|
||||
INBOX="${RUNTIME_PATH}/actions/deploy/${NODE_NAME}"
|
||||
EVENTS_DIR="${RUNTIME_PATH}/events/${NODE_NAME}"
|
||||
STATE_DIR="${RUNTIME_PATH}/state"
|
||||
MARKER_DIR="${STATE_DIR}/processed-deploy-actions"
|
||||
LOG_DIR="${RUNTIME_PATH}/logs/deploy-runner"
|
||||
HOST_DIR="${REPO_PATH}/hosts/${NODE_NAME}"
|
||||
|
||||
# Same ssh settings node-agent uses for its rsync channel: no ~/.ssh/config, no
|
||||
# host-key prompts, batch mode so a missing key fails fast instead of hanging.
|
||||
SSH_CMD="ssh -F /dev/null -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o BatchMode=yes"
|
||||
if [[ -n "${SSH_KEY:-}" ]]; then
|
||||
SSH_CMD="${SSH_CMD} -i ${SSH_KEY}"
|
||||
fi
|
||||
|
||||
log() { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [deploy-runner] $*"; }
|
||||
die() { log "ERROR: $*"; exit 1; }
|
||||
|
||||
[[ -n "$NODE_NAME" ]] || die "NODE_NAME is not set (see env.example)"
|
||||
[[ -d "$REPO_PATH" ]] || die "repo not found at ${REPO_PATH} (set REPO_PATH)"
|
||||
[[ -d "$HOST_DIR" ]] || die "no hosts/${NODE_NAME}/ directory in ${REPO_PATH}"
|
||||
[[ -x "${REPO_PATH}/scripts/deploy/deploy-service.sh" ]] \
|
||||
|| die "scripts/deploy/deploy-service.sh missing or not executable in ${REPO_PATH}"
|
||||
command -v python3 >/dev/null || die "python3 not found"
|
||||
|
||||
mkdir -p "$INBOX" "$EVENTS_DIR" "$MARKER_DIR" "$LOG_DIR" || die "cannot create runtime dirs under ${RUNTIME_PATH}"
|
||||
|
||||
# Single instance: the timer fires every 60s but a deploy may take minutes.
|
||||
exec 9>"${STATE_DIR}/deploy-runner.lock" || die "cannot open lock file"
|
||||
if ! flock -n 9; then
|
||||
log "another deploy-runner run is in progress — exiting"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── transport ────────────────────────────────────────────────────────────────
|
||||
|
||||
# Remote nodes fetch their inbox from the VPS. On the VPS itself the executor
|
||||
# writes straight into the shared /opt/homelab mount, so there is nothing to
|
||||
# pull and nothing to push — the executor reads the result event in place.
|
||||
is_remote() { [[ -n "$VPS_EVENTS_HOST" ]]; }
|
||||
|
||||
pull_actions() {
|
||||
is_remote || return 0
|
||||
local rc=0
|
||||
rsync -az --remove-source-files \
|
||||
--omit-dir-times --no-perms --no-owner --no-group \
|
||||
-e "$SSH_CMD" \
|
||||
"${VPS_EVENTS_USER}@${VPS_EVENTS_HOST}:${VPS_DEPLOY_PATH}/${NODE_NAME}/" \
|
||||
"${INBOX}/" || rc=$?
|
||||
# 23/24 = "partial transfer"/"vanished source files": what an empty or
|
||||
# concurrently-drained remote inbox looks like. Not worth logging each run.
|
||||
if [[ $rc -ne 0 && $rc -ne 23 && $rc -ne 24 ]]; then
|
||||
log "WARN: dispatch pull failed (rsync rc=${rc})"
|
||||
fi
|
||||
}
|
||||
|
||||
push_event() {
|
||||
local event_file="$1"
|
||||
is_remote || return 0
|
||||
local rc=0
|
||||
rsync -az --remove-source-files \
|
||||
--omit-dir-times --no-perms --no-owner --no-group \
|
||||
-e "$SSH_CMD" \
|
||||
"$event_file" \
|
||||
"${VPS_EVENTS_USER}@${VPS_EVENTS_HOST}:${VPS_EVENTS_PATH}/${NODE_NAME}/" || rc=$?
|
||||
if [[ $rc -ne 0 && $rc -ne 23 && $rc -ne 24 ]]; then
|
||||
# Left in place on failure: node-agent's own event shipping will pick it
|
||||
# up on its next cycle, so a result is delayed, never lost.
|
||||
log "WARN: result push failed (rsync rc=${rc}) — leaving ${event_file} for node-agent to ship"
|
||||
fi
|
||||
}
|
||||
|
||||
report() {
|
||||
local action_id="$1" service="$2" success="$3" error="$4"
|
||||
local event_file
|
||||
event_file=$(python3 "$HELPER" emit-result \
|
||||
--events-dir "$EVENTS_DIR" \
|
||||
--node "$NODE_NAME" \
|
||||
--action-id "$action_id" \
|
||||
--service "$service" \
|
||||
--success "$success" \
|
||||
--error "$error") || { log "ERROR: could not write result event for ${action_id}"; return 1; }
|
||||
log "reported ${action_id}: success=${success}${error:+ error=${error}}"
|
||||
push_event "$event_file"
|
||||
}
|
||||
|
||||
# ── main ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
pull_actions
|
||||
|
||||
shopt -s nullglob
|
||||
for action_file in "${INBOX}"/*.json; do
|
||||
OK="false"; ACTION_ID=""; SERVICE=""; ERROR=""
|
||||
eval "$(python3 "$HELPER" validate --file "$action_file" --node "$NODE_NAME" --repo "$REPO_PATH")"
|
||||
|
||||
if [[ "$OK" != "true" ]]; then
|
||||
log "rejected $(basename "$action_file"): ${ERROR}"
|
||||
if [[ -n "$ACTION_ID" ]]; then
|
||||
report "$ACTION_ID" "$SERVICE" "false" "$ERROR"
|
||||
fi
|
||||
rm -f "$action_file"
|
||||
continue
|
||||
fi
|
||||
|
||||
marker="${MARKER_DIR}/${ACTION_ID}.done"
|
||||
if [[ -e "$marker" ]]; then
|
||||
log "action ${ACTION_ID} already processed — skipping (idempotency)"
|
||||
rm -f "$action_file"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "deploying ${SERVICE} for action ${ACTION_ID}"
|
||||
action_log="${LOG_DIR}/${ACTION_ID}-$(date -u +%Y%m%dT%H%M%SZ).log"
|
||||
rc=0
|
||||
timeout "$DEPLOY_TIMEOUT_SECS" \
|
||||
"${REPO_PATH}/scripts/deploy/deploy-service.sh" \
|
||||
--repo "$REPO_PATH" \
|
||||
--host-dir "$HOST_DIR" \
|
||||
--service "$SERVICE" \
|
||||
--force-recreate \
|
||||
>"$action_log" 2>&1 || rc=$?
|
||||
|
||||
case "$rc" in
|
||||
0)
|
||||
report "$ACTION_ID" "$SERVICE" "true" ""
|
||||
;;
|
||||
3)
|
||||
report "$ACTION_ID" "$SERVICE" "false" \
|
||||
"${SERVICE} owns its deploy path (services/${SERVICE}/deploy-local.sh) — needs an operator deploy, not an automated redeploy"
|
||||
;;
|
||||
124)
|
||||
report "$ACTION_ID" "$SERVICE" "false" \
|
||||
"deploy timed out after ${DEPLOY_TIMEOUT_SECS}s (log: ${action_log})"
|
||||
;;
|
||||
*)
|
||||
# Last 20 lines are the useful part of a compose failure; the full
|
||||
# output stays in the per-action log on the node.
|
||||
err=$(tail -n 20 "$action_log" 2>/dev/null | tr -d '\000' | tail -c 2000)
|
||||
report "$ACTION_ID" "$SERVICE" "false" \
|
||||
"deploy-service.sh exited ${rc} (log: ${action_log}): ${err}"
|
||||
;;
|
||||
esac
|
||||
|
||||
touch "$marker" || log "WARN: could not write idempotency marker ${marker}"
|
||||
rm -f "$action_file"
|
||||
done
|
||||
|
||||
exit 0
|
||||
29
jobs/deploy-runner/env.example
Normal file
29
jobs/deploy-runner/env.example
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# deploy-runner runtime config — copy to /opt/homelab/config/deploy-runner/env
|
||||
# on each node and fill in. Read by homelab-deploy-runner.service
|
||||
# (EnvironmentFile), so: KEY=value, no quotes, no shell expansion.
|
||||
|
||||
# Canonical topology node name — MUST match hosts/<node>/ and the `node` field
|
||||
# in dispatched actions. Not the OS hostname.
|
||||
NODE_NAME=piha
|
||||
|
||||
# Repo checkout on this node (the runner never pulls it; deploys what is there).
|
||||
REPO_PATH=/home/oskar/homelab-codex-ws
|
||||
|
||||
RUNTIME_PATH=/opt/homelab
|
||||
|
||||
# VPS Tailscale IP — where the control plane writes dispatched actions and
|
||||
# expects result events. LEAVE EMPTY ON THE VPS ITSELF: there the executor
|
||||
# writes into the same /opt/homelab mount and no rsync hop is needed.
|
||||
VPS_EVENTS_HOST=100.95.58.48
|
||||
VPS_EVENTS_USER=oskar
|
||||
VPS_EVENTS_PATH=/opt/homelab/events
|
||||
VPS_DEPLOY_PATH=/opt/homelab/actions/deploy
|
||||
|
||||
# Per-action wall-clock ceiling. Must stay below the executor's
|
||||
# REDEPLOY_TIMEOUT_SECS (default 900) so a stuck deploy is reported by the node
|
||||
# rather than silently timed out by the control plane.
|
||||
DEPLOY_TIMEOUT_SECS=600
|
||||
|
||||
# Optional: explicit ssh key for the rsync channel. Omit when the service user's
|
||||
# default key (~/.ssh/id_*) already reaches the VPS — same key node-agent uses.
|
||||
# SSH_KEY=/home/oskar/.ssh/id_ed25519
|
||||
25
jobs/deploy-runner/systemd/homelab-deploy-runner.service
Normal file
25
jobs/deploy-runner/systemd/homelab-deploy-runner.service
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# homelab-deploy-runner.service — host-side executor for control-plane `redeploy`
|
||||
# actions (docs/architecture/RECON-multiagent-2026-07-27.md D14/D15).
|
||||
#
|
||||
# Host-level and NOT a container, deliberately: it runs `docker compose` the way
|
||||
# a human deploy does, so relative bind mounts and the compose project name
|
||||
# resolve identically. A containerised runner would resolve them against its own
|
||||
# filesystem and hand the daemon host paths that do not exist.
|
||||
#
|
||||
# Install: see ../README.md (copy this + the timer to /etc/systemd/system/,
|
||||
# create /opt/homelab/config/deploy-runner/env from env.example, daemon-reload,
|
||||
# enable the TIMER — not this unit).
|
||||
[Unit]
|
||||
Description=Homelab deploy runner (executes approved redeploy actions on this node)
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=oskar
|
||||
# Runtime config: NODE_NAME, REPO_PATH, VPS_EVENTS_HOST, ... (see env.example).
|
||||
EnvironmentFile=/opt/homelab/config/deploy-runner/env
|
||||
ExecStart=/home/oskar/homelab-codex-ws/jobs/deploy-runner/deploy-runner.sh
|
||||
# Slightly above the runner's own per-action DEPLOY_TIMEOUT_SECS (default 600)
|
||||
# so the script always gets to report a result before systemd intervenes.
|
||||
TimeoutStartSec=900
|
||||
16
jobs/deploy-runner/systemd/homelab-deploy-runner.timer
Normal file
16
jobs/deploy-runner/systemd/homelab-deploy-runner.timer
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# homelab-deploy-runner.timer — poll for dispatched redeploy actions.
|
||||
#
|
||||
# 60s matches node-agent's CHECK_INTERVAL, so remediation latency is the same on
|
||||
# both paths and well inside the executor's REDEPLOY_TIMEOUT_SECS (900).
|
||||
# Overlapping runs are prevented by the runner's own flock, not by the timer.
|
||||
[Unit]
|
||||
Description=Poll for dispatched redeploy actions every minute
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=60s
|
||||
AccuracySec=15s
|
||||
Unit=homelab-deploy-runner.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
238
jobs/deploy-runner/tests/test_action.py
Normal file
238
jobs/deploy-runner/tests/test_action.py
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
"""Tests for the deploy runner's validation / result-reporting helper.
|
||||
|
||||
The validator is the only gate between a JSON file that arrived over the
|
||||
network and `docker compose up` running on the node, so its rejection cases
|
||||
matter more than its happy path. The emit-result tests pin the event format
|
||||
against the control-plane executor's actual parser — if either side drifts,
|
||||
redeploy actions would hang in running/ until they time out.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shlex
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
_JOB_DIR = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(_JOB_DIR))
|
||||
|
||||
import action as action_mod # noqa: E402
|
||||
from action import Invalid, emit_result, validate # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(tmp_path: Path) -> Path:
|
||||
"""Minimal repo: node piha declares vikunja, which has a compose file."""
|
||||
repo = tmp_path / "repo"
|
||||
(repo / "hosts" / "piha").mkdir(parents=True)
|
||||
(repo / "hosts" / "piha" / "services.yaml").write_text(
|
||||
"host: piha\nservices:\n vikunja:\n role: task-tracker\n node-agent:\n role: monitor\n"
|
||||
)
|
||||
(repo / "services" / "vikunja").mkdir(parents=True)
|
||||
(repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n")
|
||||
(repo / "services" / "gokapi").mkdir(parents=True)
|
||||
(repo / "services" / "gokapi" / "docker-compose.yml").write_text("services: {}\n")
|
||||
return repo
|
||||
|
||||
|
||||
def _action_file(tmp_path: Path, **overrides) -> Path:
|
||||
action = {
|
||||
"action_id": "redeploy-piha-vikunja",
|
||||
"type": "redeploy",
|
||||
"node": "piha",
|
||||
"service": "vikunja",
|
||||
}
|
||||
action.update(overrides)
|
||||
path = tmp_path / "redeploy-piha-vikunja.json"
|
||||
path.write_text(json.dumps(action))
|
||||
return path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# validate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_valid_action_returns_id_and_service(tmp_path, repo):
|
||||
action_id, service = validate(_action_file(tmp_path), "piha", repo)
|
||||
assert action_id == "redeploy-piha-vikunja"
|
||||
assert service == "vikunja"
|
||||
|
||||
|
||||
def test_action_for_another_node_is_refused(tmp_path, repo):
|
||||
path = _action_file(tmp_path, node="solaria")
|
||||
with pytest.raises(Invalid, match="addressed to node"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
def test_non_redeploy_type_is_refused(tmp_path, repo):
|
||||
"""The runner executes redeploys only — container_restart stays node-agent's
|
||||
job, and anything else must never reach a shell."""
|
||||
path = _action_file(tmp_path, type="container_restart")
|
||||
with pytest.raises(Invalid, match="not executable by the deploy runner"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
def test_service_not_in_desired_state_is_refused(tmp_path, repo):
|
||||
"""gokapi has a compose file in the repo but is not in hosts/piha —
|
||||
desired state is the authority on what this node may run."""
|
||||
path = _action_file(tmp_path, service="gokapi")
|
||||
with pytest.raises(Invalid, match="not in hosts/piha/services.yaml"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["../../etc/passwd", "/absolute", "Vikunja", "", "a b", "x" * 65])
|
||||
def test_malformed_service_names_are_refused(tmp_path, repo, bad):
|
||||
path = _action_file(tmp_path, service=bad)
|
||||
with pytest.raises(Invalid, match="invalid service name"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
def test_service_without_compose_file_is_refused(tmp_path, repo):
|
||||
(repo / "services" / "vikunja" / "docker-compose.yml").unlink()
|
||||
with pytest.raises(Invalid, match="no compose file"):
|
||||
validate(_action_file(tmp_path), "piha", repo)
|
||||
|
||||
|
||||
def test_missing_manifest_for_own_node_is_refused(tmp_path, repo):
|
||||
(repo / "hosts" / "piha" / "services.yaml").unlink()
|
||||
with pytest.raises(Invalid, match="no desired-state manifest"):
|
||||
validate(_action_file(tmp_path), "piha", repo)
|
||||
|
||||
|
||||
def test_services_as_list_is_supported(tmp_path, repo):
|
||||
(repo / "hosts" / "piha" / "services.yaml").write_text("host: piha\nservices:\n - vikunja\n")
|
||||
_, service = validate(_action_file(tmp_path), "piha", repo)
|
||||
assert service == "vikunja"
|
||||
|
||||
|
||||
def test_unparseable_file_is_refused(tmp_path, repo):
|
||||
path = tmp_path / "broken.json"
|
||||
path.write_text("{not json")
|
||||
with pytest.raises(Invalid, match="unparseable"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
def test_invalid_action_id_is_refused(tmp_path, repo):
|
||||
path = _action_file(tmp_path, action_id="../escape")
|
||||
with pytest.raises(Invalid, match="invalid action_id"):
|
||||
validate(path, "piha", repo)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# validate via the CLI (shell-assignment contract used by deploy-runner.sh)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _eval_cli(argv) -> dict:
|
||||
"""Parse the KEY='value' lines the bash runner evaluates."""
|
||||
import io
|
||||
import contextlib
|
||||
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = action_mod.main(argv)
|
||||
assert rc == 0
|
||||
out = {}
|
||||
for line in buf.getvalue().splitlines():
|
||||
key, _, raw = line.partition("=")
|
||||
out[key] = shlex.split(raw)[0] if raw and raw != "''" else ""
|
||||
return out
|
||||
|
||||
|
||||
def test_cli_valid_action_emits_ok_true(tmp_path, repo):
|
||||
path = _action_file(tmp_path)
|
||||
result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)])
|
||||
assert result["OK"] == "true"
|
||||
assert result["SERVICE"] == "vikunja"
|
||||
assert result["ACTION_ID"] == "redeploy-piha-vikunja"
|
||||
assert result["ERROR"] == ""
|
||||
|
||||
|
||||
def test_cli_rejected_action_still_reports_action_id(tmp_path, repo):
|
||||
"""So the runner can report a failure result instead of silently dropping
|
||||
the action — a dropped action would sit in running/ until it times out."""
|
||||
path = _action_file(tmp_path, service="gokapi")
|
||||
result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)])
|
||||
assert result["OK"] == "false"
|
||||
assert result["ACTION_ID"] == "redeploy-piha-vikunja"
|
||||
assert "not in hosts/piha/services.yaml" in result["ERROR"]
|
||||
|
||||
|
||||
def test_cli_quotes_hostile_error_content(tmp_path, repo):
|
||||
"""A crafted service name must not be able to inject shell syntax through
|
||||
the eval'd ERROR assignment."""
|
||||
path = _action_file(tmp_path, service="'; touch /tmp/pwned; '")
|
||||
result = _eval_cli(["validate", "--file", str(path), "--node", "piha", "--repo", str(repo)])
|
||||
assert result["OK"] == "false"
|
||||
assert "touch /tmp/pwned" in result["ERROR"] # inert data, not a command
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# emit-result — format compatibility with the control-plane executor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_emit_result_shape(tmp_path):
|
||||
events = tmp_path / "events" / "piha"
|
||||
path = emit_result(events, "piha", "redeploy-piha-vikunja", "vikunja", True, "")
|
||||
event = json.loads(path.read_text())
|
||||
assert event["type"] == "action_result"
|
||||
assert event["node"] == "piha"
|
||||
assert event["severity"] == "info"
|
||||
assert event["payload"]["action_id"] == "redeploy-piha-vikunja"
|
||||
assert event["payload"]["success"] is True
|
||||
assert event["payload"]["source"] == "deploy-runner"
|
||||
|
||||
|
||||
def test_emit_result_failure_carries_error(tmp_path):
|
||||
events = tmp_path / "events" / "piha"
|
||||
path = emit_result(events, "piha", "act-1", "vikunja", False, "compose exited 1")
|
||||
event = json.loads(path.read_text())
|
||||
assert event["severity"] == "high"
|
||||
assert event["payload"]["success"] is False
|
||||
assert event["payload"]["error"] == "compose exited 1"
|
||||
|
||||
|
||||
def test_executor_finds_the_emitted_result(tmp_path, monkeypatch):
|
||||
"""The real consumer: control-plane executor._find_action_result(). This is
|
||||
the contract that makes a redeploy resolve instead of timing out."""
|
||||
executor_src = _JOB_DIR.parents[1] / "services" / "control-plane" / "src"
|
||||
sys.path.insert(0, str(executor_src))
|
||||
import executor as executor_mod
|
||||
|
||||
events_root = tmp_path / "events"
|
||||
monkeypatch.setattr(executor_mod, "EVENTS_DIR", events_root)
|
||||
monkeypatch.setattr(executor_mod, "ACTIONS_DIR", tmp_path / "actions")
|
||||
monkeypatch.setattr(executor_mod, "DISPATCH_DIR", tmp_path / "actions" / "dispatch")
|
||||
monkeypatch.setattr(executor_mod, "DEPLOY_DISPATCH_DIR", tmp_path / "actions" / "deploy")
|
||||
|
||||
started_at = time.time()
|
||||
emit_result(events_root / "piha", "piha", "redeploy-piha-vikunja", "vikunja", True, "")
|
||||
|
||||
payload = executor_mod.Executor()._find_action_result(
|
||||
"piha", "redeploy-piha-vikunja", started_at
|
||||
)
|
||||
assert payload is not None
|
||||
assert payload["success"] is True
|
||||
|
||||
|
||||
def test_executor_ignores_result_older_than_the_action(tmp_path, monkeypatch):
|
||||
"""Action IDs are deterministic and repeat, so a result from a previous run
|
||||
must not resolve the current one."""
|
||||
executor_src = _JOB_DIR.parents[1] / "services" / "control-plane" / "src"
|
||||
sys.path.insert(0, str(executor_src))
|
||||
import executor as executor_mod
|
||||
|
||||
events_root = tmp_path / "events"
|
||||
monkeypatch.setattr(executor_mod, "EVENTS_DIR", events_root)
|
||||
monkeypatch.setattr(executor_mod, "ACTIONS_DIR", tmp_path / "actions")
|
||||
monkeypatch.setattr(executor_mod, "DISPATCH_DIR", tmp_path / "actions" / "dispatch")
|
||||
monkeypatch.setattr(executor_mod, "DEPLOY_DISPATCH_DIR", tmp_path / "actions" / "deploy")
|
||||
|
||||
emit_result(events_root / "piha", "piha", "redeploy-piha-vikunja", "vikunja", True, "")
|
||||
started_at = time.time() + 60 # action started after the event was written
|
||||
|
||||
assert executor_mod.Executor()._find_action_result(
|
||||
"piha", "redeploy-piha-vikunja", started_at
|
||||
) is None
|
||||
150
jobs/deploy-runner/tests/test_deploy_service.py
Normal file
150
jobs/deploy-runner/tests/test_deploy_service.py
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
"""Contract tests for scripts/deploy/deploy-service.sh.
|
||||
|
||||
Why these live here: the deploy runner and the human whole-node deploy both go
|
||||
through this script, and what must stay identical between them is the argv it
|
||||
hands to `docker compose`. docker compose derives the project name from the
|
||||
directory of the FIRST -f file — a caller that assembles those arguments
|
||||
differently lands in a different project, where `--remove-orphans` tears down
|
||||
the running stack (the 2026-06-25 control-plane wipe on VPS).
|
||||
|
||||
Each test runs the real script with a stub `docker` on PATH that records its
|
||||
argv, so nothing is executed against a daemon.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO = Path(__file__).resolve().parents[3]
|
||||
SCRIPT = REPO / "scripts" / "deploy" / "deploy-service.sh"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def env(tmp_path: Path):
|
||||
"""A fake repo + host dir + a `docker` stub that logs its arguments."""
|
||||
repo = tmp_path / "repo"
|
||||
(repo / "services" / "vikunja").mkdir(parents=True)
|
||||
(repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n")
|
||||
host_dir = repo / "hosts" / "piha"
|
||||
host_dir.mkdir(parents=True)
|
||||
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
argv_log = tmp_path / "docker-argv.txt"
|
||||
stub = bindir / "docker"
|
||||
stub.write_text(f'#!/usr/bin/env bash\nprintf "%s\\n" "$@" > {argv_log}\nexit 0\n')
|
||||
stub.chmod(0o755)
|
||||
|
||||
return {"repo": repo, "host_dir": host_dir, "bindir": bindir, "argv_log": argv_log}
|
||||
|
||||
|
||||
def run(env, *args, service="vikunja"):
|
||||
proc_env = dict(os.environ, PATH=f"{env['bindir']}:{os.environ['PATH']}")
|
||||
return subprocess.run(
|
||||
[str(SCRIPT), "--repo", str(env["repo"]), "--host-dir", str(env["host_dir"]),
|
||||
"--service", service, *args],
|
||||
capture_output=True, text=True, env=proc_env,
|
||||
)
|
||||
|
||||
|
||||
def argv(env) -> list[str]:
|
||||
return env["argv_log"].read_text().splitlines()
|
||||
|
||||
|
||||
def test_base_invocation(env):
|
||||
result = run(env)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert argv(env) == [
|
||||
"compose", "-f", str(env["repo"] / "services" / "vikunja" / "docker-compose.yml"),
|
||||
"up", "-d",
|
||||
]
|
||||
|
||||
|
||||
def test_project_name_invariant(env):
|
||||
"""No -p and no --project-directory: the project name must come from the
|
||||
compose file's own directory, exactly as it does for a human deploy."""
|
||||
run(env, "--force-recreate")
|
||||
assert "-p" not in argv(env)
|
||||
assert "--project-directory" not in argv(env)
|
||||
assert argv(env)[1] == "-f" # first -f is the service's own compose file
|
||||
|
||||
|
||||
def test_override_is_second_f(env):
|
||||
override = env["host_dir"] / "runtime" / "vikunja"
|
||||
override.mkdir(parents=True)
|
||||
(override / "docker-compose.override.yml").write_text("services: {}\n")
|
||||
|
||||
run(env)
|
||||
args = argv(env)
|
||||
assert args[1:5] == [
|
||||
"-f", str(env["repo"] / "services" / "vikunja" / "docker-compose.yml"),
|
||||
"-f", str(override / "docker-compose.override.yml"),
|
||||
]
|
||||
|
||||
|
||||
def test_env_file_is_passed_when_present(env):
|
||||
(env["repo"] / "services" / "vikunja" / ".env").write_text("FOO=bar\n")
|
||||
run(env)
|
||||
args = argv(env)
|
||||
assert "--env-file" in args
|
||||
assert args[args.index("--env-file") + 1] == str(env["repo"] / "services" / "vikunja" / ".env")
|
||||
|
||||
|
||||
def test_force_recreate_flag(env):
|
||||
"""The remediation path needs it: plain `up -d` is a no-op when config is
|
||||
unchanged, i.e. exactly the unhealthy-container case a redeploy targets."""
|
||||
run(env, "--force-recreate")
|
||||
assert "--force-recreate" in argv(env)
|
||||
|
||||
|
||||
def test_no_remove_orphans_by_default(env):
|
||||
"""The agent path must never be able to delete containers on a project-name
|
||||
mismatch — only the human whole-node deploy opts in."""
|
||||
run(env, "--force-recreate")
|
||||
assert "--remove-orphans" not in argv(env)
|
||||
|
||||
|
||||
def test_remove_orphans_when_requested(env):
|
||||
run(env, "--remove-orphans")
|
||||
assert "--remove-orphans" in argv(env)
|
||||
|
||||
|
||||
def test_build_if_needed_only_builds_with_dockerfile(env):
|
||||
run(env, "--build-if-needed")
|
||||
assert "--build" not in argv(env)
|
||||
|
||||
(env["repo"] / "services" / "vikunja" / "Dockerfile").write_text("FROM scratch\n")
|
||||
run(env, "--build-if-needed")
|
||||
assert "--build" in argv(env)
|
||||
|
||||
|
||||
def test_redeploy_path_never_builds(env):
|
||||
"""The runner does not pass --build-if-needed: building on the 4 GiB VPS
|
||||
mid-incident is an OOM risk, and a redeploy is a reconcile, not a code ship."""
|
||||
(env["repo"] / "services" / "vikunja" / "Dockerfile").write_text("FROM scratch\n")
|
||||
run(env, "--force-recreate")
|
||||
assert "--build" not in argv(env)
|
||||
|
||||
|
||||
def test_service_with_own_deploy_path_exits_3(env):
|
||||
(env["repo"] / "services" / "vikunja" / "deploy-local.sh").write_text("#!/bin/sh\n")
|
||||
result = run(env)
|
||||
assert result.returncode == 3
|
||||
assert not env["argv_log"].exists() # docker never invoked
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["../../etc", "/abs", "Vikunja", "a b"])
|
||||
def test_invalid_service_name_exits_2(env, bad):
|
||||
result = run(env, service=bad)
|
||||
assert result.returncode == 2
|
||||
assert not env["argv_log"].exists()
|
||||
|
||||
|
||||
def test_missing_compose_file_exits_2(env):
|
||||
(env["repo"] / "services" / "vikunja" / "docker-compose.yml").unlink()
|
||||
result = run(env)
|
||||
assert result.returncode == 2
|
||||
assert not env["argv_log"].exists()
|
||||
163
jobs/deploy-runner/tests/test_runner_integration.py
Normal file
163
jobs/deploy-runner/tests/test_runner_integration.py
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
"""End-to-end test of deploy-runner.sh on a fake node.
|
||||
|
||||
Runs the real bash runner against temp directories with a stubbed `docker`, in
|
||||
local (VPS) mode so no rsync/ssh is involved. Covers the full loop the control
|
||||
plane depends on: dispatch file in → validate → deploy → action_result event
|
||||
out → idempotency marker → inbox drained.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO = Path(__file__).resolve().parents[3]
|
||||
RUNNER = REPO / "jobs" / "deploy-runner" / "deploy-runner.sh"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def node(tmp_path: Path):
|
||||
"""A fake node: repo checkout, /opt/homelab-alike runtime, docker stub."""
|
||||
repo = tmp_path / "repo"
|
||||
(repo / "scripts" / "deploy").mkdir(parents=True)
|
||||
shutil.copy(REPO / "scripts" / "deploy" / "deploy-service.sh",
|
||||
repo / "scripts" / "deploy" / "deploy-service.sh")
|
||||
os.chmod(repo / "scripts" / "deploy" / "deploy-service.sh", 0o755)
|
||||
|
||||
(repo / "hosts" / "piha").mkdir(parents=True)
|
||||
(repo / "hosts" / "piha" / "services.yaml").write_text(
|
||||
"host: piha\nservices:\n vikunja:\n role: task-tracker\n"
|
||||
)
|
||||
(repo / "services" / "vikunja").mkdir(parents=True)
|
||||
(repo / "services" / "vikunja" / "docker-compose.yml").write_text("services: {}\n")
|
||||
|
||||
runtime = tmp_path / "runtime"
|
||||
(runtime / "actions" / "deploy" / "piha").mkdir(parents=True)
|
||||
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
argv_log = tmp_path / "docker-argv.txt"
|
||||
(bindir / "docker").write_text(
|
||||
f'#!/usr/bin/env bash\nprintf "%s " "$@" >> {argv_log}\necho >> {argv_log}\n'
|
||||
f'exit ${{DOCKER_STUB_EXIT:-0}}\n'
|
||||
)
|
||||
os.chmod(bindir / "docker", 0o755)
|
||||
|
||||
return {"repo": repo, "runtime": runtime, "bindir": bindir, "argv_log": argv_log,
|
||||
"inbox": runtime / "actions" / "deploy" / "piha",
|
||||
"events": runtime / "events" / "piha"}
|
||||
|
||||
|
||||
def dispatch(fake_node, action_id="redeploy-piha-vikunja", **overrides):
|
||||
action = {"action_id": action_id, "type": "redeploy", "node": "piha",
|
||||
"service": "vikunja", "dispatched_at": 0}
|
||||
action.update(overrides)
|
||||
(fake_node["inbox"] / f"{action_id}.json").write_text(json.dumps(action))
|
||||
|
||||
|
||||
def run_runner(fake_node, **extra_env):
|
||||
env = dict(os.environ)
|
||||
env.update({
|
||||
"PATH": f"{fake_node['bindir']}:{os.environ['PATH']}",
|
||||
"NODE_NAME": "piha",
|
||||
"REPO_PATH": str(fake_node["repo"]),
|
||||
"RUNTIME_PATH": str(fake_node["runtime"]),
|
||||
"VPS_EVENTS_HOST": "", # local mode: executor shares the same filesystem
|
||||
})
|
||||
env.update(extra_env)
|
||||
return subprocess.run([str(RUNNER)], capture_output=True, text=True, env=env)
|
||||
|
||||
|
||||
def results(node) -> list[dict]:
|
||||
if not node["events"].exists():
|
||||
return []
|
||||
return [json.loads(p.read_text())
|
||||
for p in sorted(node["events"].glob("evt-*-action_result-*.json"))]
|
||||
|
||||
|
||||
def test_successful_redeploy_reports_and_drains(node):
|
||||
dispatch(node)
|
||||
proc = run_runner(node)
|
||||
assert proc.returncode == 0, proc.stdout + proc.stderr
|
||||
|
||||
assert "compose -f" in node["argv_log"].read_text()
|
||||
assert "--force-recreate" in node["argv_log"].read_text()
|
||||
|
||||
events = results(node)
|
||||
assert len(events) == 1
|
||||
assert events[0]["payload"] == {
|
||||
"action_id": "redeploy-piha-vikunja", "success": True, "error": "",
|
||||
"node": "piha", "source": "deploy-runner",
|
||||
}
|
||||
|
||||
assert list(node["inbox"].glob("*.json")) == []
|
||||
assert (node["runtime"] / "state" / "processed-deploy-actions"
|
||||
/ "redeploy-piha-vikunja.done").exists()
|
||||
|
||||
|
||||
def test_failed_deploy_reports_compose_output(node):
|
||||
dispatch(node)
|
||||
proc = run_runner(node, DOCKER_STUB_EXIT="1")
|
||||
assert proc.returncode == 0 # the runner itself succeeds; the action fails
|
||||
|
||||
event = results(node)[0]
|
||||
assert event["payload"]["success"] is False
|
||||
assert "deploy-service.sh exited 1" in event["payload"]["error"]
|
||||
assert event["severity"] == "high"
|
||||
# Full output kept on the node for forensics
|
||||
assert list((node["runtime"] / "logs" / "deploy-runner").glob("*.log"))
|
||||
|
||||
|
||||
def test_rejected_action_is_reported_not_silently_dropped(node):
|
||||
"""An action for a service this node does not declare must come back as a
|
||||
failure — otherwise it sits in running/ until the executor times it out."""
|
||||
dispatch(node, service="gokapi")
|
||||
run_runner(node)
|
||||
|
||||
event = results(node)[0]
|
||||
assert event["payload"]["success"] is False
|
||||
assert "not in hosts/piha/services.yaml" in event["payload"]["error"]
|
||||
assert not node["argv_log"].exists() # docker never invoked
|
||||
assert list(node["inbox"].glob("*.json")) == []
|
||||
|
||||
|
||||
def test_action_for_another_node_is_refused(node):
|
||||
dispatch(node, node="solaria")
|
||||
run_runner(node)
|
||||
|
||||
event = results(node)[0]
|
||||
assert "addressed to node" in event["payload"]["error"]
|
||||
assert not node["argv_log"].exists()
|
||||
|
||||
|
||||
def test_container_restart_is_not_executed_by_the_runner(node):
|
||||
"""node-agent's job. The runner refuses anything but redeploy."""
|
||||
dispatch(node, type="container_restart")
|
||||
run_runner(node)
|
||||
|
||||
event = results(node)[0]
|
||||
assert "not executable by the deploy runner" in event["payload"]["error"]
|
||||
assert not node["argv_log"].exists()
|
||||
|
||||
|
||||
def test_already_processed_action_is_not_redeployed(node):
|
||||
dispatch(node)
|
||||
run_runner(node)
|
||||
first = len(results(node))
|
||||
|
||||
dispatch(node) # same action_id arrives again
|
||||
run_runner(node)
|
||||
|
||||
assert len(results(node)) == first # no second result
|
||||
assert len(node["argv_log"].read_text().splitlines()) == 1 # docker ran once
|
||||
assert list(node["inbox"].glob("*.json")) == []
|
||||
|
||||
|
||||
def test_missing_node_name_fails_loudly(node):
|
||||
proc = run_runner(node, NODE_NAME="")
|
||||
assert proc.returncode == 1
|
||||
assert "NODE_NAME is not set" in proc.stdout + proc.stderr
|
||||
|
|
@ -6,7 +6,9 @@ set -e
|
|||
|
||||
# Configuration
|
||||
REPO_PATH="${HOME}/homelab-codex-ws"
|
||||
RUNTIME_PATH="/opt/homelab"
|
||||
# Env-overridable so the script can be exercised against a scratch runtime dir
|
||||
# without writing into the node's real /opt/homelab.
|
||||
RUNTIME_PATH="${RUNTIME_PATH:-/opt/homelab}"
|
||||
HOSTNAME=$(hostname | tr '[:lower:]' '[:upper:]')
|
||||
CURRENT_OS_HOST=$(hostname)
|
||||
|
||||
|
|
@ -81,57 +83,33 @@ if [ ${#SERVICES[@]} -eq 0 ]; then
|
|||
fi
|
||||
|
||||
# 3. Deploy Services
|
||||
#
|
||||
# The per-service compose invocation lives in deploy-service.sh, shared with the
|
||||
# agent-driven single-service redeploy (services/deploy-runner/). Both callers
|
||||
# MUST assemble the -f arguments identically: docker compose derives the project
|
||||
# name from the first -f file's directory, and a mismatch plus --remove-orphans
|
||||
# is what wiped the control-plane on VPS on 2026-06-25. Services with their own
|
||||
# deploy-local.sh are skipped by deploy-service.sh (exit 3).
|
||||
DEPLOY_SERVICE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/deploy-service.sh"
|
||||
|
||||
for service in "${SERVICES[@]}"; do
|
||||
# Services that ship their own deploy-local.sh have a dedicated, orchestrated
|
||||
# deploy path (e.g. control-plane via deploy-control-plane.sh → deploy-local.sh).
|
||||
# Deploying them in this loop too would invoke `docker compose up` with a
|
||||
# different Compose project name than their dedicated path uses (this loop runs
|
||||
# from ${REPO_PATH}; deploy-local.sh runs from services/<svc>/), so `up
|
||||
# -d --remove-orphans` would Recreate and tear down the already-running
|
||||
# containers — exactly what wiped the control-plane on VPS on 2026-06-25.
|
||||
# Their own deploy path owns them; skip here.
|
||||
if [ -f "${REPO_PATH}/services/${service}/deploy-local.sh" ]; then
|
||||
echo "Skipping ${service}: ma własną ścieżkę deployu (services/${service}/deploy-local.sh), pomijam"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "Deploying service: ${service}..."
|
||||
|
||||
COMPOSE_FILE="${REPO_PATH}/services/${service}/docker-compose.yml"
|
||||
|
||||
if [ ! -f "$COMPOSE_FILE" ]; then
|
||||
echo "Warning: Compose file not found for ${service} at ${COMPOSE_FILE}"
|
||||
continue
|
||||
fi
|
||||
|
||||
TARGET_DIR="${RUNTIME_PATH}/services/${service}"
|
||||
mkdir -p "$TARGET_DIR"
|
||||
|
||||
OVERRIDE_FILE="${HOST_DIR}/runtime/${service}/docker-compose.override.yml"
|
||||
rc=0
|
||||
"$DEPLOY_SERVICE" \
|
||||
--repo "$REPO_PATH" \
|
||||
--host-dir "$HOST_DIR" \
|
||||
--service "$service" \
|
||||
--build-if-needed \
|
||||
--remove-orphans || rc=$?
|
||||
|
||||
COMPOSE_CMD="docker compose -f ${COMPOSE_FILE}"
|
||||
if [ -f "$OVERRIDE_FILE" ]; then
|
||||
echo "Using override file for ${service}"
|
||||
COMPOSE_CMD="${COMPOSE_CMD} -f ${OVERRIDE_FILE}"
|
||||
fi
|
||||
|
||||
ENV_FILE="${REPO_PATH}/services/${service}/.env"
|
||||
if [ -f "$ENV_FILE" ]; then
|
||||
COMPOSE_CMD="${COMPOSE_CMD} --env-file ${ENV_FILE}"
|
||||
fi
|
||||
|
||||
# Services with their own Dockerfile need a rebuild on every deploy —
|
||||
# otherwise `up -d` sees "container running, image tag unchanged" and skips
|
||||
# rebuilding even when src/ changed, silently leaving the old code running
|
||||
# (backlog c858dbc). Services with a prebuilt image (no Dockerfile) don't
|
||||
# need --build.
|
||||
if [ -f "${REPO_PATH}/services/${service}/Dockerfile" ]; then
|
||||
echo "Building ${service} (has Dockerfile)"
|
||||
$COMPOSE_CMD up -d --build --remove-orphans
|
||||
else
|
||||
echo "Using prebuilt image for ${service}"
|
||||
$COMPOSE_CMD up -d --remove-orphans
|
||||
fi
|
||||
case "$rc" in
|
||||
0) ;; # deployed
|
||||
3) ;; # owns its deploy path — already reported, skip
|
||||
2) echo "Warning: skipping ${service} (validation failed, see above)" ;;
|
||||
*) exit "$rc" ;; # real deploy failure — preserve set -e semantics
|
||||
esac
|
||||
done
|
||||
|
||||
echo "--- Deployment Complete ---"
|
||||
|
|
|
|||
134
scripts/deploy/deploy-service.sh
Executable file
134
scripts/deploy/deploy-service.sh
Executable file
|
|
@ -0,0 +1,134 @@
|
|||
#!/usr/bin/env bash
|
||||
# scripts/deploy/deploy-service.sh — deploy exactly ONE service on the local node.
|
||||
#
|
||||
# Extracted from deploy-node.sh so that the human whole-node deploy and the
|
||||
# agent-driven single-service redeploy (services/deploy-runner/) share ONE
|
||||
# compose invocation. That sharing is the whole point: docker compose derives
|
||||
# the PROJECT NAME from the directory of the first -f file (verified:
|
||||
# `docker compose -f services/node-agent/docker-compose.yml config` → name
|
||||
# "node-agent", independent of cwd; adding --project-directory changes it).
|
||||
# A caller that assembles the -f arguments differently lands in a DIFFERENT
|
||||
# project, where `--remove-orphans` tears down the already-running stack —
|
||||
# exactly what wiped the control-plane on VPS on 2026-06-25. Every caller goes
|
||||
# through here.
|
||||
#
|
||||
# Usage:
|
||||
# deploy-service.sh --repo <repo-path> --host-dir <hosts/<node> path>
|
||||
# --service <name>
|
||||
# [--build-if-needed] [--force-recreate] [--remove-orphans]
|
||||
#
|
||||
# Flags:
|
||||
# --build-if-needed add --build when the service ships its own Dockerfile
|
||||
# (whole-node deploy path — see the rationale below)
|
||||
# --force-recreate recreate containers even when config is unchanged
|
||||
# (remediation path: plain `up -d` is a no-op for an
|
||||
# unhealthy-but-unchanged service, i.e. it would "succeed"
|
||||
# without doing anything)
|
||||
# --remove-orphans human whole-node deploy only; never used by the agent
|
||||
# path, where a project-name mismatch must not be able to
|
||||
# delete containers
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 deployed
|
||||
# 2 usage / validation error
|
||||
# 3 service owns its deploy path (services/<svc>/deploy-local.sh) — skipped
|
||||
# * docker compose exit code
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_PATH=""
|
||||
HOST_DIR=""
|
||||
SERVICE=""
|
||||
BUILD_IF_NEEDED=false
|
||||
FORCE_RECREATE=false
|
||||
REMOVE_ORPHANS=false
|
||||
|
||||
usage() {
|
||||
sed -n '2,30p' "${BASH_SOURCE[0]}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--repo) REPO_PATH="${2:-}"; shift 2 ;;
|
||||
--host-dir) HOST_DIR="${2:-}"; shift 2 ;;
|
||||
--service) SERVICE="${2:-}"; shift 2 ;;
|
||||
--build-if-needed) BUILD_IF_NEEDED=true; shift ;;
|
||||
--force-recreate) FORCE_RECREATE=true; shift ;;
|
||||
--remove-orphans) REMOVE_ORPHANS=true; shift ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo "deploy-service.sh: unknown argument: $1" >&2; usage ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -n "$REPO_PATH" && -n "$HOST_DIR" && -n "$SERVICE" ]] || {
|
||||
echo "deploy-service.sh: --repo, --host-dir and --service are required" >&2
|
||||
usage
|
||||
}
|
||||
|
||||
# Service names are used to build filesystem paths. Reject anything that is not
|
||||
# a plain kebab-case service name so a malformed (or maliciously crafted)
|
||||
# dispatched action can never escape services/ via ../.
|
||||
if [[ ! "$SERVICE" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]]; then
|
||||
echo "deploy-service.sh: invalid service name: '${SERVICE}'" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
[[ -d "$REPO_PATH" ]] || { echo "deploy-service.sh: repo not found: ${REPO_PATH}" >&2; exit 2; }
|
||||
[[ -d "$HOST_DIR" ]] || { echo "deploy-service.sh: host dir not found: ${HOST_DIR}" >&2; exit 2; }
|
||||
|
||||
SERVICE_DIR="${REPO_PATH}/services/${SERVICE}"
|
||||
COMPOSE_FILE="${SERVICE_DIR}/docker-compose.yml"
|
||||
|
||||
if [[ ! -f "$COMPOSE_FILE" ]]; then
|
||||
echo "deploy-service.sh: compose file not found for ${SERVICE} at ${COMPOSE_FILE}" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Services that ship their own deploy-local.sh have a dedicated, orchestrated
|
||||
# deploy path (e.g. control-plane via deploy-control-plane.sh → deploy-local.sh)
|
||||
# which does host-level preparation (dirs, ownership, sudo) this script cannot
|
||||
# do. Deploying them from here would also run `up` against a stack whose own
|
||||
# path expects to own it. Their own deploy path owns them; skip.
|
||||
if [[ -f "${SERVICE_DIR}/deploy-local.sh" ]]; then
|
||||
echo "Skipping ${SERVICE}: ma własną ścieżkę deployu (services/${SERVICE}/deploy-local.sh), pomijam"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
COMPOSE_ARGS=(docker compose -f "$COMPOSE_FILE")
|
||||
|
||||
OVERRIDE_FILE="${HOST_DIR}/runtime/${SERVICE}/docker-compose.override.yml"
|
||||
if [[ -f "$OVERRIDE_FILE" ]]; then
|
||||
echo "Using override file for ${SERVICE}"
|
||||
COMPOSE_ARGS+=(-f "$OVERRIDE_FILE")
|
||||
fi
|
||||
|
||||
ENV_FILE="${SERVICE_DIR}/.env"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
COMPOSE_ARGS+=(--env-file "$ENV_FILE")
|
||||
fi
|
||||
|
||||
COMPOSE_ARGS+=(up -d)
|
||||
|
||||
# Services with their own Dockerfile need a rebuild on a whole-node deploy —
|
||||
# otherwise `up -d` sees "container running, image tag unchanged" and skips
|
||||
# rebuilding even when src/ changed, silently leaving the old code running
|
||||
# (backlog c858dbc). The agent redeploy path deliberately does NOT pass
|
||||
# --build-if-needed: a redeploy is a reconcile, not a code ship, and building
|
||||
# on the 4 GiB VPS mid-incident is an OOM risk.
|
||||
if [[ "$BUILD_IF_NEEDED" == "true" && -f "${SERVICE_DIR}/Dockerfile" ]]; then
|
||||
echo "Building ${SERVICE} (has Dockerfile)"
|
||||
COMPOSE_ARGS+=(--build)
|
||||
fi
|
||||
|
||||
if [[ "$FORCE_RECREATE" == "true" ]]; then
|
||||
COMPOSE_ARGS+=(--force-recreate)
|
||||
fi
|
||||
|
||||
if [[ "$REMOVE_ORPHANS" == "true" ]]; then
|
||||
COMPOSE_ARGS+=(--remove-orphans)
|
||||
fi
|
||||
|
||||
echo "Deploying service: ${SERVICE}..."
|
||||
echo "+ ${COMPOSE_ARGS[*]}"
|
||||
"${COMPOSE_ARGS[@]}"
|
||||
|
|
@ -79,7 +79,11 @@ services:
|
|||
command: python src/executor.py
|
||||
volumes:
|
||||
- /opt/homelab:/opt/homelab
|
||||
- ../..:/repo
|
||||
# Read-only since the redeploy fix: the executor used to run
|
||||
# scripts/deploy/deploy-node.sh out of this mount (it never worked — see
|
||||
# jobs/deploy-runner/README.md). Deploys now happen on the node itself, so
|
||||
# nothing here needs write access to the checkout.
|
||||
- ../..:/repo:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
|
|
|
|||
|
|
@ -21,6 +21,15 @@ RUNTIME_PATH = os.getenv("RUNTIME_PATH", "/opt/homelab")
|
|||
ACTIONS_DIR = Path(RUNTIME_PATH) / "actions"
|
||||
EVENTS_DIR = Path(RUNTIME_PATH) / "events"
|
||||
DISPATCH_DIR = ACTIONS_DIR / "dispatch"
|
||||
# Separate inbox for redeploy actions, consumed by the host-side deploy runner
|
||||
# (jobs/deploy-runner/) rather than by node-agent. It must NOT share
|
||||
# DISPATCH_DIR: node-agent's process_dispatched_actions() deletes and
|
||||
# failure-reports every file it finds in its inbox, so a redeploy landing there
|
||||
# would be killed before the runner ever saw it.
|
||||
DEPLOY_DISPATCH_DIR = ACTIONS_DIR / "deploy"
|
||||
# The executor no longer reads the repo at all (the old redeploy path ran a
|
||||
# script out of it). Kept only so an operator can still see which checkout the
|
||||
# container is wired to; nothing in this module resolves paths against it.
|
||||
REPO_ROOT = Path(os.getenv("REPO_ROOT", "/repo"))
|
||||
|
||||
# SSH configuration
|
||||
|
|
@ -41,6 +50,15 @@ SSH_OPTIONS = [
|
|||
# code change.
|
||||
ACTION_TIMEOUT_SECS = int(os.getenv("ACTION_TIMEOUT_SECS", "300"))
|
||||
|
||||
# Redeploy gets its own, longer budget: the deploy runner polls once a minute
|
||||
# and a `docker compose up` may pull images before it can report anything.
|
||||
REDEPLOY_TIMEOUT_SECS = int(os.getenv("REDEPLOY_TIMEOUT_SECS", "900"))
|
||||
|
||||
# Action types the executor hands to an on-node agent instead of resolving
|
||||
# synchronously. They stay in running/ until an action_result event arrives (or
|
||||
# the type's timeout expires) — see _reconcile_running_actions.
|
||||
DISPATCHED_ACTION_TYPES = {"container_restart", "redeploy"}
|
||||
|
||||
# Matches evt-<node>-<unixts>-<type>-<svc>.json, same convention as
|
||||
# node_agent.py / observer.py / operator_ui.py.
|
||||
_EVENT_TS_RE = re.compile(r"-(\d{9,11})-")
|
||||
|
|
@ -58,6 +76,7 @@ class Executor:
|
|||
for s in ["approved", "running", "completed", "failed", "rejected"]:
|
||||
(ACTIONS_DIR / s).mkdir(parents=True, exist_ok=True)
|
||||
DISPATCH_DIR.mkdir(parents=True, exist_ok=True)
|
||||
DEPLOY_DISPATCH_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def process_actions(self):
|
||||
# Update heartbeat
|
||||
|
|
@ -104,19 +123,25 @@ class Executor:
|
|||
service = data.get("service")
|
||||
|
||||
if action_type == "redeploy":
|
||||
# Full service redeploy via the repo deploy script
|
||||
cmd = [
|
||||
str(REPO_ROOT / "scripts" / "deploy" / "deploy-node.sh"),
|
||||
node,
|
||||
service
|
||||
]
|
||||
logger.info(f"Running command: {' '.join(cmd)}")
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, cwd=str(REPO_ROOT))
|
||||
if result.returncode == 0:
|
||||
success = True
|
||||
else:
|
||||
# Single-service redeploy, dispatched to the host-side deploy
|
||||
# runner on the target node (jobs/deploy-runner/). Same pull
|
||||
# architecture as container_restart — the executor never runs a
|
||||
# deploy itself. It used to try: it ran deploy-node.sh inside
|
||||
# this container, a script that ignores its arguments and needs
|
||||
# a repo at ${HOME}/homelab-codex-ws (plus git and the docker
|
||||
# CLI) that does not exist here, so every redeploy failed at the
|
||||
# first line (recon D14/D15).
|
||||
if not node or not service:
|
||||
success = False
|
||||
error_msg = result.stderr or result.stdout
|
||||
error_msg = (
|
||||
f"redeploy requires both node and service "
|
||||
f"(node={node!r}, service={service!r})"
|
||||
)
|
||||
else:
|
||||
self._dispatch_redeploy(action_id, node, service)
|
||||
# Stays in "running" until the runner reports an
|
||||
# action_result — same as container_restart below.
|
||||
return
|
||||
|
||||
elif action_type == "container_restart":
|
||||
# No SSH from the executor (see CLAUDE.md / docs/backlog.md): the
|
||||
|
|
@ -210,12 +235,47 @@ class Executor:
|
|||
except Exception as e:
|
||||
logger.error(f"Failed to dispatch {action_id} to {node}: {e}")
|
||||
|
||||
def _reconcile_running_actions(self):
|
||||
"""Resolve container_restart actions previously dispatched to a node-agent.
|
||||
def _dispatch_redeploy(self, action_id, node, service):
|
||||
"""Write a redeploy action for the deploy runner on `node` to pick up.
|
||||
|
||||
Other action types (redeploy/disk_cleanup/alert_only) resolve
|
||||
synchronously inside _execute_action and never linger in running/, so
|
||||
they are not touched here.
|
||||
Deliberately a different inbox from container_restart: node-agent
|
||||
consumes actions/dispatch/<node>/ and rejects (and deletes) anything
|
||||
that is not container_restart, so redeploys get their own
|
||||
actions/deploy/<node>/ tree. The runner on the node pulls it over the
|
||||
same rsync/ssh channel node-agent already uses, deploys locally with
|
||||
scripts/deploy/deploy-service.sh, and reports back as an action_result
|
||||
event — the executor still never initiates a connection to a node.
|
||||
|
||||
Does not resolve the action itself; _reconcile_running_actions() does.
|
||||
"""
|
||||
inbox = DEPLOY_DISPATCH_DIR / node
|
||||
inbox.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"action_id": action_id,
|
||||
"type": "redeploy",
|
||||
"node": node,
|
||||
"service": service,
|
||||
"dispatched_at": time.time(),
|
||||
}
|
||||
try:
|
||||
_atomic_write_json(inbox / f"{action_id}.json", payload)
|
||||
logger.info(
|
||||
f"Dispatched redeploy {action_id} (service={service}) "
|
||||
f"to deploy runner on {node}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to dispatch {action_id} to {node}: {e}")
|
||||
|
||||
def _reconcile_running_actions(self):
|
||||
"""Resolve actions previously dispatched to an on-node agent.
|
||||
|
||||
Covers both dispatched types: container_restart (executed by node-agent)
|
||||
and redeploy (executed by the host-side deploy runner). Both report the
|
||||
outcome the same way — an action_result event carrying the action_id —
|
||||
so the resolution logic is identical; only the timeout differs.
|
||||
|
||||
Synchronous types (disk_cleanup/alert_only) resolve inside
|
||||
_execute_action and never linger in running/, so they are not touched.
|
||||
"""
|
||||
running_dir = ACTIONS_DIR / "running"
|
||||
if not running_dir.exists():
|
||||
|
|
@ -229,24 +289,29 @@ class Executor:
|
|||
logger.error(f"Failed to read running action {action_file.name}: {e}")
|
||||
continue
|
||||
|
||||
if data.get("type") != "container_restart":
|
||||
action_type = data.get("type")
|
||||
if action_type not in DISPATCHED_ACTION_TYPES:
|
||||
continue
|
||||
|
||||
action_id = data.get("action_id") or action_file.stem
|
||||
node = data.get("node")
|
||||
started_at = data.get("started_at") or 0
|
||||
executor_name = "deploy runner" if action_type == "redeploy" else "node-agent"
|
||||
timeout_secs = (
|
||||
REDEPLOY_TIMEOUT_SECS if action_type == "redeploy" else ACTION_TIMEOUT_SECS
|
||||
)
|
||||
|
||||
result = self._find_action_result(node, action_id, started_at)
|
||||
if result is not None:
|
||||
success = bool(result.get("success"))
|
||||
error_msg = "" if success else (result.get("error") or "node-agent reported failure")
|
||||
error_msg = "" if success else (result.get("error") or f"{executor_name} reported failure")
|
||||
self._finalize_action(action_id, action_file, data, success, error_msg)
|
||||
continue
|
||||
|
||||
if started_at and (time.time() - started_at) > ACTION_TIMEOUT_SECS:
|
||||
if started_at and (time.time() - started_at) > timeout_secs:
|
||||
error_msg = (
|
||||
f"Timed out after {ACTION_TIMEOUT_SECS}s waiting for node-agent "
|
||||
f"on '{node}' to report a result for container_restart "
|
||||
f"Timed out after {timeout_secs}s waiting for {executor_name} "
|
||||
f"on '{node}' to report a result for {action_type} "
|
||||
f"(action_id={action_id})"
|
||||
)
|
||||
logger.error(f"Action {action_id} timed out: {error_msg}")
|
||||
|
|
|
|||
|
|
@ -37,13 +37,14 @@ except Exception:
|
|||
# rather than a full redeploy: the container is present but not running, or
|
||||
# running with a failing health check — a restart plausibly heals both.
|
||||
# healthcheck_failed added 2026-07-29 (recon D14/D15): it used to route to
|
||||
# redeploy, which is broken as wired (executor calls deploy-node.sh with
|
||||
# arguments it ignores, at a path that does not exist in the container), so
|
||||
# 3376 healthcheck_failed events dead-ended. redeploy returns to the map only
|
||||
# once it actually works (etap 2). Everything not listed here
|
||||
# (service_unhealthy, deployment_failed, missing_service) still falls through
|
||||
# to redeploy — theoretical until etap 2 fixes the executor, kept as-is so the
|
||||
# drift stays visible in pending actions.
|
||||
# redeploy, which was broken as wired (the executor ran deploy-node.sh with
|
||||
# arguments it ignores, from a repo path that does not exist in its container),
|
||||
# so 3376 healthcheck_failed events dead-ended. Redeploy works as of 2026-08-03
|
||||
# (executor dispatch → jobs/deploy-runner/ on the node), but healthcheck_failed
|
||||
# deliberately STAYS here: restart-first is the cheaper, lower-risk remediation
|
||||
# for a container that is up with a failing probe. Everything not listed here
|
||||
# (service_unhealthy, deployment_failed, missing_service) falls through to
|
||||
# redeploy, which now actually executes.
|
||||
# mqtt_unreachable was removed 2026-07-28: the observer never creates incidents
|
||||
# with that trigger_type, so the branch was dead code (recon
|
||||
# docs/architecture/RECON-multiagent-2026-07-27.md, D15). stability-agent still
|
||||
|
|
@ -446,10 +447,9 @@ class Supervisor:
|
|||
else:
|
||||
# Full redeploy: container is running but service is broken,
|
||||
# or the cause is unknown / not a simple restart candidate.
|
||||
# NOTE: the redeploy action type is currently theoretical — the
|
||||
# executor's redeploy path is broken until etap 2 (see
|
||||
# CONTAINER_RESTART_TRIGGERS comment). Generated anyway so the
|
||||
# drift is visible to the operator in pending actions.
|
||||
# Executed on the target node by jobs/deploy-runner/ once an
|
||||
# operator approves — `docker compose up -d --force-recreate` for
|
||||
# this one service, from the checkout the node already has.
|
||||
action = {
|
||||
"action_id": action_id,
|
||||
"timestamp": time.time(),
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@ def _setup_executor(tmp_path: Path, monkeypatch) -> Executor:
|
|||
monkeypatch.setattr(executor_module, "ACTIONS_DIR", actions)
|
||||
monkeypatch.setattr(executor_module, "EVENTS_DIR", events)
|
||||
monkeypatch.setattr(executor_module, "DISPATCH_DIR", actions / "dispatch")
|
||||
# Must be patched too: _ensure_dirs() creates it, and the module default
|
||||
# points at the real /opt/homelab.
|
||||
monkeypatch.setattr(executor_module, "DEPLOY_DISPATCH_DIR", actions / "deploy")
|
||||
monkeypatch.setattr(executor_module, "REPO_ROOT", repo)
|
||||
return Executor()
|
||||
|
||||
|
|
@ -212,17 +215,19 @@ def test_reconcile_does_not_time_out_before_deadline(tmp_path, monkeypatch):
|
|||
# Other action types are untouched by the reconcile loop
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_reconcile_ignores_non_container_restart_running_actions(tmp_path, monkeypatch):
|
||||
def test_reconcile_ignores_synchronous_running_actions(tmp_path, monkeypatch):
|
||||
"""Only dispatched types (container_restart, redeploy) are reconciled here.
|
||||
disk_cleanup/alert_only resolve inside _execute_action and never linger."""
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
running_path = tmp_path / "actions" / "running" / "redeploy-x.json"
|
||||
running_path = tmp_path / "actions" / "running" / "disk-cleanup-x.json"
|
||||
running_path.write_text(json.dumps({
|
||||
"action_id": "redeploy-x", "type": "redeploy", "node": "piha",
|
||||
"action_id": "disk-cleanup-x", "type": "disk_cleanup", "node": "piha",
|
||||
"started_at": time.time() - 999_999,
|
||||
}))
|
||||
|
||||
ex._reconcile_running_actions()
|
||||
|
||||
assert running_path.exists() # untouched: redeploy resolves synchronously elsewhere
|
||||
assert running_path.exists() # untouched
|
||||
|
||||
|
||||
def test_alert_only_action_resolves_synchronously_not_via_dispatch(tmp_path, monkeypatch):
|
||||
|
|
|
|||
189
services/control-plane/tests/test_executor_redeploy.py
Normal file
189
services/control-plane/tests/test_executor_redeploy.py
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
"""Tests for the Executor redeploy path (recon D14/D15 — etap 2).
|
||||
|
||||
Before this, `redeploy` ran scripts/deploy/deploy-node.sh inside the executor
|
||||
container: a script that ignores its arguments, expects a repo at
|
||||
${HOME}/homelab-codex-ws (absent there, along with git and the docker CLI), and
|
||||
would have deployed the EXECUTOR's own host's full service set rather than the
|
||||
action's target. Every redeploy failed, which is why healthcheck_failed had to
|
||||
be rerouted to container_restart and the action queue never drained.
|
||||
|
||||
Now redeploy follows the same pull architecture as container_restart: the
|
||||
executor writes a dispatch file, the host-side deploy runner on the target node
|
||||
(jobs/deploy-runner/) executes it and reports back an action_result event.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
import executor as executor_module
|
||||
from executor import Executor
|
||||
|
||||
|
||||
def _setup_executor(tmp_path: Path, monkeypatch) -> Executor:
|
||||
actions = tmp_path / "actions"
|
||||
events = tmp_path / "events"
|
||||
for d in (actions, events):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
monkeypatch.setattr(executor_module, "ACTIONS_DIR", actions)
|
||||
monkeypatch.setattr(executor_module, "EVENTS_DIR", events)
|
||||
monkeypatch.setattr(executor_module, "DISPATCH_DIR", actions / "dispatch")
|
||||
monkeypatch.setattr(executor_module, "DEPLOY_DISPATCH_DIR", actions / "deploy")
|
||||
return Executor()
|
||||
|
||||
|
||||
def _write_approved(tmp_path, action_id, node="piha", service="vikunja"):
|
||||
action = {
|
||||
"action_id": action_id,
|
||||
"type": "redeploy",
|
||||
"node": node,
|
||||
"service": service,
|
||||
"status": "approved",
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
path = tmp_path / "actions" / "approved" / f"{action_id}.json"
|
||||
path.write_text(json.dumps(action))
|
||||
return path
|
||||
|
||||
|
||||
def _write_action_result_event(tmp_path, node, action_id, success, error="", ts=None):
|
||||
ts = int(ts if ts is not None else time.time())
|
||||
node_dir = tmp_path / "events" / node
|
||||
node_dir.mkdir(parents=True, exist_ok=True)
|
||||
event_id = f"evt-{node}-{ts}-action_result-{action_id}"
|
||||
(node_dir / f"{event_id}.json").write_text(json.dumps({
|
||||
"id": event_id,
|
||||
"timestamp": ts,
|
||||
"type": "action_result",
|
||||
"node": node,
|
||||
"payload": {"action_id": action_id, "success": success, "error": error,
|
||||
"node": node, "source": "deploy-runner"},
|
||||
}))
|
||||
|
||||
|
||||
def _exists(tmp_path, state, action_id):
|
||||
return (tmp_path / "actions" / state / f"{action_id}.json").exists()
|
||||
|
||||
|
||||
def _read(tmp_path, state, action_id):
|
||||
return json.loads((tmp_path / "actions" / state / f"{action_id}.json").read_text())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dispatch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_redeploy_is_dispatched_not_executed_locally(tmp_path, monkeypatch):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
|
||||
def fail(*args, **kwargs):
|
||||
raise AssertionError("executor must never run a deploy itself")
|
||||
|
||||
monkeypatch.setattr(executor_module.subprocess, "run", fail)
|
||||
|
||||
ex._execute_action(_write_approved(tmp_path, "redeploy-piha-vikunja"))
|
||||
|
||||
dispatch_file = tmp_path / "actions" / "deploy" / "piha" / "redeploy-piha-vikunja.json"
|
||||
assert dispatch_file.exists()
|
||||
payload = json.loads(dispatch_file.read_text())
|
||||
assert payload["type"] == "redeploy"
|
||||
assert payload["node"] == "piha"
|
||||
assert payload["service"] == "vikunja"
|
||||
assert "dispatched_at" in payload
|
||||
|
||||
|
||||
def test_redeploy_inbox_is_separate_from_node_agent_dispatch(tmp_path, monkeypatch):
|
||||
"""node-agent deletes and failure-reports anything in its own inbox that is
|
||||
not container_restart, so a redeploy must never be written there."""
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "redeploy-piha-vikunja"))
|
||||
|
||||
node_agent_inbox = tmp_path / "actions" / "dispatch"
|
||||
assert not list(node_agent_inbox.glob("**/*.json"))
|
||||
|
||||
|
||||
def test_redeploy_stays_running_until_reported(tmp_path, monkeypatch):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-1"))
|
||||
|
||||
assert _exists(tmp_path, "running", "rd-1")
|
||||
assert not _exists(tmp_path, "completed", "rd-1")
|
||||
assert not _exists(tmp_path, "failed", "rd-1")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("node,service", [("", "vikunja"), ("piha", ""), ("", "")])
|
||||
def test_redeploy_without_node_or_service_fails_immediately(tmp_path, monkeypatch, node, service):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-bad", node=node, service=service))
|
||||
|
||||
assert _exists(tmp_path, "failed", "rd-bad")
|
||||
assert "requires both node and service" in _read(tmp_path, "failed", "rd-bad")["error"]
|
||||
assert not list((tmp_path / "actions" / "deploy").glob("**/*.json"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reconcile
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_redeploy_completes_on_success_result(tmp_path, monkeypatch):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-2"))
|
||||
|
||||
_write_action_result_event(tmp_path, "piha", "rd-2", success=True)
|
||||
ex._reconcile_running_actions()
|
||||
|
||||
assert _exists(tmp_path, "completed", "rd-2")
|
||||
assert not _exists(tmp_path, "running", "rd-2")
|
||||
|
||||
|
||||
def test_redeploy_fails_on_failure_result(tmp_path, monkeypatch):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-3"))
|
||||
|
||||
_write_action_result_event(tmp_path, "piha", "rd-3", success=False,
|
||||
error="deploy-service.sh exited 1")
|
||||
ex._reconcile_running_actions()
|
||||
|
||||
assert _read(tmp_path, "failed", "rd-3")["error"] == "deploy-service.sh exited 1"
|
||||
|
||||
|
||||
def test_redeploy_uses_its_own_longer_timeout(tmp_path, monkeypatch):
|
||||
"""A redeploy must not be timed out on the container_restart budget: the
|
||||
runner polls once a minute and may pull images before reporting."""
|
||||
monkeypatch.setattr(executor_module, "ACTION_TIMEOUT_SECS", 300)
|
||||
monkeypatch.setattr(executor_module, "REDEPLOY_TIMEOUT_SECS", 900)
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-4"))
|
||||
|
||||
running_path = tmp_path / "actions" / "running" / "rd-4.json"
|
||||
data = json.loads(running_path.read_text())
|
||||
data["started_at"] = time.time() - 600 # past 300s, inside 900s
|
||||
running_path.write_text(json.dumps(data))
|
||||
|
||||
ex._reconcile_running_actions()
|
||||
assert _exists(tmp_path, "running", "rd-4")
|
||||
|
||||
data["started_at"] = time.time() - 1000 # past 900s
|
||||
running_path.write_text(json.dumps(data))
|
||||
|
||||
ex._reconcile_running_actions()
|
||||
assert _exists(tmp_path, "failed", "rd-4")
|
||||
error = _read(tmp_path, "failed", "rd-4")["error"]
|
||||
assert "Timed out after 900s" in error
|
||||
assert "deploy runner" in error
|
||||
|
||||
|
||||
def test_stale_result_from_previous_run_does_not_resolve(tmp_path, monkeypatch):
|
||||
ex = _setup_executor(tmp_path, monkeypatch)
|
||||
_write_action_result_event(tmp_path, "piha", "rd-5", success=True,
|
||||
ts=int(time.time()) - 86_400)
|
||||
|
||||
ex._execute_action(_write_approved(tmp_path, "rd-5"))
|
||||
ex._reconcile_running_actions()
|
||||
|
||||
assert _exists(tmp_path, "running", "rd-5")
|
||||
Loading…
Reference in a new issue