Compare commits

...

8 commits

Author SHA1 Message Date
oskar 531117c2ba docs(backlog): follow-upy z etapu 0 — 40-register.sh stary schemat topologii, komentarz mqtt_unreachable w observerze
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:56 +02:00
oskar 24a5fb4f53 docs(claude): drop mqtt_unreachable row from supervisor routing table
Follows the supervisor change: mqtt_unreachable never becomes an
incident (recon D15), so the documented container_restart routing was
false. Table now matches code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar c7fb73f351 docs(architecture): ai-cluster decommission decision (legacy, not migrated)
Bus codex/* dead since 2026-06-09 (recon C9). Records what survives into
subsystem B (patterns only: task bus, worker roles, enforced command
allowlists, preview mode), that task/ai-cluster-solaria stays unmerged
as documentation, and that the vps containers get stopped in a separate
supervised session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar da4e989930 fix(supervisor): drop dead mqtt_unreachable restart trigger
The observer never creates incidents with trigger_type=mqtt_unreachable,
so the CONTAINER_RESTART_TRIGGERS branch for it could never fire (recon
D15). stability-agent keeps emitting the event; it just never becomes an
incident. Tests: 145 passed.

SERVICE_NAMES in ai-cluster's service_ops_worker.py (the other dead
constant from the plan) is NOT touched: that code is legacy-frozen in
the unmerged task/ai-cluster-solaria worktree and nothing on this branch
references it (verified by grep — only the recon and plan docs mention it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar 6086b65423 chore(deploy): remove dead deploy-role.sh
The script composes from roles/$ROLE/compose/docker-compose.yml, but no
roles/ directory exists anywhere in the repo (recon B7) and nothing
references the script. Dead since the roles concept was abandoned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar 5a338dc1eb fix(hosts): align manifests with observed reality (recon B6/B7/C8/F20.5)
- hosts/solaria/services.yaml: add stability-agent and node_exporter
  (running on solaria since deployment, absent from desired state —
  recon B6); add planner-agent with monitor:false (legacy ai-cluster
  family, decision pending, do not remediate)
- services/stability-agent/service.yaml: owner_node chelsty (not a real
  node) -> per-host, same convention as node_exporter/ha-diag-agent; it
  runs on vps, piha, solaria (F20.5)
- services/mosquitto/: mark NOT DEPLOYED / legacy in service.yaml and
  README — the manifest matches nothing that runs (vps broker = legacy
  ai-cluster stack, piha broker = host systemd package, chelsty has its
  own runtime config; recon C8). Kept until the MQTT topology decision
  (recon open question 7); not deleted yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar 6f64bd4eb9 feat(topology): node status active|dormant + dormant handling in control plane
topology.yaml (per its own rule that hosts/*/services.yaml are
authoritative, recon F20.1):
- add status field; chelsty-infra + chelsty-ha -> dormant (site hardware
  down since ~2026-06-01, revival planned)
- add lustro as a full active node (runs node-agent, ships events, F20.11)
- drop per-node service lists (vps list contradicted hosts/vps, F20.1;
  piha/solaria lists were stale too, F20.8/F20.9) — node-level truth only
- deployment.mode pull -> push: every deploy script SSH-pushes from
  saturn (F20.10)

Dormant semantics in code:
- observer (scripts/observer/observer.py): loads status from topology;
  _prune_stale_world skips dormant nodes — last-known world state stays
  frozen, no node_offline/node_stale/node_online events emitted
- supervisor (services/control-plane/src/supervisor.py): reloads dormant
  set each reconcile; dormant hosts' services excluded from desired state
  (existing pending actions auto-cancel via
  service_removed_from_desired_state), disk_cleanup skipped, node/HA
  events from dormant nodes not routed to alerts

Tests: services/control-plane/tests/test_dormant_nodes.py (9 cases);
full suite 145 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
oskar 4c201b4783 docs(architecture): ARCHITEKTURA.md — standing decisions 2026-07-28
One page of binding architecture decisions per the 2026-07-27 recon:
two subsystems (A keep-it-running / B do-the-work), node roles incl.
solaria's by-design duty cycle and dormant chelsty site, ai-cluster
declared legacy (bus dead since 2026-06-09), approvals stay HITL with
Telegram as the etap-2 channel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 19:00:32 +02:00
13 changed files with 445 additions and 73 deletions

View file

@ -108,7 +108,6 @@ Normalized event types: `deployment_started/completed/failed`, `service_unhealth
| Event type | Source | Action generated | Cooldown |
|---|---|---|---|
| `containers_not_running` | stability-agent | `container_restart` | dedup via stable ID |
| `mqtt_unreachable` | stability-agent | `container_restart` | dedup via stable ID |
| `service_unhealthy` / other | stability-agent | `redeploy` | dedup via stable ID |
| `disk_pressure` (high) | stability-agent | `disk_cleanup` | dedup via stable ID |
| `ha_websocket_dead` | ha-diag-agent | `container_restart` (homeassistant) | 30 min after completion |

View file

@ -0,0 +1,42 @@
# Architektura — decyzje obowiązujące
Stan decyzji na 2026-07-28. Podstawa dowodowa:
[RECON-multiagent-2026-07-27.md](RECON-multiagent-2026-07-27.md); plan wykonawczy:
[PLAN-subsystem-a-2026-07-28.md](PLAN-subsystem-a-2026-07-28.md). Zmiana którejkolwiek
decyzji wymaga aktualizacji tego pliku z nową datą.
## Dwa subsystemy (2026-07-28)
- **A — keep-it-running**: control-plane (observer, supervisor, executor, operator-ui),
node-agenty, stability-agenty, self-healing floty. **To jest fokus tego repo.**
- **B — do-the-work**: dyspozytor zadań (agent) + nogi KB / Home Assistant /
homelab-ops. Osobny wysiłek, osobny projekt. Z ai-clustra dziedziczy **wzorce**
(bus zadań, role workerów, allowlisty komend) — **nie kod**.
## Role node'ów (2026-07-28)
| Node | Rola | Status |
|---|---|---|
| **vps** | ingress publiczny + control-plane, nic więcej (4 GB RAM, publiczny) | active |
| **piha** | serwer domowy 24/7 — dom wszystkiego, co musi zawsze działać; przyszły dom dyspozytora subsystemu B | active |
| **solaria** | desktop obliczeniowy on-demand; wyłączany ~16 h/dobę **z założenia** — dobowe eventy offline/online to oczekiwany cykl pracy, nie awaria; nic wymagającego 24/7 nie może tu mieszkać | active |
| **lustro** | Raspberry Pi z MagicMirror; pełnoprawny monitorowany node (node-agent, eventy) | active |
| **chelsty-infra** | druga lokalizacja: hypervisor LTE, Zigbee/MQTT | dormant (sprzęt padł ~2026-06-01; reanimacja później) |
| **chelsty-ha** | druga lokalizacja: VM Home Assistant | dormant (jak wyżej) |
Node'y `dormant`: observer trzyma ostatni znany stan, ale nie emituje eventów
liveness; supervisor nie generuje dla nich żadnych akcji.
## ai-cluster = LEGACY (2026-07-28)
Stack ai-cluster na vps (openclaw, codex-worker, planner-worker, service-ops-worker,
redis, mosquitto) jest **wygaszany, nie migrowany**. Bus `codex/*` martwy od
2026-06-09 (zero nowych połączeń). Branch `task/ai-cluster-solaria` zostaje
**niezmergowany** — pełni rolę dokumentacji. Kontenery na vps zostaną zatrzymane w
osobnej, nadzorowanej sesji. Szczegóły: [ai-cluster-LEGACY.md](ai-cluster-LEGACY.md).
## Approvale zostają HITL (2026-07-28)
Human-in-the-loop obowiązuje. Restarty kontenerów mogą w przyszłości przejść na
autonomię, ale `redeploy` i `disk_cleanup` zawsze wymagają zgody człowieka.
Kanałem powiadomień i approvali będzie **Telegram** (etap 2 planu).

View file

@ -0,0 +1,43 @@
# ai-cluster — LEGACY, wygaszany (decyzja 2026-07-28)
## Decyzja
Stack **ai-cluster** działający na vps (`ai-cluster-openclaw-1`, `codex-worker`,
`planner-worker`, `service-ops-worker`, `redis`, `mosquitto`) jest **wygaszany,
nie migrowany**. Podstawa (recon
[RECON-multiagent-2026-07-27.md](RECON-multiagent-2026-07-27.md), C9):
bus `codex/*` jest martwy od **2026-06-09** — zero nowych połączeń przez ~7 tygodni,
workery trzymają tylko puste długożyjące połączenia.
Konsekwencje:
- **Migracja na solarię NIE wchodzi.** Branch `task/ai-cluster-solaria`
(@ `b124e54`) zostaje **niezmergowany** — pełni rolę dokumentacji stanu prac
i reconu; nie kasować.
- ai-cluster nie ma katalogu w `services/` na masterze i **nie dostanie go**
nie wciągamy legacy do GitOps.
- `planner-agent` na solarii (ta sama rodzina) ma w
`hosts/solaria/services.yaml` `monitor: false` — decyzja o jego losie osobno.
- Kontenery na vps **zostaną zatrzymane w osobnej, nadzorowanej sesji**
(stop stacka, obserwacja `free -m`, rollback = start) — poza tym taskiem;
do tego czasu działają dalej pod hard mem_limitami.
## Co przeżywa w subsystemie B (wzorce, nie kod)
Subsystem B (dyspozytor + KB + HA + homelab-ops; osobny projekt) dziedziczy
z ai-clustra **wzorce projektowe**:
- bus zadań z routingiem po polu `target` (`role:*` / `AGENT_ID`) i wąskim ACL,
- podział na wyspecjalizowane workery-role (dev / planner / service-ops),
- allowlisty kształtu komend przed wykonaniem czegokolwiek na hoście — z lekcją
z reconu (A2): allowlist musi być **egzekwowana i celowana** (martwa stała
`SERVICE_NAMES` + substring-match po `docker ps` to antywzorzec),
- tryb preview/diagnose przed wykonaniem.
Kod ai-clustra **nie jest** przenoszony.
## Czego NIE robić
- Nie deployować, nie restartować i nie "naprawiać" stacka ai-cluster na vps.
- Nie mergować `task/ai-cluster-solaria`.
- Nie podpinać nowych klientów pod `codex/*` na brokerze ai-clustra.

View file

@ -1207,3 +1207,16 @@ Pomiar throughput GPU vs CPU baseline (0.79s/chunk) — patrz
cicho no-opuje się przez istniejący constraint. Schema change do zrobienia
przy fazie 3 (patrz `jobs/documents-ingest/README.md`, sekcja "Idempotency"
kroku 6 embed).
## Follow-upy z etapu 0 (truth cleanup, 2026-07-29)
**Źródło**: sesja merge `task/etap0-truth` (topologia: status active|dormant,
listy serwisów usunięte z topology.yaml — node-level truth only).
- **40-register.sh emituje stary schemat topologii** — szablon bloku node'a
w `scripts/onboard/steps/40-register.sh` nadal zawiera listę `services:`
i nie ma pola `status:`; wyrównać z nowym schematem node-level-only
(`inventory/topology.yaml`).
- **Historyczny komentarz mqtt_unreachable w observerze** — `scripts/observer/
observer.py:866` wspomina routing `mqtt_unreachable -> container_restart`
usunięty z supervisora (recon D15); sprzątnąć przy najbliższej edycji pliku.

View file

@ -14,6 +14,38 @@ services:
data_path: /opt/homelab/state
logs_path: /opt/homelab/events
stability-agent:
role: node-watchdog # read-only docker.sock watchdog, emits filesystem events
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
runtime:
data_path: /opt/homelab/state
logs_path: /opt/homelab/events
node_exporter:
role: metrics-exporter
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
planner-agent:
# legacy ai-cluster family — decision pending, do not remediate
monitor: false
role: ai-planner
deployment_model: docker-compose
exposure: local-only
offline_required: false
depends_on:
local: []
external: []
ollama:
role: llm-inference # GPU-backed inference (RTX 4070 Ti SUPER, driver restored 2026-07-16): embeddings (bge-m3) + coder models
deployment_model: docker-compose

View file

@ -3,7 +3,9 @@ topology:
git_provider: forgejo
deployment:
mode: pull
# Every deploy script SSH-pushes from saturn to the target node (recon
# F20.10); "pull" never matched reality.
mode: push
orchestrator: saturn
# Ingress dla usług domowych przez NPM @ PIHA + wildcard cert kapala.org.
@ -31,54 +33,56 @@ ingress:
access: tailscale
cert: wildcard-kapala
# Node-level truth ONLY. Per-node service lists live in hosts/<node>/services.yaml
# (authoritative — the lists formerly duplicated here contradicted them, recon F20.1).
#
# status:
# active — node is monitored and remediated normally.
# dormant — node exists but is expected offline: the observer keeps its
# last-known world state and emits NO liveness events for it; the
# supervisor generates NO actions for it. See ARCHITEKTURA.md.
nodes:
saturn:
status: active
roles:
- control
- development
piha:
status: active
roles:
- infra
- monitoring
services:
- node-agent
- ha-diag-agent
- brain-watchdog
- vikunja # Task management (vikunja + postgres), public via npm
- kb-postgres # KB spine: Postgres 16 + pgvector, port 5433 (always-on)
- llm-gateway # FastAPI router -> Ollama @ SOLARIA (Tailscale-internal :8080)
- node_exporter # per-host (also vps) — textfile collector for kb-ingest metrics
solaria:
# Powered off ~16 h/day BY DESIGN (on-demand compute desktop): one daily
# offline/online liveness cycle is expected, not a fault. Nothing that
# must run 24/7 may live here.
status: active
roles:
- compute
- ai
services:
- node-agent
vps:
status: active
roles:
- edge
- ingress
- control-plane
services:
# Repo-managed GitOps services (hosts/vps/services.yaml is authoritative)
- node-agent
- control-plane # executor, observer, supervisor, operator-ui
- node_exporter
- fleet-prometheus # Fleet liveness source of truth (Tailscale-internal)
- stability-agent
- npm # Nginx Proxy Manager — public ingress, TLS termination
- outline # Team wiki (outline + postgres + redis)
- joplin # Note sync server (joplin-server + postgres)
- ai-cluster # AI workers: codex-worker, openclaw, planner-worker,
# service-ops-worker, redis, mosquitto
- gokapi # Public file-share (Firefox Send alt), share.okit.pl via npm@VPS,
# local disk storage, E2E encryption ON — separate from private Nextcloud
lustro:
# MagicMirror Raspberry Pi — full monitored node: runs node-agent and
# ships events to the control plane (recon F20.11).
status: active
roles:
- edge
- display
chelsty-infra:
status: dormant
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
dormant_since: "2026-06-01"
site: chelsty
roles:
- remote
@ -90,15 +94,15 @@ nodes:
intermittent: true
home_automation:
offline_operation_required: true
services:
- zigbee2mqtt
- mosquitto
coordinator:
model: SLZB-06U
connection: network
usb: false
chelsty-ha:
status: dormant
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
dormant_since: "2026-06-01"
site: chelsty
roles:
- remote
@ -108,11 +112,3 @@ nodes:
intermittent: true
home_automation:
offline_operation_required: true
services:
- homeassistant
lustro:
roles:
- edge
services:
- node-agent

View file

@ -1,21 +0,0 @@
#!/usr/bin/env bash
set -e
ROLE=$1
HOST=$2
if [ -z "$ROLE" ] || [ -z "$HOST" ]; then
echo "Usage:"
echo "./deploy-role.sh <role> <host>"
exit 1
fi
echo "Deploying role: $ROLE to host: $HOST"
ssh $HOST "
mkdir -p ~/homelab-codex-ws &&
cd ~/homelab-codex-ws &&
git pull &&
docker compose -f roles/$ROLE/compose/docker-compose.yml up -d
"

View file

@ -235,6 +235,18 @@ class Observer:
}
}
self.inventory = self._load_inventory()
# Nodes declared `status: dormant` in inventory/topology.yaml: expected
# offline (e.g. chelsty site, hardware down since ~2026-06-01). The
# observer keeps their last-known world state but never reclassifies
# liveness for them, so no node_offline/node_stale/node_online events
# are emitted while a node is dormant. See docs/architecture/ARCHITEKTURA.md.
self.dormant_nodes = {
name for name, info in self.inventory["nodes"].items()
if info.get("status") == "dormant"
}
if self.dormant_nodes:
logger.info("Dormant nodes (no liveness tracking): %s",
sorted(self.dormant_nodes))
self._ensure_dirs()
self._load_checkpoint()
# Persistent SHADOW_LIVENESS_MISMATCH sink (survives container recreate).
@ -274,9 +286,12 @@ class Observer:
with open(INVENTORY_TOPOLOGY, "r") as f:
topo = yaml.safe_load(f)
for node_name, node_info in topo.get("nodes", {}).items():
node_info = node_info or {}
inventory["nodes"][node_name] = {
"roles": node_info.get("roles", []),
"connectivity": node_info.get("connectivity", {})
"connectivity": node_info.get("connectivity", {}),
# topology node status: active (default) | dormant
"status": node_info.get("status", "active"),
}
# Load service assignments from hosts files
@ -558,6 +573,9 @@ class Observer:
# when disabled/unreachable — fail-open, event liveness is authoritative.
prom_liveness_map = self._query_prometheus_liveness()
for node_name, node_info in self.world_state["nodes"].items():
if node_name in self.dormant_nodes:
# Dormant node: keep last-known state frozen, emit nothing.
continue
roles = (node_info.get("roles")
or self.inventory["nodes"].get(node_name, {}).get("roles", []))
liveness = compute_liveness(

View file

@ -33,10 +33,13 @@ try:
except Exception:
NODE_ALIAS_MAP = {}
# Event trigger types that should result in a lightweight container_restart
# rather than a full redeploy. The container is present but not running,
# or a dependency (MQTT) is unreachable — a restart is the right first step.
CONTAINER_RESTART_TRIGGERS = {"containers_not_running", "mqtt_unreachable"}
# Incident trigger types that should result in a lightweight container_restart
# rather than a full redeploy: the container is present but not running.
# mqtt_unreachable was removed 2026-07-28: the observer never creates incidents
# with that trigger_type, so the branch was dead code (recon
# docs/architecture/RECON-multiagent-2026-07-27.md, D15). stability-agent still
# emits the mqtt_unreachable *event*; it just never becomes an incident.
CONTAINER_RESTART_TRIGGERS = {"containers_not_running"}
# Nodes where automatic disk_cleanup actions must NOT be generated.
# On chelsty nodes disk fullness is overwhelmingly caused by Frigate recordings
@ -129,6 +132,13 @@ class Supervisor:
def __init__(self):
self.desired_state = {"services": {}}
self.actual_state = {"services": {}, "nodes": {}, "incidents": {}}
# Nodes declared `status: dormant` in inventory/topology.yaml (e.g. the
# chelsty site, hardware down since ~2026-06-01). The supervisor
# generates NO actions of any kind for them: their services are excluded
# from desired state (which also auto-cancels their stale pending
# actions), and node/HA events from them are not routed to alerts.
# See docs/architecture/ARCHITEKTURA.md.
self.dormant_nodes: set = set()
# In-memory set of already-routed HA event IDs; prevents re-processing
# on each reconcile cycle. Grows to at most ~hundreds of entries/day.
self._ha_processed_event_ids: set = set()
@ -179,6 +189,21 @@ class Supervisor:
# State loading
# ------------------------------------------------------------------
def _load_dormant_nodes(self):
"""Refresh the set of dormant nodes from inventory/topology.yaml."""
dormant = set()
topo_file = REPO_ROOT / "inventory" / "topology.yaml"
try:
if topo_file.exists():
with open(topo_file, "r") as f:
topo = yaml.safe_load(f) or {}
for name, info in (topo.get("nodes") or {}).items():
if (info or {}).get("status") == "dormant":
dormant.add(name)
except Exception as e:
logger.error(f"Failed to load dormant nodes from {topo_file}: {e}")
self.dormant_nodes = dormant
def _load_desired_state(self):
services = {}
hosts_dir = REPO_ROOT / "hosts"
@ -194,6 +219,11 @@ class Supervisor:
with open(svc_file, "r") as f:
data = yaml.safe_load(f)
host_name = data.get("host")
if host_name in self.dormant_nodes:
logger.debug(
f"Skipping desired services of {host_name}: node dormant"
)
continue
for svc_name, svc_info in data.get("services", {}).items():
svc_info = svc_info or {}
# monitor: false — service is documented as desired but
@ -276,7 +306,7 @@ class Supervisor:
"""
Return the trigger_type of the active incident for a service, or None.
trigger_type is set by the observer when it creates an incident from
a specific event type (e.g. 'containers_not_running', 'mqtt_unreachable').
a specific event type (e.g. 'containers_not_running').
"""
svc_info = self.actual_state["services"].get(svc_key, {})
incident_id = svc_info.get("incident_id")
@ -299,6 +329,7 @@ class Supervisor:
except Exception as e:
logger.error(f"Failed to touch heartbeat file: {e}")
self._load_dormant_nodes()
self._load_desired_state()
if not self._load_actual_state():
return # world state unreadable this cycle — skip to avoid false drift
@ -334,7 +365,7 @@ class Supervisor:
# 3. Generate node-level recommendations (disk pressure)
for node_name, node_info in self.actual_state["nodes"].items():
if node_name in NO_DISK_CLEANUP_NODES:
if node_name in NO_DISK_CLEANUP_NODES or node_name in self.dormant_nodes:
continue
if node_info.get("disk_pressure") == "high":
self._generate_disk_cleanup_recommendation(node_name)
@ -381,8 +412,7 @@ class Supervisor:
if trigger_type in CONTAINER_RESTART_TRIGGERS:
# Lightweight remediation: the container exists but is not running
# (containers_not_running) or its MQTT dependency is unreachable
# (mqtt_unreachable). A docker restart is sufficient and low-risk.
# (containers_not_running). A docker restart is sufficient and low-risk.
container_name = self._get_container_name(service)
action = {
"action_id": action_id,
@ -582,6 +612,9 @@ class Supervisor:
node = event.get("node", "")
if not node:
return
if node in self.dormant_nodes:
logger.debug(f"Suppressing {event_type} on {node}: node dormant")
return
if event_type in HA_CONTAINER_RESTART_EVENTS:
if self._is_ha_in_transition(node):
@ -772,6 +805,9 @@ class Supervisor:
node = event.get("node") or event.get("payload", {}).get("affected_node")
if not node:
return
if node in self.dormant_nodes:
logger.debug(f"Suppressing {event_type} on {node}: node dormant")
return
action_id = f"alert-{event_type.replace('_', '-')}-{node}"

View file

@ -0,0 +1,197 @@
"""Dormant-node handling (topology `status: dormant`, etap 0 truth cleanup).
Contract (docs/architecture/ARCHITEKTURA.md): a dormant node keeps its
last-known world state, the observer emits NO liveness events for it, and the
supervisor generates NO actions of any kind for it.
"""
from __future__ import annotations
import json
import sys
import time
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "scripts"))
import observer.observer as obs_mod
from observer.observer import Observer
import supervisor as supervisor_module
from supervisor import Supervisor
TOPOLOGY_YAML = (
"nodes:\n"
" piha:\n"
" status: active\n"
" roles: [infra]\n"
" oldnode:\n"
" status: dormant\n"
" dormant_reason: \"hardware down\"\n"
" roles: [remote]\n"
)
# ---------------------------------------------------------------------------
# Observer
# ---------------------------------------------------------------------------
@pytest.fixture
def obs_paths(tmp_path, monkeypatch):
world = tmp_path / "world"
state = tmp_path / "state"
events = tmp_path / "events"
logs = tmp_path / "logs"
repo = tmp_path / "repo"
for d in (world, state, events, logs, repo / "inventory", repo / "hosts"):
d.mkdir(parents=True, exist_ok=True)
(repo / "inventory" / "topology.yaml").write_text(TOPOLOGY_YAML)
monkeypatch.setattr(obs_mod, "WORLD_DIR", world)
monkeypatch.setattr(obs_mod, "STATE_DIR", state)
monkeypatch.setattr(obs_mod, "EVENTS_DIR", events)
monkeypatch.setattr(obs_mod, "LOGS_DIR", logs)
monkeypatch.setattr(obs_mod, "INVENTORY_TOPOLOGY", repo / "inventory" / "topology.yaml")
monkeypatch.setattr(obs_mod, "REPO_ROOT", repo)
monkeypatch.setattr(obs_mod, "FAILED_EVENTS_DIR", state / "observer_failed_events")
monkeypatch.setattr(obs_mod, "OBSERVER_STATE_FILE", state / "observer_checkpoint.json")
monkeypatch.setattr(obs_mod, "PROM_SHADOW_URL", "")
monkeypatch.setattr(obs_mod, "SHADOW_LOG_DIR", logs / "observer")
return tmp_path
def test_observer_loads_dormant_set_from_topology(obs_paths):
obs = Observer()
assert obs.dormant_nodes == {"oldnode"}
def test_observer_freezes_dormant_node_and_emits_nothing(obs_paths):
"""A dormant node whose heartbeats stopped long ago must NOT be
reclassified (staledead would normally fire node_offline)."""
obs = Observer()
stale_age = int(time.time()) - 999999
obs.world_state["nodes"] = {
"oldnode": {"status": "stale", "liveness": "stale", "last_seen": stale_age},
"piha": {"status": "unknown", "last_seen": int(time.time())},
}
obs._prune_stale_world()
frozen = obs.world_state["nodes"]["oldnode"]
assert frozen["liveness"] == "stale" # unchanged, despite dead-level age
assert frozen["status"] == "stale" # unchanged
# No liveness event file was written for the dormant node.
assert not list((obs_paths / "events").glob("**/evt-oldnode-*"))
# The active node is still processed normally.
assert obs.world_state["nodes"]["piha"]["liveness"] == "fresh"
def test_observer_keeps_dormant_node_in_world_state(obs_paths):
"""Dormant nodes are in the inventory, so pruning must not remove them."""
obs = Observer()
obs.world_state["nodes"] = {"oldnode": {"status": "offline", "last_seen": 1}}
obs._prune_stale_world()
assert "oldnode" in obs.world_state["nodes"]
# ---------------------------------------------------------------------------
# Supervisor
# ---------------------------------------------------------------------------
@pytest.fixture
def sup(tmp_path, monkeypatch):
actions = tmp_path / "actions"
events = tmp_path / "events"
world = tmp_path / "world"
state = tmp_path / "state"
repo = tmp_path / "repo"
for d in (actions, events, world, state, repo / "inventory", repo / "hosts" / "oldnode"):
d.mkdir(parents=True, exist_ok=True)
(repo / "inventory" / "topology.yaml").write_text(TOPOLOGY_YAML)
(repo / "hosts" / "oldnode" / "services.yaml").write_text(
"host: oldnode\nservices:\n some-svc:\n role: x\n"
)
monkeypatch.setattr(supervisor_module, "ACTIONS_DIR", actions)
monkeypatch.setattr(supervisor_module, "EVENTS_DIR", events)
monkeypatch.setattr(supervisor_module, "WORLD_DIR", world)
monkeypatch.setattr(supervisor_module, "REPO_ROOT", repo)
s = Supervisor()
return s
def _pending(tmp_path):
return list((tmp_path / "actions" / "pending").glob("*.json"))
def test_supervisor_loads_dormant_set(sup):
sup._load_dormant_nodes()
assert sup.dormant_nodes == {"oldnode"}
def test_dormant_services_excluded_from_desired_state(sup):
sup._load_dormant_nodes()
sup._load_desired_state()
assert "oldnode/some-svc" not in sup.desired_state["services"]
def test_reconcile_generates_no_actions_for_dormant_node(sup, tmp_path):
"""Full reconcile: missing service + high disk pressure on a dormant node
must produce zero pending actions."""
world = tmp_path / "world"
(world / "nodes.json").write_text(json.dumps(
{"oldnode": {"status": "offline", "disk_pressure": "high"}}
))
(world / "services.json").write_text("{}")
(world / "incidents.json").write_text("{}")
sup.reconcile()
assert _pending(tmp_path) == []
def test_node_event_from_dormant_node_not_routed(sup, tmp_path):
sup._load_dormant_nodes()
event = {
"id": "evt-oldnode-1-node_offline-node",
"type": "node_offline",
"node": "oldnode",
"payload": {"affected_node": "oldnode"},
}
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
sup._process_ha_events()
assert _pending(tmp_path) == []
def test_ha_event_from_dormant_node_not_routed(sup, tmp_path):
sup._load_dormant_nodes()
event = {
"id": "evt-oldnode-2-ha_entity_unavailable_long-homeassistant",
"type": "ha_entity_unavailable_long",
"node": "oldnode",
"payload": {},
}
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
sup._process_ha_events()
assert _pending(tmp_path) == []
def test_active_node_events_still_routed(sup, tmp_path):
"""Sanity: the dormant guard must not swallow active-node events."""
sup._load_dormant_nodes()
event = {
"id": "evt-piha-3-node_stale-node",
"type": "node_stale",
"node": "piha",
"payload": {"affected_node": "piha"},
}
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
sup._process_ha_events()
assert len(_pending(tmp_path)) == 1

View file

@ -1,9 +1,15 @@
# Mosquitto MQTT Broker
# Mosquitto MQTT Broker — NOT DEPLOYED / LEGACY
> **Status (2026-07-28, truth cleanup):** this manifest matches **nothing that
> actually runs** (recon `docs/architecture/RECON-multiagent-2026-07-27.md`, C8).
> The broker on **vps** is part of the legacy ai-cluster stack (being
> decommissioned, see `docs/architecture/ai-cluster-LEGACY.md`); the broker on
> **piha** is a host systemd mosquitto (OS package) with no repo definition;
> **chelsty** has its own config under `hosts/chelsty-infra/runtime/mosquitto/`.
> Do not deploy from this directory. Kept until the MQTT topology decision
> (recon open question 7) is made.
Eclipse Mosquitto is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1.
## Usage
Deployed on the `piha` node.
Port 1883 for standard MQTT.
Port 9001 for WebSockets.

View file

@ -1,6 +1,13 @@
# NOT DEPLOYED — LEGACY MANIFEST (2026-07-28 truth cleanup, recon C8).
# This definition matches NOTHING that actually runs: the broker on vps belongs
# to the legacy ai-cluster stack (being decommissioned), and piha's broker is a
# host systemd mosquitto (OS package, no repo definition). The chelsty broker
# has its own config under hosts/chelsty-infra/runtime/mosquitto/. Kept for
# reference until the MQTT topology decision (recon open question 7) is made —
# do not deploy from this directory.
service:
name: mosquitto
owner_node: vps
owner_node: vps # WRONG in practice — see header; README used to say piha
exposure: private
dependencies: []
ports:

View file

@ -1,6 +1,10 @@
service:
name: stability-agent
owner_node: chelsty
# Deployed per-host: runs on vps, piha and solaria (and chelsty-infra once the
# site is revived). Was owner_node: chelsty — not a real node name; fixed during
# the 2026-07 truth cleanup (recon B7/F20.5), closing the docs/backlog.md
# "stability-agent / node_exporter owner_node single" item for stability-agent.
owner_node: per-host
exposure: private
dependencies: []
healthcheck: