Compare commits
8 commits
670cb71c99
...
531117c2ba
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
531117c2ba | ||
|
|
24a5fb4f53 | ||
|
|
c7fb73f351 | ||
|
|
da4e989930 | ||
|
|
6086b65423 | ||
|
|
5a338dc1eb | ||
|
|
6f64bd4eb9 | ||
|
|
4c201b4783 |
|
|
@ -108,7 +108,6 @@ Normalized event types: `deployment_started/completed/failed`, `service_unhealth
|
|||
| Event type | Source | Action generated | Cooldown |
|
||||
|---|---|---|---|
|
||||
| `containers_not_running` | stability-agent | `container_restart` | dedup via stable ID |
|
||||
| `mqtt_unreachable` | stability-agent | `container_restart` | dedup via stable ID |
|
||||
| `service_unhealthy` / other | stability-agent | `redeploy` | dedup via stable ID |
|
||||
| `disk_pressure` (high) | stability-agent | `disk_cleanup` | dedup via stable ID |
|
||||
| `ha_websocket_dead` | ha-diag-agent | `container_restart` (homeassistant) | 30 min after completion |
|
||||
|
|
|
|||
42
docs/architecture/ARCHITEKTURA.md
Normal file
42
docs/architecture/ARCHITEKTURA.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Architektura — decyzje obowiązujące
|
||||
|
||||
Stan decyzji na 2026-07-28. Podstawa dowodowa:
|
||||
[RECON-multiagent-2026-07-27.md](RECON-multiagent-2026-07-27.md); plan wykonawczy:
|
||||
[PLAN-subsystem-a-2026-07-28.md](PLAN-subsystem-a-2026-07-28.md). Zmiana którejkolwiek
|
||||
decyzji wymaga aktualizacji tego pliku z nową datą.
|
||||
|
||||
## Dwa subsystemy (2026-07-28)
|
||||
|
||||
- **A — keep-it-running**: control-plane (observer, supervisor, executor, operator-ui),
|
||||
node-agenty, stability-agenty, self-healing floty. **To jest fokus tego repo.**
|
||||
- **B — do-the-work**: dyspozytor zadań (agent) + nogi KB / Home Assistant /
|
||||
homelab-ops. Osobny wysiłek, osobny projekt. Z ai-clustra dziedziczy **wzorce**
|
||||
(bus zadań, role workerów, allowlisty komend) — **nie kod**.
|
||||
|
||||
## Role node'ów (2026-07-28)
|
||||
|
||||
| Node | Rola | Status |
|
||||
|---|---|---|
|
||||
| **vps** | ingress publiczny + control-plane, nic więcej (4 GB RAM, publiczny) | active |
|
||||
| **piha** | serwer domowy 24/7 — dom wszystkiego, co musi zawsze działać; przyszły dom dyspozytora subsystemu B | active |
|
||||
| **solaria** | desktop obliczeniowy on-demand; wyłączany ~16 h/dobę **z założenia** — dobowe eventy offline/online to oczekiwany cykl pracy, nie awaria; nic wymagającego 24/7 nie może tu mieszkać | active |
|
||||
| **lustro** | Raspberry Pi z MagicMirror; pełnoprawny monitorowany node (node-agent, eventy) | active |
|
||||
| **chelsty-infra** | druga lokalizacja: hypervisor LTE, Zigbee/MQTT | dormant (sprzęt padł ~2026-06-01; reanimacja później) |
|
||||
| **chelsty-ha** | druga lokalizacja: VM Home Assistant | dormant (jak wyżej) |
|
||||
|
||||
Node'y `dormant`: observer trzyma ostatni znany stan, ale nie emituje eventów
|
||||
liveness; supervisor nie generuje dla nich żadnych akcji.
|
||||
|
||||
## ai-cluster = LEGACY (2026-07-28)
|
||||
|
||||
Stack ai-cluster na vps (openclaw, codex-worker, planner-worker, service-ops-worker,
|
||||
redis, mosquitto) jest **wygaszany, nie migrowany**. Bus `codex/*` martwy od
|
||||
2026-06-09 (zero nowych połączeń). Branch `task/ai-cluster-solaria` zostaje
|
||||
**niezmergowany** — pełni rolę dokumentacji. Kontenery na vps zostaną zatrzymane w
|
||||
osobnej, nadzorowanej sesji. Szczegóły: [ai-cluster-LEGACY.md](ai-cluster-LEGACY.md).
|
||||
|
||||
## Approvale zostają HITL (2026-07-28)
|
||||
|
||||
Human-in-the-loop obowiązuje. Restarty kontenerów mogą w przyszłości przejść na
|
||||
autonomię, ale `redeploy` i `disk_cleanup` zawsze wymagają zgody człowieka.
|
||||
Kanałem powiadomień i approvali będzie **Telegram** (etap 2 planu).
|
||||
43
docs/architecture/ai-cluster-LEGACY.md
Normal file
43
docs/architecture/ai-cluster-LEGACY.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# ai-cluster — LEGACY, wygaszany (decyzja 2026-07-28)
|
||||
|
||||
## Decyzja
|
||||
|
||||
Stack **ai-cluster** działający na vps (`ai-cluster-openclaw-1`, `codex-worker`,
|
||||
`planner-worker`, `service-ops-worker`, `redis`, `mosquitto`) jest **wygaszany,
|
||||
nie migrowany**. Podstawa (recon
|
||||
[RECON-multiagent-2026-07-27.md](RECON-multiagent-2026-07-27.md), C9):
|
||||
bus `codex/*` jest martwy od **2026-06-09** — zero nowych połączeń przez ~7 tygodni,
|
||||
workery trzymają tylko puste długożyjące połączenia.
|
||||
|
||||
Konsekwencje:
|
||||
|
||||
- **Migracja na solarię NIE wchodzi.** Branch `task/ai-cluster-solaria`
|
||||
(@ `b124e54`) zostaje **niezmergowany** — pełni rolę dokumentacji stanu prac
|
||||
i reconu; nie kasować.
|
||||
- ai-cluster nie ma katalogu w `services/` na masterze i **nie dostanie go** —
|
||||
nie wciągamy legacy do GitOps.
|
||||
- `planner-agent` na solarii (ta sama rodzina) ma w
|
||||
`hosts/solaria/services.yaml` `monitor: false` — decyzja o jego losie osobno.
|
||||
- Kontenery na vps **zostaną zatrzymane w osobnej, nadzorowanej sesji**
|
||||
(stop stacka, obserwacja `free -m`, rollback = start) — poza tym taskiem;
|
||||
do tego czasu działają dalej pod hard mem_limitami.
|
||||
|
||||
## Co przeżywa w subsystemie B (wzorce, nie kod)
|
||||
|
||||
Subsystem B (dyspozytor + KB + HA + homelab-ops; osobny projekt) dziedziczy
|
||||
z ai-clustra **wzorce projektowe**:
|
||||
|
||||
- bus zadań z routingiem po polu `target` (`role:*` / `AGENT_ID`) i wąskim ACL,
|
||||
- podział na wyspecjalizowane workery-role (dev / planner / service-ops),
|
||||
- allowlisty kształtu komend przed wykonaniem czegokolwiek na hoście — z lekcją
|
||||
z reconu (A2): allowlist musi być **egzekwowana i celowana** (martwa stała
|
||||
`SERVICE_NAMES` + substring-match po `docker ps` to antywzorzec),
|
||||
- tryb preview/diagnose przed wykonaniem.
|
||||
|
||||
Kod ai-clustra **nie jest** przenoszony.
|
||||
|
||||
## Czego NIE robić
|
||||
|
||||
- Nie deployować, nie restartować i nie "naprawiać" stacka ai-cluster na vps.
|
||||
- Nie mergować `task/ai-cluster-solaria`.
|
||||
- Nie podpinać nowych klientów pod `codex/*` na brokerze ai-clustra.
|
||||
|
|
@ -1207,3 +1207,16 @@ Pomiar throughput GPU vs CPU baseline (0.79s/chunk) — patrz
|
|||
cicho no-opuje się przez istniejący constraint. Schema change do zrobienia
|
||||
przy fazie 3 (patrz `jobs/documents-ingest/README.md`, sekcja "Idempotency"
|
||||
kroku 6 embed).
|
||||
|
||||
## Follow-upy z etapu 0 (truth cleanup, 2026-07-29)
|
||||
|
||||
**Źródło**: sesja merge `task/etap0-truth` (topologia: status active|dormant,
|
||||
listy serwisów usunięte z topology.yaml — node-level truth only).
|
||||
|
||||
- **40-register.sh emituje stary schemat topologii** — szablon bloku node'a
|
||||
w `scripts/onboard/steps/40-register.sh` nadal zawiera listę `services:`
|
||||
i nie ma pola `status:`; wyrównać z nowym schematem node-level-only
|
||||
(`inventory/topology.yaml`).
|
||||
- **Historyczny komentarz mqtt_unreachable w observerze** — `scripts/observer/
|
||||
observer.py:866` wspomina routing `mqtt_unreachable -> container_restart`
|
||||
usunięty z supervisora (recon D15); sprzątnąć przy najbliższej edycji pliku.
|
||||
|
|
|
|||
|
|
@ -14,6 +14,38 @@ services:
|
|||
data_path: /opt/homelab/state
|
||||
logs_path: /opt/homelab/events
|
||||
|
||||
stability-agent:
|
||||
role: node-watchdog # read-only docker.sock watchdog, emits filesystem events
|
||||
deployment_model: docker-compose
|
||||
exposure: local-only
|
||||
offline_required: true
|
||||
depends_on:
|
||||
local: []
|
||||
external: []
|
||||
runtime:
|
||||
data_path: /opt/homelab/state
|
||||
logs_path: /opt/homelab/events
|
||||
|
||||
node_exporter:
|
||||
role: metrics-exporter
|
||||
deployment_model: docker-compose
|
||||
exposure: local-only
|
||||
offline_required: true
|
||||
depends_on:
|
||||
local: []
|
||||
external: []
|
||||
|
||||
planner-agent:
|
||||
# legacy ai-cluster family — decision pending, do not remediate
|
||||
monitor: false
|
||||
role: ai-planner
|
||||
deployment_model: docker-compose
|
||||
exposure: local-only
|
||||
offline_required: false
|
||||
depends_on:
|
||||
local: []
|
||||
external: []
|
||||
|
||||
ollama:
|
||||
role: llm-inference # GPU-backed inference (RTX 4070 Ti SUPER, driver restored 2026-07-16): embeddings (bge-m3) + coder models
|
||||
deployment_model: docker-compose
|
||||
|
|
|
|||
|
|
@ -3,7 +3,9 @@ topology:
|
|||
git_provider: forgejo
|
||||
|
||||
deployment:
|
||||
mode: pull
|
||||
# Every deploy script SSH-pushes from saturn to the target node (recon
|
||||
# F20.10); "pull" never matched reality.
|
||||
mode: push
|
||||
orchestrator: saturn
|
||||
|
||||
# Ingress dla usług domowych przez NPM @ PIHA + wildcard cert kapala.org.
|
||||
|
|
@ -31,54 +33,56 @@ ingress:
|
|||
access: tailscale
|
||||
cert: wildcard-kapala
|
||||
|
||||
# Node-level truth ONLY. Per-node service lists live in hosts/<node>/services.yaml
|
||||
# (authoritative — the lists formerly duplicated here contradicted them, recon F20.1).
|
||||
#
|
||||
# status:
|
||||
# active — node is monitored and remediated normally.
|
||||
# dormant — node exists but is expected offline: the observer keeps its
|
||||
# last-known world state and emits NO liveness events for it; the
|
||||
# supervisor generates NO actions for it. See ARCHITEKTURA.md.
|
||||
nodes:
|
||||
|
||||
saturn:
|
||||
status: active
|
||||
roles:
|
||||
- control
|
||||
- development
|
||||
|
||||
piha:
|
||||
status: active
|
||||
roles:
|
||||
- infra
|
||||
- monitoring
|
||||
services:
|
||||
- node-agent
|
||||
- ha-diag-agent
|
||||
- brain-watchdog
|
||||
- vikunja # Task management (vikunja + postgres), public via npm
|
||||
- kb-postgres # KB spine: Postgres 16 + pgvector, port 5433 (always-on)
|
||||
- llm-gateway # FastAPI router -> Ollama @ SOLARIA (Tailscale-internal :8080)
|
||||
- node_exporter # per-host (also vps) — textfile collector for kb-ingest metrics
|
||||
|
||||
solaria:
|
||||
# Powered off ~16 h/day BY DESIGN (on-demand compute desktop): one daily
|
||||
# offline/online liveness cycle is expected, not a fault. Nothing that
|
||||
# must run 24/7 may live here.
|
||||
status: active
|
||||
roles:
|
||||
- compute
|
||||
- ai
|
||||
services:
|
||||
- node-agent
|
||||
|
||||
vps:
|
||||
status: active
|
||||
roles:
|
||||
- edge
|
||||
- ingress
|
||||
- control-plane
|
||||
services:
|
||||
# Repo-managed GitOps services (hosts/vps/services.yaml is authoritative)
|
||||
- node-agent
|
||||
- control-plane # executor, observer, supervisor, operator-ui
|
||||
- node_exporter
|
||||
- fleet-prometheus # Fleet liveness source of truth (Tailscale-internal)
|
||||
- stability-agent
|
||||
- npm # Nginx Proxy Manager — public ingress, TLS termination
|
||||
- outline # Team wiki (outline + postgres + redis)
|
||||
- joplin # Note sync server (joplin-server + postgres)
|
||||
- ai-cluster # AI workers: codex-worker, openclaw, planner-worker,
|
||||
# service-ops-worker, redis, mosquitto
|
||||
- gokapi # Public file-share (Firefox Send alt), share.okit.pl via npm@VPS,
|
||||
# local disk storage, E2E encryption ON — separate from private Nextcloud
|
||||
|
||||
lustro:
|
||||
# MagicMirror Raspberry Pi — full monitored node: runs node-agent and
|
||||
# ships events to the control plane (recon F20.11).
|
||||
status: active
|
||||
roles:
|
||||
- edge
|
||||
- display
|
||||
|
||||
chelsty-infra:
|
||||
status: dormant
|
||||
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
||||
dormant_since: "2026-06-01"
|
||||
site: chelsty
|
||||
roles:
|
||||
- remote
|
||||
|
|
@ -90,15 +94,15 @@ nodes:
|
|||
intermittent: true
|
||||
home_automation:
|
||||
offline_operation_required: true
|
||||
services:
|
||||
- zigbee2mqtt
|
||||
- mosquitto
|
||||
coordinator:
|
||||
model: SLZB-06U
|
||||
connection: network
|
||||
usb: false
|
||||
|
||||
chelsty-ha:
|
||||
status: dormant
|
||||
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
||||
dormant_since: "2026-06-01"
|
||||
site: chelsty
|
||||
roles:
|
||||
- remote
|
||||
|
|
@ -108,11 +112,3 @@ nodes:
|
|||
intermittent: true
|
||||
home_automation:
|
||||
offline_operation_required: true
|
||||
services:
|
||||
- homeassistant
|
||||
|
||||
lustro:
|
||||
roles:
|
||||
- edge
|
||||
services:
|
||||
- node-agent
|
||||
|
|
|
|||
|
|
@ -1,21 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
ROLE=$1
|
||||
HOST=$2
|
||||
|
||||
if [ -z "$ROLE" ] || [ -z "$HOST" ]; then
|
||||
echo "Usage:"
|
||||
echo "./deploy-role.sh <role> <host>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Deploying role: $ROLE to host: $HOST"
|
||||
|
||||
ssh $HOST "
|
||||
mkdir -p ~/homelab-codex-ws &&
|
||||
cd ~/homelab-codex-ws &&
|
||||
git pull &&
|
||||
docker compose -f roles/$ROLE/compose/docker-compose.yml up -d
|
||||
"
|
||||
|
|
@ -235,6 +235,18 @@ class Observer:
|
|||
}
|
||||
}
|
||||
self.inventory = self._load_inventory()
|
||||
# Nodes declared `status: dormant` in inventory/topology.yaml: expected
|
||||
# offline (e.g. chelsty site, hardware down since ~2026-06-01). The
|
||||
# observer keeps their last-known world state but never reclassifies
|
||||
# liveness for them, so no node_offline/node_stale/node_online events
|
||||
# are emitted while a node is dormant. See docs/architecture/ARCHITEKTURA.md.
|
||||
self.dormant_nodes = {
|
||||
name for name, info in self.inventory["nodes"].items()
|
||||
if info.get("status") == "dormant"
|
||||
}
|
||||
if self.dormant_nodes:
|
||||
logger.info("Dormant nodes (no liveness tracking): %s",
|
||||
sorted(self.dormant_nodes))
|
||||
self._ensure_dirs()
|
||||
self._load_checkpoint()
|
||||
# Persistent SHADOW_LIVENESS_MISMATCH sink (survives container recreate).
|
||||
|
|
@ -274,9 +286,12 @@ class Observer:
|
|||
with open(INVENTORY_TOPOLOGY, "r") as f:
|
||||
topo = yaml.safe_load(f)
|
||||
for node_name, node_info in topo.get("nodes", {}).items():
|
||||
node_info = node_info or {}
|
||||
inventory["nodes"][node_name] = {
|
||||
"roles": node_info.get("roles", []),
|
||||
"connectivity": node_info.get("connectivity", {})
|
||||
"connectivity": node_info.get("connectivity", {}),
|
||||
# topology node status: active (default) | dormant
|
||||
"status": node_info.get("status", "active"),
|
||||
}
|
||||
|
||||
# Load service assignments from hosts files
|
||||
|
|
@ -558,6 +573,9 @@ class Observer:
|
|||
# when disabled/unreachable — fail-open, event liveness is authoritative.
|
||||
prom_liveness_map = self._query_prometheus_liveness()
|
||||
for node_name, node_info in self.world_state["nodes"].items():
|
||||
if node_name in self.dormant_nodes:
|
||||
# Dormant node: keep last-known state frozen, emit nothing.
|
||||
continue
|
||||
roles = (node_info.get("roles")
|
||||
or self.inventory["nodes"].get(node_name, {}).get("roles", []))
|
||||
liveness = compute_liveness(
|
||||
|
|
|
|||
|
|
@ -33,10 +33,13 @@ try:
|
|||
except Exception:
|
||||
NODE_ALIAS_MAP = {}
|
||||
|
||||
# Event trigger types that should result in a lightweight container_restart
|
||||
# rather than a full redeploy. The container is present but not running,
|
||||
# or a dependency (MQTT) is unreachable — a restart is the right first step.
|
||||
CONTAINER_RESTART_TRIGGERS = {"containers_not_running", "mqtt_unreachable"}
|
||||
# Incident trigger types that should result in a lightweight container_restart
|
||||
# rather than a full redeploy: the container is present but not running.
|
||||
# mqtt_unreachable was removed 2026-07-28: the observer never creates incidents
|
||||
# with that trigger_type, so the branch was dead code (recon
|
||||
# docs/architecture/RECON-multiagent-2026-07-27.md, D15). stability-agent still
|
||||
# emits the mqtt_unreachable *event*; it just never becomes an incident.
|
||||
CONTAINER_RESTART_TRIGGERS = {"containers_not_running"}
|
||||
|
||||
# Nodes where automatic disk_cleanup actions must NOT be generated.
|
||||
# On chelsty nodes disk fullness is overwhelmingly caused by Frigate recordings
|
||||
|
|
@ -129,6 +132,13 @@ class Supervisor:
|
|||
def __init__(self):
|
||||
self.desired_state = {"services": {}}
|
||||
self.actual_state = {"services": {}, "nodes": {}, "incidents": {}}
|
||||
# Nodes declared `status: dormant` in inventory/topology.yaml (e.g. the
|
||||
# chelsty site, hardware down since ~2026-06-01). The supervisor
|
||||
# generates NO actions of any kind for them: their services are excluded
|
||||
# from desired state (which also auto-cancels their stale pending
|
||||
# actions), and node/HA events from them are not routed to alerts.
|
||||
# See docs/architecture/ARCHITEKTURA.md.
|
||||
self.dormant_nodes: set = set()
|
||||
# In-memory set of already-routed HA event IDs; prevents re-processing
|
||||
# on each reconcile cycle. Grows to at most ~hundreds of entries/day.
|
||||
self._ha_processed_event_ids: set = set()
|
||||
|
|
@ -179,6 +189,21 @@ class Supervisor:
|
|||
# State loading
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _load_dormant_nodes(self):
|
||||
"""Refresh the set of dormant nodes from inventory/topology.yaml."""
|
||||
dormant = set()
|
||||
topo_file = REPO_ROOT / "inventory" / "topology.yaml"
|
||||
try:
|
||||
if topo_file.exists():
|
||||
with open(topo_file, "r") as f:
|
||||
topo = yaml.safe_load(f) or {}
|
||||
for name, info in (topo.get("nodes") or {}).items():
|
||||
if (info or {}).get("status") == "dormant":
|
||||
dormant.add(name)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load dormant nodes from {topo_file}: {e}")
|
||||
self.dormant_nodes = dormant
|
||||
|
||||
def _load_desired_state(self):
|
||||
services = {}
|
||||
hosts_dir = REPO_ROOT / "hosts"
|
||||
|
|
@ -194,6 +219,11 @@ class Supervisor:
|
|||
with open(svc_file, "r") as f:
|
||||
data = yaml.safe_load(f)
|
||||
host_name = data.get("host")
|
||||
if host_name in self.dormant_nodes:
|
||||
logger.debug(
|
||||
f"Skipping desired services of {host_name}: node dormant"
|
||||
)
|
||||
continue
|
||||
for svc_name, svc_info in data.get("services", {}).items():
|
||||
svc_info = svc_info or {}
|
||||
# monitor: false — service is documented as desired but
|
||||
|
|
@ -276,7 +306,7 @@ class Supervisor:
|
|||
"""
|
||||
Return the trigger_type of the active incident for a service, or None.
|
||||
trigger_type is set by the observer when it creates an incident from
|
||||
a specific event type (e.g. 'containers_not_running', 'mqtt_unreachable').
|
||||
a specific event type (e.g. 'containers_not_running').
|
||||
"""
|
||||
svc_info = self.actual_state["services"].get(svc_key, {})
|
||||
incident_id = svc_info.get("incident_id")
|
||||
|
|
@ -299,6 +329,7 @@ class Supervisor:
|
|||
except Exception as e:
|
||||
logger.error(f"Failed to touch heartbeat file: {e}")
|
||||
|
||||
self._load_dormant_nodes()
|
||||
self._load_desired_state()
|
||||
if not self._load_actual_state():
|
||||
return # world state unreadable this cycle — skip to avoid false drift
|
||||
|
|
@ -334,7 +365,7 @@ class Supervisor:
|
|||
|
||||
# 3. Generate node-level recommendations (disk pressure)
|
||||
for node_name, node_info in self.actual_state["nodes"].items():
|
||||
if node_name in NO_DISK_CLEANUP_NODES:
|
||||
if node_name in NO_DISK_CLEANUP_NODES or node_name in self.dormant_nodes:
|
||||
continue
|
||||
if node_info.get("disk_pressure") == "high":
|
||||
self._generate_disk_cleanup_recommendation(node_name)
|
||||
|
|
@ -381,8 +412,7 @@ class Supervisor:
|
|||
|
||||
if trigger_type in CONTAINER_RESTART_TRIGGERS:
|
||||
# Lightweight remediation: the container exists but is not running
|
||||
# (containers_not_running) or its MQTT dependency is unreachable
|
||||
# (mqtt_unreachable). A docker restart is sufficient and low-risk.
|
||||
# (containers_not_running). A docker restart is sufficient and low-risk.
|
||||
container_name = self._get_container_name(service)
|
||||
action = {
|
||||
"action_id": action_id,
|
||||
|
|
@ -582,6 +612,9 @@ class Supervisor:
|
|||
node = event.get("node", "")
|
||||
if not node:
|
||||
return
|
||||
if node in self.dormant_nodes:
|
||||
logger.debug(f"Suppressing {event_type} on {node}: node dormant")
|
||||
return
|
||||
|
||||
if event_type in HA_CONTAINER_RESTART_EVENTS:
|
||||
if self._is_ha_in_transition(node):
|
||||
|
|
@ -772,6 +805,9 @@ class Supervisor:
|
|||
node = event.get("node") or event.get("payload", {}).get("affected_node")
|
||||
if not node:
|
||||
return
|
||||
if node in self.dormant_nodes:
|
||||
logger.debug(f"Suppressing {event_type} on {node}: node dormant")
|
||||
return
|
||||
|
||||
action_id = f"alert-{event_type.replace('_', '-')}-{node}"
|
||||
|
||||
|
|
|
|||
197
services/control-plane/tests/test_dormant_nodes.py
Normal file
197
services/control-plane/tests/test_dormant_nodes.py
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
"""Dormant-node handling (topology `status: dormant`, etap 0 truth cleanup).
|
||||
|
||||
Contract (docs/architecture/ARCHITEKTURA.md): a dormant node keeps its
|
||||
last-known world state, the observer emits NO liveness events for it, and the
|
||||
supervisor generates NO actions of any kind for it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "scripts"))
|
||||
|
||||
import observer.observer as obs_mod
|
||||
from observer.observer import Observer
|
||||
import supervisor as supervisor_module
|
||||
from supervisor import Supervisor
|
||||
|
||||
TOPOLOGY_YAML = (
|
||||
"nodes:\n"
|
||||
" piha:\n"
|
||||
" status: active\n"
|
||||
" roles: [infra]\n"
|
||||
" oldnode:\n"
|
||||
" status: dormant\n"
|
||||
" dormant_reason: \"hardware down\"\n"
|
||||
" roles: [remote]\n"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Observer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def obs_paths(tmp_path, monkeypatch):
|
||||
world = tmp_path / "world"
|
||||
state = tmp_path / "state"
|
||||
events = tmp_path / "events"
|
||||
logs = tmp_path / "logs"
|
||||
repo = tmp_path / "repo"
|
||||
for d in (world, state, events, logs, repo / "inventory", repo / "hosts"):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
(repo / "inventory" / "topology.yaml").write_text(TOPOLOGY_YAML)
|
||||
monkeypatch.setattr(obs_mod, "WORLD_DIR", world)
|
||||
monkeypatch.setattr(obs_mod, "STATE_DIR", state)
|
||||
monkeypatch.setattr(obs_mod, "EVENTS_DIR", events)
|
||||
monkeypatch.setattr(obs_mod, "LOGS_DIR", logs)
|
||||
monkeypatch.setattr(obs_mod, "INVENTORY_TOPOLOGY", repo / "inventory" / "topology.yaml")
|
||||
monkeypatch.setattr(obs_mod, "REPO_ROOT", repo)
|
||||
monkeypatch.setattr(obs_mod, "FAILED_EVENTS_DIR", state / "observer_failed_events")
|
||||
monkeypatch.setattr(obs_mod, "OBSERVER_STATE_FILE", state / "observer_checkpoint.json")
|
||||
monkeypatch.setattr(obs_mod, "PROM_SHADOW_URL", "")
|
||||
monkeypatch.setattr(obs_mod, "SHADOW_LOG_DIR", logs / "observer")
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_observer_loads_dormant_set_from_topology(obs_paths):
|
||||
obs = Observer()
|
||||
assert obs.dormant_nodes == {"oldnode"}
|
||||
|
||||
|
||||
def test_observer_freezes_dormant_node_and_emits_nothing(obs_paths):
|
||||
"""A dormant node whose heartbeats stopped long ago must NOT be
|
||||
reclassified (stale→dead would normally fire node_offline)."""
|
||||
obs = Observer()
|
||||
stale_age = int(time.time()) - 999999
|
||||
obs.world_state["nodes"] = {
|
||||
"oldnode": {"status": "stale", "liveness": "stale", "last_seen": stale_age},
|
||||
"piha": {"status": "unknown", "last_seen": int(time.time())},
|
||||
}
|
||||
|
||||
obs._prune_stale_world()
|
||||
|
||||
frozen = obs.world_state["nodes"]["oldnode"]
|
||||
assert frozen["liveness"] == "stale" # unchanged, despite dead-level age
|
||||
assert frozen["status"] == "stale" # unchanged
|
||||
# No liveness event file was written for the dormant node.
|
||||
assert not list((obs_paths / "events").glob("**/evt-oldnode-*"))
|
||||
# The active node is still processed normally.
|
||||
assert obs.world_state["nodes"]["piha"]["liveness"] == "fresh"
|
||||
|
||||
|
||||
def test_observer_keeps_dormant_node_in_world_state(obs_paths):
|
||||
"""Dormant nodes are in the inventory, so pruning must not remove them."""
|
||||
obs = Observer()
|
||||
obs.world_state["nodes"] = {"oldnode": {"status": "offline", "last_seen": 1}}
|
||||
|
||||
obs._prune_stale_world()
|
||||
|
||||
assert "oldnode" in obs.world_state["nodes"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Supervisor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def sup(tmp_path, monkeypatch):
|
||||
actions = tmp_path / "actions"
|
||||
events = tmp_path / "events"
|
||||
world = tmp_path / "world"
|
||||
state = tmp_path / "state"
|
||||
repo = tmp_path / "repo"
|
||||
for d in (actions, events, world, state, repo / "inventory", repo / "hosts" / "oldnode"):
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
(repo / "inventory" / "topology.yaml").write_text(TOPOLOGY_YAML)
|
||||
(repo / "hosts" / "oldnode" / "services.yaml").write_text(
|
||||
"host: oldnode\nservices:\n some-svc:\n role: x\n"
|
||||
)
|
||||
monkeypatch.setattr(supervisor_module, "ACTIONS_DIR", actions)
|
||||
monkeypatch.setattr(supervisor_module, "EVENTS_DIR", events)
|
||||
monkeypatch.setattr(supervisor_module, "WORLD_DIR", world)
|
||||
monkeypatch.setattr(supervisor_module, "REPO_ROOT", repo)
|
||||
s = Supervisor()
|
||||
return s
|
||||
|
||||
|
||||
def _pending(tmp_path):
|
||||
return list((tmp_path / "actions" / "pending").glob("*.json"))
|
||||
|
||||
|
||||
def test_supervisor_loads_dormant_set(sup):
|
||||
sup._load_dormant_nodes()
|
||||
assert sup.dormant_nodes == {"oldnode"}
|
||||
|
||||
|
||||
def test_dormant_services_excluded_from_desired_state(sup):
|
||||
sup._load_dormant_nodes()
|
||||
sup._load_desired_state()
|
||||
assert "oldnode/some-svc" not in sup.desired_state["services"]
|
||||
|
||||
|
||||
def test_reconcile_generates_no_actions_for_dormant_node(sup, tmp_path):
|
||||
"""Full reconcile: missing service + high disk pressure on a dormant node
|
||||
must produce zero pending actions."""
|
||||
world = tmp_path / "world"
|
||||
(world / "nodes.json").write_text(json.dumps(
|
||||
{"oldnode": {"status": "offline", "disk_pressure": "high"}}
|
||||
))
|
||||
(world / "services.json").write_text("{}")
|
||||
(world / "incidents.json").write_text("{}")
|
||||
|
||||
sup.reconcile()
|
||||
|
||||
assert _pending(tmp_path) == []
|
||||
|
||||
|
||||
def test_node_event_from_dormant_node_not_routed(sup, tmp_path):
|
||||
sup._load_dormant_nodes()
|
||||
event = {
|
||||
"id": "evt-oldnode-1-node_offline-node",
|
||||
"type": "node_offline",
|
||||
"node": "oldnode",
|
||||
"payload": {"affected_node": "oldnode"},
|
||||
}
|
||||
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
|
||||
|
||||
sup._process_ha_events()
|
||||
|
||||
assert _pending(tmp_path) == []
|
||||
|
||||
|
||||
def test_ha_event_from_dormant_node_not_routed(sup, tmp_path):
|
||||
sup._load_dormant_nodes()
|
||||
event = {
|
||||
"id": "evt-oldnode-2-ha_entity_unavailable_long-homeassistant",
|
||||
"type": "ha_entity_unavailable_long",
|
||||
"node": "oldnode",
|
||||
"payload": {},
|
||||
}
|
||||
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
|
||||
|
||||
sup._process_ha_events()
|
||||
|
||||
assert _pending(tmp_path) == []
|
||||
|
||||
|
||||
def test_active_node_events_still_routed(sup, tmp_path):
|
||||
"""Sanity: the dormant guard must not swallow active-node events."""
|
||||
sup._load_dormant_nodes()
|
||||
event = {
|
||||
"id": "evt-piha-3-node_stale-node",
|
||||
"type": "node_stale",
|
||||
"node": "piha",
|
||||
"payload": {"affected_node": "piha"},
|
||||
}
|
||||
(tmp_path / "events" / f"{event['id']}.json").write_text(json.dumps(event))
|
||||
|
||||
sup._process_ha_events()
|
||||
|
||||
assert len(_pending(tmp_path)) == 1
|
||||
|
|
@ -1,9 +1,15 @@
|
|||
# Mosquitto MQTT Broker
|
||||
# Mosquitto MQTT Broker — NOT DEPLOYED / LEGACY
|
||||
|
||||
> **Status (2026-07-28, truth cleanup):** this manifest matches **nothing that
|
||||
> actually runs** (recon `docs/architecture/RECON-multiagent-2026-07-27.md`, C8).
|
||||
> The broker on **vps** is part of the legacy ai-cluster stack (being
|
||||
> decommissioned, see `docs/architecture/ai-cluster-LEGACY.md`); the broker on
|
||||
> **piha** is a host systemd mosquitto (OS package) with no repo definition;
|
||||
> **chelsty** has its own config under `hosts/chelsty-infra/runtime/mosquitto/`.
|
||||
> Do not deploy from this directory. Kept until the MQTT topology decision
|
||||
> (recon open question 7) is made.
|
||||
|
||||
Eclipse Mosquitto is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1.
|
||||
|
||||
## Usage
|
||||
Deployed on the `piha` node.
|
||||
|
||||
Port 1883 for standard MQTT.
|
||||
Port 9001 for WebSockets.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
# NOT DEPLOYED — LEGACY MANIFEST (2026-07-28 truth cleanup, recon C8).
|
||||
# This definition matches NOTHING that actually runs: the broker on vps belongs
|
||||
# to the legacy ai-cluster stack (being decommissioned), and piha's broker is a
|
||||
# host systemd mosquitto (OS package, no repo definition). The chelsty broker
|
||||
# has its own config under hosts/chelsty-infra/runtime/mosquitto/. Kept for
|
||||
# reference until the MQTT topology decision (recon open question 7) is made —
|
||||
# do not deploy from this directory.
|
||||
service:
|
||||
name: mosquitto
|
||||
owner_node: vps
|
||||
owner_node: vps # WRONG in practice — see header; README used to say piha
|
||||
exposure: private
|
||||
dependencies: []
|
||||
ports:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
service:
|
||||
name: stability-agent
|
||||
owner_node: chelsty
|
||||
# Deployed per-host: runs on vps, piha and solaria (and chelsty-infra once the
|
||||
# site is revived). Was owner_node: chelsty — not a real node name; fixed during
|
||||
# the 2026-07 truth cleanup (recon B7/F20.5), closing the docs/backlog.md
|
||||
# "stability-agent / node_exporter owner_node single" item for stability-agent.
|
||||
owner_node: per-host
|
||||
exposure: private
|
||||
dependencies: []
|
||||
healthcheck:
|
||||
|
|
|
|||
Loading…
Reference in a new issue