_resolve_incident() only ever fires from process_event() on a
service_healthy/service_recovered event. A service that is removed,
renamed, or was only ever a one-off test never emits that event again,
so its incident stays "active" in world/incidents.json forever — this
is what left 5 incidents wedged on VPS until a manual on-node edit
during the 2026-08-26 recon session (docs/sessions/2026-08-26.md).
Two independent unwedging mechanisms, both in observer._prune_stale_world
(runs every cycle, so no new event is required to trigger either):
(a) Time-based fallback: any active incident with last_occurrence older
than INCIDENT_STALE_RESOLVE_SECS (env, default 24h) auto-resolves
with resolved_reason="auto_stale_no_events_24h". Unlike the existing
orphan case (Case 2, 5-min guard, only unlinked incidents), this
also clears a service's lingering incident_id link — that link is
exactly what a decommissioned service's incident never gets a
chance to clear via the normal event path.
(b) Manual path: an operator touches
world/resolve-requests/<incident-id>; the observer consumes the
flag file each cycle, force-resolves with resolved_reason=
"manual_operator", and always removes the flag (even for an
unknown/already-resolved id) so a mistyped flag can't sit forever
looking unprocessed.
Chose a flag file over adding a mutation endpoint to operator_ui.py:
/action/mutate only knows actions/<status>/<id>.json, there is no
incidents equivalent, and world/incidents.json is exclusively
observer-owned (rewritten wholesale every cycle by _save_world) —
a second writer (the HTTP handler thread) would race the observer's
own writes. A flag file needs no new HTTP surface and reuses the
same "operator drops a file, the owning process consumes it"
pattern the actions pending/approved queue already uses. Smaller
diff, no new attack surface on a server with no auth on writes.
Tests added to test_incident_lifecycle.py: stale-resolve past the
threshold (service still linked), negative case (fresh active incident
stays active), configurable threshold, manual-flag resolve + flag
removal, flag for an unknown incident, flag for an already-resolved
incident. Full control-plane suite: 179 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WDKj5LRY8vdQMx57dfNnu
Wyciek plikow dispatch potwierdzony 2026-08-06 (session log, follow-up #1):
LUSTRO re-pullowalo te same dwie akcje co 60 s przez wiele dni, odbijajac sie
od bramki idempotencji, i nie zostawilo po sobie ani jednej linii w logach.
Przyczyna zlozona z dwoch niezaleznych defektow:
1. Executor tworzyl actions/dispatch/<node>/ z 0o755 (aerbot:aerbot). Rsync-pull
z noda uwierzytelnia sie jako inny uzytkownik, bedacy tylko *czlonkiem* tej
grupy. --remove-source-files musi zrobic unlink pliku, a unlink wymaga prawa
zapisu w katalogu nadrzednym, nie na samym pliku. Zrodlo przezywalo pobranie.
(dispatch/piha mialo historycznie 775 i dlatego dzialalo.)
2. node-agent traktowal rc=23 jako benign obok 0 i 24, wiec rsync zglaszal
porazke, a agent ja polykal.
Executor: _ensure_inbox_dir() = mkdir + bezwarunkowy os.chmod(0o775). chmod jest
bezwarunkowy z dwoch powodow: mkdir(mode=) jest maskowany przez umask procesu
(przy 0o022 daje dokladnie feralne 0o755), a inboxy zalozone przez wczesniejszy
build juz istnieja na flocie z 0o755. Naprawa w miejscu zapisu, a nie skanem przy
starcie: jedno idempotentne wywolanie na tej samej sciezce kodu, ktora pisze plik
dispatch, wiec nie da sie rozjechac z pisarzami. Blad chmod nie jest fatalny —
akcja i tak sie wykonuje, a nieskasowane zrodlo widac teraz po stronie noda.
Objete tez actions/deploy/<node>/ (deploy-runner): ten sam wzorzec drenowania
tym samym rsync-pullem, ten sam defekt, jedno wywolanie obok.
node-agent: klasyfikacja kodow wyjscia zamiast wspolnej listy benign.
Weryfikacja empiryczna rsync 3.4.1 pokazala, ze rc=23 pokrywa dwa rozne
przypadki, a rozroznia je dopiero stderr:
* `change_dir ... No such file or directory` — executor zaklada inbox dopiero
przy pierwszym dispatchu, wiec kazdy nod, do ktorego nic nie poszlo, dostaje
rc=23 co cykl. DEBUG — inaczej byloby po linii na minute z wiekszosci floty
i realny sygnal utonalby w szumie.
* `sender failed to remove <plik>: Permission denied` — wlasnie ten wyciek.
WARNING z pelnym stderr.
Pusty (ale istniejacy) inbox to rc=0, nie 23 — dotychczasowy komentarz w kodzie
mowil inaczej. rc=24 zostaje benign (wyscig z executorem piszacym inbox),
pozostale kody to teraz ERROR, nie WARNING. Zachowanie funkcjonalne bez zmian:
retry i idempotencja dzialaja jak dotad, zmienia sie wylacznie widocznosc.
Testy: 4 nowe w test_executor_dispatch.py (oba inboxy 0o775 pod umask 0o022,
naprawa istniejacego 0o755 in place, dispatch przezywa nieudany chmod), 5 w
test_action_dispatch.py na klasyfikacje rc. Zastapiony
test_pull_treats_empty_source_returncodes_as_non_error — kodyfikowal wlasnie to
zalozenie, ktore okazalo sie bugiem. Oba zestawy sprawdzone mutacja: bez chmod
padaja 3 testy executora, przy starej liscie benign pada test rc=23.
node-agent 70 passed, control-plane 173 passed.
Refs docs/sessions/2026-08-06.md (follow-up #1)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.
Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:
- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
dormant_nodes and duty_cycle_nodes from one topology read. dormant
behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
are logged at INFO and return; no action is written.
duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.
Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.
Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.
Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
126 plikow (md, yaml, sh, py) odwolywalo sie do sciezek sprzed migracji.
15 markdown-linkow [..](..) -> policzona sciezka WZGLEDNA wobec pliku
odsylajacego (wczesniej czesc z nich byla repo-root-relative i nie
rozwiazywala sie z katalogu, w ktorym lezala)
200 odwolan tekstowych (backticki, proza, yaml, importy w kodzie)
-> nowa sciezka repo-root-relative, zgodnie z konwencja repo
5 linkow rodzenstwa (gole nazwy plikow, np. "](DEPLOY.md)") — dzialaly
tylko w starym katalogu; przeliczone recznie
Objete m.in.: CLAUDE.md (scripts/onboard/README.md -> kb/runbooks/
node-onboarding-tool.md, docs/backlog.md -> kb/phases/backlog.md),
README.md, .claude/skills/, 20 session logow, kod jobow.
Ostatnie 5 odwolan pochodzi z tresci wciagnietej rebasem z origin/master
(session log 2026-07-31, override node-agenta na SOLARII, dwie pozycje
backlogu) — wskazywaly na docs/incidents/, docs/kb/modules/ i
services/narty27/README.md sprzed migracji.
Dodany wzajemny link miedzy kb/services/control-plane.md (stub kodu)
a kb/subsystems/control-plane.md (opis, deprecated) — dwa dokumenty o tym
samym systemie, latwe do pomylenia.
Weryfikacja na 790 plikach: 0 odwolan do starych sciezek,
0 martwych linkow markdown. Lint OKF: 190/190 plikow ZGODNE.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Executor odpalal scripts/deploy/deploy-node.sh <node> <service> wewnatrz
swojego kontenera: skrypt ignoruje oba argumenty i wymaga repo w
${HOME}/homelab-codex-ws (w kontenerze HOME=/home/homelab) -> exit 1 w 18.
linii. Za tym brak git, brak klienta docker w obrazie, a gdyby przeszedl —
deploy calego zestawu uslug hosta executora zamiast wezla z akcji. Kazdy
redeploy padal (recon D14/D15; 18 pending / 0 completed).
Redeploy idzie teraz ta sama sciezka pull co container_restart — VPS nigdy
nie inicjuje polaczenia do wezla:
executor -> actions/deploy/<node>/<id>.json
-> deploy-runner (systemd na hoscie) rsync-pull, walidacja, deploy
-> action_result event -> executor rozlicza completed/failed
- scripts/deploy/deploy-service.sh: deploy jednej uslugi, wspoldzielony z
deploy-node.sh, wiec inwokacja compose (a przez to nazwa projektu) jest
identyczna jak przy deployu recznym
- jobs/deploy-runner/: host-level, nie kontener — compose rozwiazuje
wzgledne bindy i nazwe projektu tak jak przy deployu czlowieka;
niezalezny od node-agenta, wiec potrafi zredeployowac takze jego
- walidacja: tylko typ redeploy, node musi sie zgadzac, usluga musi byc w
hosts/<node>/services.yaml, zadna tresc z payloadu nie trafia do shella
- --force-recreate bez --build i bez --remove-orphans: redeploy to
rekoncyliacja, nie wysylka kodu
- executor: REDEPLOY_TIMEOUT_SECS=900, /repo zjechany do :ro (nieuzywany)
248 testow zielonych; deploy-node.sh przecwiczony na atrapie dockera —
argv compose bez zmian. Instalacja unitow na wezlach i E2E: backlog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
healthcheck_failed incidents fell through to redeploy, which is broken as
wired (executor calls deploy-node.sh with arguments it ignores, at a path
that does not exist in the container) — so 3376 healthcheck_failed events
dead-ended with no working remediation (recon D14/D15). A container restart
plausibly heals a failing healthcheck and rides the executor path that
actually works; redeploy returns to the map once etap 2 fixes the executor.
service_unhealthy / deployment_failed / missing_service stay on redeploy —
theoretical until etap 2, kept so drift remains visible in pending actions
(noted in comments). CLAUDE.md routing table updated to match; stale
mqtt_unreachable example in the observer's trigger_type comment refreshed.
Tests: trigger-type recognition and the end-to-end observer→supervisor
reconcile test parametrized over both container_restart triggers, with an
assertion that no redeploy action is also generated. Full control-plane
suite: 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
topology.yaml (per its own rule that hosts/*/services.yaml are
authoritative, recon F20.1):
- add status field; chelsty-infra + chelsty-ha -> dormant (site hardware
down since ~2026-06-01, revival planned)
- add lustro as a full active node (runs node-agent, ships events, F20.11)
- drop per-node service lists (vps list contradicted hosts/vps, F20.1;
piha/solaria lists were stale too, F20.8/F20.9) — node-level truth only
- deployment.mode pull -> push: every deploy script SSH-pushes from
saturn (F20.10)
Dormant semantics in code:
- observer (scripts/observer/observer.py): loads status from topology;
_prune_stale_world skips dormant nodes — last-known world state stays
frozen, no node_offline/node_stale/node_online events emitted
- supervisor (services/control-plane/src/supervisor.py): reloads dormant
set each reconcile; dormant hosts' services excluded from desired state
(existing pending actions auto-cancel via
service_removed_from_desired_state), disk_cleanup skipped, node/HA
events from dormant nodes not routed to alerts
Tests: services/control-plane/tests/test_dormant_nodes.py (9 cases);
full suite 145 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Executor nie ma klienta ssh ani klucza do floty (uid 1000 homelab, brak
~/.ssh, brak resolucji nazw wezlow) — container_restart przez subprocess ssh
failowal w 6ms na kazdej probie. Zamiast dodawac SSH do executora, kierunek
jest odwrocony: executor zapisuje zlecenie do
/opt/homelab/actions/dispatch/<node>/<action_id>.json, a node-agent na
docelowym wezle (ktory ma dzialajacy docker.sock i juz ma klucz SSH do VPS
uzywany do shippingu eventow) sam je odbiera i wykonuje lokalnie.
- executor: _dispatch_container_restart pisze zlecenie zamiast ssh;
_reconcile_running_actions konsumuje zwrotne action_result eventy i
timeoutuje akcje bez odpowiedzi (ACTION_TIMEOUT_SECS, domyslnie 300s).
redeploy/disk_cleanup/alert_only bez zmian.
- node-agent: nowy krok w petli — rsync-pull wlasnej podkatalogu dispatch z
VPS (ten sam klucz co _ship_events_to_vps, w przeciwnym kierunku; no-op na
VPS, gdzie katalog jest lokalny), walidacja (node_name, whitelist tylko
container_restart, odmowa restartu wlasnego kontenera), wykonanie przez
docker SDK, raport jako event action_result (istniejacy kanal shippingu).
Idempotencja przez znacznik w /opt/homelab/state/processed-actions/.
- 26 nowych testow (10 executor, 16 node-agent), pelny suite obu serwisow
183/183 zielony.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test_run_once_* cases were flaky/order-dependent. Root cause: observer.observer
derives OBSERVER_STATE_FILE from STATE_DIR at import time. The helper patched
STATE_DIR but never OBSERVER_STATE_FILE, so run_once()/_save_checkpoint() wrote the
checkpoint to the real /opt/homelab/state/observer_checkpoint.json. Those node_checkpoints
(tmp paths tagged with a pytest run number) leaked across tests and across pytest runs;
run_once's `file_path > checkpoint` string compare then skipped/kept events based on
run-number ordering. The helper also never restored the module globals it overwrote.
Replace both ad-hoc helpers with an autouse monkeypatch fixture that redirects every
observer path — including OBSERVER_STATE_FILE — into the per-test tmp_path and reverts
them afterward. Tests no longer touch real disk and are deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nodes that crash or lose connectivity without emitting node_offline
stay online in world state indefinitely. _prune_stale_world() now
flips any online node to offline if its last_seen is older than
NODE_OFFLINE_TTL_SECS (default 300 s = 5× the 60 s heartbeat interval).
Nodes with last_seen=None (never reported) and already-offline nodes
are left unchanged. Five new tests cover all branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recovery from bad merge of task/observer-poison-quarantine (c255a02)
which carried false deletes from a stale branch base. Re-applies only
the genuine observer changes on top of correct master state.
When an event file fails to parse (malformed JSON, truncated, corrupted),
the observer previously kept retrying on every cycle while the node's
checkpoint stayed pinned — all subsequent good events for that node lost.
Now: first parse failure -> atomic os.replace to STATE_DIR/observer_failed_events/<node>/
with collision handling. Checkpoint advances, downstream events flow.
Move failures are logged but don't crash the loop.
Complementary to the atomic_write_json fix on state files; this addresses
the same race-pattern on event files instead.
Regression test asserts: bad event quarantined to failed_events dir,
removed from hot path, subsequent good event processed (node online),
checkpoint moves to good event.
Two root causes for stale "active" incidents on the dashboard:
1. TypeError bug in _prune_stale_world: last_occurrence / resolved_at
can be an ISO-8601 string (stability-agent via events.py) or a Unix
int (node-agent). The previous session's auto-resolve did plain
`time.time() - last_occ` which raises TypeError for strings,
silently preventing _save_world() from being called and leaving
incidents perpetually "active" on disk.
Fix: add _parse_ts(ts) -> float that handles int, float, and
ISO-8601 strings uniformly. All timestamp arithmetic now goes through
it; returns 0.0 on None / garbage to keep comparisons safe.
2. Orphaned active incidents: _resolve_incident clears service["incident_id"]
and marks the incident "resolved" in memory, but if incidents.json was
truncated mid-write (pre-atomic-write era), the observer loaded it at
next startup with status="active" and no service entry pointing to it.
No code ever touched these orphans again.
Fix: _prune_stale_world now runs two cleanup passes each cycle:
- Case 1 (healthy-linked): service.status=="healthy" AND incident_id
still set → resolve immediately (service cannot have active incident)
- Case 2 (orphaned): active incident with no service link AND
last_occurrence > 5 min ago → resolve (5-min guard for creation race)
Both cases are wrapped in try/except so a bug here never crashes the
observer loop or blocks _save_world.
Also fixes the 7-day stale-incident prune to use _parse_ts so
ISO-string resolved_at values are handled correctly.
3. Operator UI: current_incidents() now filters to status=="active" only.
Resolved incidents were previously included in the /incidents endpoint,
making the dashboard show a wall of historical records as if active.
Nocturnal job investigation: _cleanup_control_plane_fs in node-agent runs
every 60s on VPS (not midnight-specific); it reads observer_checkpoint.json
(now written atomically) and deletes old event files. No non-atomic writes
found. Midnight clustering was likely external (logrotate / OS flush);
the supervisor's resilient loader already handles such transient issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
11 new test cases in test_state_reliability.py covering:
- atomic_write_json: produces valid JSON, no .tmp left behind, overwrites,
works with nested structures
- _load_actual_state: returns False on empty / truncated file, returns True
on valid files, preserves last-known-good state across a parse failure
- reconcile: empty/truncated services.json or incidents.json generates zero
actions (skip-cycle semantics proven end-to-end)
- healthy service with valid world state generates no spurious action
All 32 tests (11 new + 21 existing) pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala