_resolve_incident() only ever fires from process_event() on a
service_healthy/service_recovered event. A service that is removed,
renamed, or was only ever a one-off test never emits that event again,
so its incident stays "active" in world/incidents.json forever — this
is what left 5 incidents wedged on VPS until a manual on-node edit
during the 2026-08-26 recon session (docs/sessions/2026-08-26.md).
Two independent unwedging mechanisms, both in observer._prune_stale_world
(runs every cycle, so no new event is required to trigger either):
(a) Time-based fallback: any active incident with last_occurrence older
than INCIDENT_STALE_RESOLVE_SECS (env, default 24h) auto-resolves
with resolved_reason="auto_stale_no_events_24h". Unlike the existing
orphan case (Case 2, 5-min guard, only unlinked incidents), this
also clears a service's lingering incident_id link — that link is
exactly what a decommissioned service's incident never gets a
chance to clear via the normal event path.
(b) Manual path: an operator touches
world/resolve-requests/<incident-id>; the observer consumes the
flag file each cycle, force-resolves with resolved_reason=
"manual_operator", and always removes the flag (even for an
unknown/already-resolved id) so a mistyped flag can't sit forever
looking unprocessed.
Chose a flag file over adding a mutation endpoint to operator_ui.py:
/action/mutate only knows actions/<status>/<id>.json, there is no
incidents equivalent, and world/incidents.json is exclusively
observer-owned (rewritten wholesale every cycle by _save_world) —
a second writer (the HTTP handler thread) would race the observer's
own writes. A flag file needs no new HTTP surface and reuses the
same "operator drops a file, the owning process consumes it"
pattern the actions pending/approved queue already uses. Smaller
diff, no new attack surface on a server with no auth on writes.
Tests added to test_incident_lifecycle.py: stale-resolve past the
threshold (service still linked), negative case (fresh active incident
stays active), configurable threshold, manual-flag resolve + flag
removal, flag for an unknown incident, flag for an already-resolved
incident. Full control-plane suite: 179 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WDKj5LRY8vdQMx57dfNnu
Automatyzuje kroki 2-3 z kb/runbooks/kb-site-deploy.md (generacja, kontrola
wyciekow, upload przez helper alpine, smoke test) w jednym fail-closed
skrypcie zamiast recznego klikania. Runbook zaktualizowany, zeby
referencjonowal skrypt, z zachowaniem wyjasnien co/dlaczego dla kazdej bramki.
Dodaje skill kb-publish, ktory przypomina operatorowi o publikacji po
zmianach w kb/**/*.md, ale nigdy sam nie uruchamia skryptu (dotyka prod
przez SSH).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wystawka KB stoi za bramka NPM na ?key=<token>. Linki generowane przez
gen_pages.py (index -> dokument, dokument <-> dokument, powrot do indexu)
tokenu nie nosily, wiec kazde klikniecie ze strony wpadalo w 403 —
dzialal wylacznie recznie sklejony URL do indexu.
Token podaje sie przy generacji: --access-token TOKEN albo zmienna
ACCESS_TOKEN. Nie ma go w repo w zadnej formie — to parametr runtime,
nie stala w kodzie. Bez tokenu generacja dziala jak dotad, z golymi
linkami (tryb lokalnego podgladu); wyjscie jest wtedy bajt w bajt takie
samo jak przed zmiana.
with_token() doklada ?key=... przed ewentualna kotwica i uzywa & gdy URL
ma juz wlasne query params (dzis nie ma — obrona na zapas). Token jedzie
przez urllib.parse.quote. Kotwice (#sekcja) i linki zewnetrzne zostaja
nietkniete. Wartosc nigdy nie leci na stdout — build() loguje tylko
TAK/NIE, bo logi z generacji bywaja wklejane.
--check: prawdziwy token (32+ hex) wygladal dla skanera dokladnie jak
wyciek `token-hex`. scan_line() wycina teraz wartosc `key=` WYLACZNIE
wewnatrz atrybutu href — ten sam token w tresci strony, po innym
parametrze niz key, albo poza href nadal jest raportowany jako wyciek.
Test: 11 stron public + index; z --access-token TEST123 wszystkie 12
linkow spisu, link doc->doc (agent-operating-procedures ->
action-approval-model) i kazdy powrot "All documents" niosa ?key=TEST123;
canonical swiadomie bez tokenu (metadana, nie nawigacja). Bez tokenu
diff vs HEAD pusty poza znacznikiem czasu. gen_pages --check exit 0 dla
generacji bez tokenu, z TEST123 i z realistycznym tokenem 64-hex;
check_okf.py exit 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Warstwa "nie daj sie przypadkiem znalezc" dla publicznej wystawki KB:
- gen_pages.py: <meta name="robots" content="noindex, nofollow"> w <head>
kazdej generowanej strony (page_shell, wiec takze index).
- gen_pages.py: DEFAULT_BASE_URL -> https://kb-e2a24af3.okit.pl. Slug musi
zgadzac sie z rekordem DNS i vhostem w npm@PIHA (runbook kb-site-deploy).
- services/kb-site: static/robots.txt (Disallow: /) montowany ro na
/usr/share/nginx/html/robots.txt. Plik nie jest dokumentem KB, wiec jedzie
z repo, a nie z wolumenu podmienianego przy kazdej publikacji.
- kb/services/kb-site.md: sekcja "Access" — token w query paramie na warstwie
nginx/NPM (sekret zyje tylko w NPM, nie w repo) + obscure subdomain +
noindex. Explicit: to obscurity, nie kontrola dostepu — token w URL laduje
w access logach, historii przegladarki i naglowku Referer.
Bramka publikacji bez zmian: gen_pages.py --check exit 0 (22 wyciszone
whitelista, jak dotad).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decyzja operatora 2026-08-04: /opt/homelab to standardowa sciezka deploy rootu
homelaba, ta sama na kazdym wezle, opisana wprost w publicznej czesci modelu
(standards, service-model, observer, event-system). Nie ujawnia sekretow ani
topologii, wiec zostaje na stronie publicznej.
scripts/kb/check_whitelist.txt: wpis `/opt/homelab` z uzasadnieniem i data.
Zakres wyjatku jest waski — sprawdzone, ze wycisza wylacznie warianty
`/opt/homelab/...`; `/home/oskar/...`, `/opt/other/...`, adresy RFC1918 i porty
dalej zapalaja czerwone.
gen_pages.py: load_whitelist() obcina komentarz `#` w dowolnym miejscu linii,
nie tylko na jej poczatku — wyjatek ma stac obok uzasadnienia, a nie osobno.
Zaden ze skanowanych wzorcow nie zawiera `#`, wiec obciecie jest bezpieczne.
kb/runbooks/kb-site-deploy.md §2: zapisany aktualny stan bramki (exit 0, 22
trafienia wyciszone) zamiast opisu decyzji do podjecia.
Po zmianie: python3 scripts/kb/gen_pages.py --check -> CZYSTO, exit 0.
Tryb --check nie generuje niczego: skanuje build/kb-site/**/*.html linia po
linii i przerywa z exit 1 na pierwszym zestawie trafien. Wzorce:
ip-rfc1918 192.168./10./172.16-31.
ip-tailscale 100.64-127.
ip-public-v4 reszta poprawnych adresow v4 (loopback, link-local, multicast,
broadcast i pule dokumentacyjne RFC 5737 sa neutralne)
ip-v6 adresy z "::" albo >=4 grupami (3 grupy to zwykle godzina)
port :NNNN w zakresie 1024-65535
path-host /home/... i /opt/...
token-hex ciagi hex >=32 znakow
token-b64 ciagi base64-podobne >=40 znakow mieszajace cyfry i litery
(sciezki absolutne odsiane — raportuje je path-host)
Raport: plik:linia [wzorzec] trafienie, na koncu licznik per wzorzec.
scripts/kb/check_whitelist.txt — swiadome wyjatki, na start PUSTY (same
komentarze z opisem formatu). Wpis to `<fragment>` albo
`<sciezka strony>|<fragment>`; fragment dopasowuje sie jako podciag trafienia,
wiec jeden wpis `/opt/homelab` wycisza wszystkie warianty.
Uruchomione lokalnie na 11 wygenerowanych stronach: 22 trafienia, wszystkie
path-host (/opt/homelab/... w dokumentach public), zero IP, portow i tokenow.
Whitelist zostaje pusta — decyzja co z tymi sciezkami zrobic nalezy do
operatora (patrz kb/runbooks/kb-site-deploy.md).
scripts/kb/gen_pages.py — kb/**/*.md -> build/kb-site/ (index.html pogrupowany
per type + strona na dokument). Renderer markdown na samej bibliotece
standardowej, wzorowany na ~/narty-2027/saalbach-kb/gen_pages.py; parser
frontmattera wspoldzielony z check_okf.py, zeby lint i generator widzialy
frontmatter tak samo.
Kwalifikacja fail-closed: publikowany jest wylacznie dokument z jawnym
`visibility: public`. Brak frontmattera, niepoprawny YAML, brak pola albo inna
wartosc = private. Linki do dokumentow nieopublikowanych nie sa renderowane jako
linki — zostaje etykieta z dopiskiem [private]; render_inline ma druga bramke
(linkuje tylko http/mailto/kotwice/.html), wiec martwy odnosnik nie ma jak
przeciec na strone.
BASE_URL jest parametrem (--base-url, domyslnie https://kb.okit.pl) i trafia do
<link rel="canonical">. Stopka kazdej strony: data generacji + git rev-parse
--short HEAD.
check_okf.py: EXCLUDE_DIRS = ("build",) — wyjscie generatora nie jest zrodlem
i nie podlega lintowi. build/ dopisany do .gitignore.
Uruchomione lokalnie: 150 dokumentow kb/, 10 public, 140 pominietych.
Weryfikacja 0-odwolan z 01db57a liczyla tylko podzbior prefiksow — poza nim
zostalo 13 wskaznikow w plikach niemarkdownowych i w prozie dokumentow:
docs/kb/modules/05-faza3-plan.md -> kb/phases/kb-m5-faza3.md (2x systemd)
docs/kb/modules/05-faza4-plan.md -> kb/phases/kb-m5-faza4.md (kb-query app.js)
docs/kb/modules/DECYZJE-*.md -> kb/decisions/kb-dokumenty-otwarte.md
docs/backlog.md (npm panel admina) -> kb/decisions/backlog-aktywne.md
docs/backlog/ (uid/gid floty) -> kb/decisions/backlog-uid-gid-flota.md
docs/kb/modules/0X-*.md -> kb/phases/kb-m*.md
docs/incidents/2026-07-30-*.md -> kb/incidents/ (3x node-agent)
docs/architecture/RECON-multi*.md -> kb/subsystems/recon-multiagent.md
jobs/deploy-runner/README.md -> kb/services/job-deploy-runner.md
Wyjatek zamierzony: `docs/kb/modules/05-faza3-pilot-streszczen.md` w §11 planu
fazy 3 to nazwa artefaktu, ktory nigdy nie powstal — przepiety na docelowa
konwencje (kb/phases/kb-m5-faza3-pilot-streszczen.md), zeby przyszly plik
wyladowal w nowym drzewie, a nie w skasowanym katalogu.
Weryfikacja: skan po 67 sciezkach zmigrowanych w tej galezi (git grep -F na
kazdej) = 0 trafien poza docs/sessions (logi historyczne, celowo nietkniete);
0 martwych linkow markdown na 190 plikach; check_okf.py 190/190 ZGODNE.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Naprawa kontraktu CLAUDE.md §Service Structure (opcja b). Migracja do KB
zabrala README z katalogow serwisow i hostow, przez co 0/26 katalogow
services/ spelnialo wymagany layout. Wskazniki przywracaja nawigacje,
nie duplikujac tresci.
31 wskaznikow, jednolity format, dokladnie 5 linii:
# <nazwa>
<jedno zdanie opisu>
Dokumentacja: [kb/...](../../kb/...)
Opis nie jest pisany od zera — wyciagany z kb-doca: pierwsze pelne zdanie
pierwszego akapitu (sklejane z zawinietych linii, ciete tylko tam, gdzie
backticki i nawiasy sa zbilansowane), a dla node'ow czlon tytulu H1 po
myslniku. Dla ha-mcp opis z H1, bo pierwszy akapit zaczyna sie od markera
statusu. Wiodace markery "**Status: ...**" sa zdejmowane.
26 x services/<svc>/README.md, 5 x hosts/<node>/README.md.
WYJATEK services/home-assistant/config/ken-legacy/README.md: pelne
ostrzezenie "historical archive, do not deploy" przywrocone doslownie
z historii (odzyskane z drzewa sprzed migracji) + link do kb-doca.
Ostrzezenie musi stac tam, gdzie chroni — w katalogu archiwum, nie tylko
w KB. Odwolanie do services/home-assistant/DESIGN.md przepiete na
kb/decisions/ha-configs-as-code.md + kb/incidents/2026-07-22-ha-dwie-instancje.md.
check_okf.py: POINTER_GLOBS + is_pointer() wykluczaja wskazniki ze scope'u
lintu. Wskazniki celowo NIE maja frontmattera OKF — to nawigacja, nie
dokumenty KB. Wykluczenie zapisane wprost, zeby poszerzenie SCOPE nie
zaczelo ich nagle walidowac.
Bez wskaznikow: hosts/chelsty-ha/ i hosts/lustro/ — nie maja dokumentow
w kb/nodes/ (luka odnotowana juz w reconie etapu 1). Utworzenie ich
wymagaloby napisania nowej dokumentacji, czyli wyjscia poza konwersje.
Lint: 190/190 ZGODNE. Weryfikacja 822 plikow: 0 martwych linkow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
126 plikow (md, yaml, sh, py) odwolywalo sie do sciezek sprzed migracji.
15 markdown-linkow [..](..) -> policzona sciezka WZGLEDNA wobec pliku
odsylajacego (wczesniej czesc z nich byla repo-root-relative i nie
rozwiazywala sie z katalogu, w ktorym lezala)
200 odwolan tekstowych (backticki, proza, yaml, importy w kodzie)
-> nowa sciezka repo-root-relative, zgodnie z konwencja repo
5 linkow rodzenstwa (gole nazwy plikow, np. "](DEPLOY.md)") — dzialaly
tylko w starym katalogu; przeliczone recznie
Objete m.in.: CLAUDE.md (scripts/onboard/README.md -> kb/runbooks/
node-onboarding-tool.md, docs/backlog.md -> kb/phases/backlog.md),
README.md, .claude/skills/, 20 session logow, kod jobow.
Ostatnie 5 odwolan pochodzi z tresci wciagnietej rebasem z origin/master
(session log 2026-07-31, override node-agenta na SOLARII, dwie pozycje
backlogu) — wskazywaly na docs/incidents/, docs/kb/modules/ i
services/narty27/README.md sprzed migracji.
Dodany wzajemny link miedzy kb/services/control-plane.md (stub kodu)
a kb/subsystems/control-plane.md (opis, deprecated) — dwa dokumenty o tym
samym systemie, latwe do pomylenia.
Weryfikacja na 790 plikach: 0 odwolan do starych sciezek,
0 martwych linkow markdown. Lint OKF: 190/190 plikow ZGODNE.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zaadaptowany z ~/narty-2027/saalbach-kb/check_okf.py. Tamten sprawdzal
wylacznie obecnosc frontmattera i niepuste `type`. Tutaj dochodza reguly
tego repo: okf przypiete do "0.1", type/visibility/status z zamknietych
list, updated/as_of jako YYYY-MM-DD, as_of wymagane wylacznie dla
type: audit, superseded_by wymagane wylacznie dla status: deprecated,
stub jako bool, links rozwiazywalne wzgledem katalogu dokumentu.
Zakres walidacji: kb/ + docs/sessions/. Reszta repo (CLAUDE.md, README.md,
.claude/skills/) lezy poza baza wiedzy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Executor odpalal scripts/deploy/deploy-node.sh <node> <service> wewnatrz
swojego kontenera: skrypt ignoruje oba argumenty i wymaga repo w
${HOME}/homelab-codex-ws (w kontenerze HOME=/home/homelab) -> exit 1 w 18.
linii. Za tym brak git, brak klienta docker w obrazie, a gdyby przeszedl —
deploy calego zestawu uslug hosta executora zamiast wezla z akcji. Kazdy
redeploy padal (recon D14/D15; 18 pending / 0 completed).
Redeploy idzie teraz ta sama sciezka pull co container_restart — VPS nigdy
nie inicjuje polaczenia do wezla:
executor -> actions/deploy/<node>/<id>.json
-> deploy-runner (systemd na hoscie) rsync-pull, walidacja, deploy
-> action_result event -> executor rozlicza completed/failed
- scripts/deploy/deploy-service.sh: deploy jednej uslugi, wspoldzielony z
deploy-node.sh, wiec inwokacja compose (a przez to nazwa projektu) jest
identyczna jak przy deployu recznym
- jobs/deploy-runner/: host-level, nie kontener — compose rozwiazuje
wzgledne bindy i nazwe projektu tak jak przy deployu czlowieka;
niezalezny od node-agenta, wiec potrafi zredeployowac takze jego
- walidacja: tylko typ redeploy, node musi sie zgadzac, usluga musi byc w
hosts/<node>/services.yaml, zadna tresc z payloadu nie trafia do shella
- --force-recreate bez --build i bez --remove-orphans: redeploy to
rekoncyliacja, nie wysylka kodu
- executor: REDEPLOY_TIMEOUT_SECS=900, /repo zjechany do :ro (nieuzywany)
248 testow zielonych; deploy-node.sh przecwiczony na atrapie dockera —
argv compose bez zmian. Instalacja unitow na wezlach i E2E: backlog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brakowalo certu dla narty27.kapala.org (host 14 na npm@VPS) — 443 zwracalo
tlsv1 unrecognized name. Proby POST /api/nginx/certificates z meta
{letsencrypt_email, letsencrypt_agree} konczyly sie 400 "data/meta must NOT
have additional properties".
Zweryfikowane w GET /api/schema oraz w /app/internal/certificate.js kontenera
(NPM 2.14.0): meta ma additionalProperties:false i nie zna pol email/agree.
Certbot dostaje --agree-tos na sztywno, a -m <email> z konta uzytkownika NPM.
Poprawny payload to {provider, domain_names, meta:{dns_challenge:false}}.
- create-cert --domain (powtarzalne), HTTP-01, dry-run domyslnie + --apply
- POST /nginx/certificates jest synchroniczne (reload + certbot + reload),
wiec ma wlasny timeout 120 s zamiast globalnych 15 s; po nim polling
GET /nginx/certificates/<id> az do expires_on, z czytelnym bledem po czasie
- gdy POST padnie na timeoucie klienta, cert jest odszukiwany po domenach
(certbot moze wciaz pracowac po stronie serwera)
- set-cert: przyjmuje takze --host-id/--cert-id obok pozycyjnych i umie
ustawic --ssl-forced przy tej samej zmianie
- README: schemat payloadu, skad bierze sie email, przyklad dla narty27
Cert #38 wystawiony (expires 2026-11-01), podpiety pod host 14 z ssl_forced;
https://narty27.kapala.org/viz.html -> 200, http -> 301.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
healthcheck_failed incidents fell through to redeploy, which is broken as
wired (executor calls deploy-node.sh with arguments it ignores, at a path
that does not exist in the container) — so 3376 healthcheck_failed events
dead-ended with no working remediation (recon D14/D15). A container restart
plausibly heals a failing healthcheck and rides the executor path that
actually works; redeploy returns to the map once etap 2 fixes the executor.
service_unhealthy / deployment_failed / missing_service stay on redeploy —
theoretical until etap 2, kept so drift remains visible in pending actions
(noted in comments). CLAUDE.md routing table updated to match; stale
mqtt_unreachable example in the observer's trigger_type comment refreshed.
Tests: trigger-type recognition and the end-to-end observer→supervisor
reconcile test parametrized over both container_restart triggers, with an
assertion that no redeploy action is also generated. Full control-plane
suite: 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The script composes from roles/$ROLE/compose/docker-compose.yml, but no
roles/ directory exists anywhere in the repo (recon B7) and nothing
references the script. Dead since the roles concept was abandoned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
topology.yaml (per its own rule that hosts/*/services.yaml are
authoritative, recon F20.1):
- add status field; chelsty-infra + chelsty-ha -> dormant (site hardware
down since ~2026-06-01, revival planned)
- add lustro as a full active node (runs node-agent, ships events, F20.11)
- drop per-node service lists (vps list contradicted hosts/vps, F20.1;
piha/solaria lists were stale too, F20.8/F20.9) — node-level truth only
- deployment.mode pull -> push: every deploy script SSH-pushes from
saturn (F20.10)
Dormant semantics in code:
- observer (scripts/observer/observer.py): loads status from topology;
_prune_stale_world skips dormant nodes — last-known world state stays
frozen, no node_offline/node_stale/node_online events emitted
- supervisor (services/control-plane/src/supervisor.py): reloads dormant
set each reconcile; dormant hosts' services excluded from desired state
(existing pending actions auto-cancel via
service_removed_from_desired_state), disk_cleanup skipped, node/HA
events from dormant nodes not routed to alerts
Tests: services/control-plane/tests/test_dormant_nodes.py (9 cases);
full suite 145 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ken is HAOS (no SSH/docker exec path), so it needs a REST/WebSocket-only
adapter: automations/scripts/scenes via one GET per
/api/config/<domain>/config/<id>, dashboards/area+entity registries/
input_* helpers via the HA WebSocket API (read-only commands only).
scripts/ha/lib/ha_api.py and ha_ws.py never take a token as a value —
only a token_path, read from disk in-process — so the bearer token never
touches a subprocess argv or a log line. ha_ws.py depends on the optional
websocket-client package and raises a clear, actionable ImportError if
it's missing rather than a raw traceback; import.sh still completes the
REST-only part of the import in that case.
Automations/scripts/scenes reuse split.write_split() so both adapters
produce byte-identical config/<instance>/ layouts and the same
idempotent stale-file cleanup on re-run.
docker-exec adapter logic is untouched.
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first
user-facing HTTP entry point to the KB. FastAPI wrapping
kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via
Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source
links) and GET /healthz. Search API only, no answer synthesis (phase 5) and
no server-side dist filtering — the 0.45/0.55 colour thresholds are a
frontend concern (plan §7, a later step).
Hard startup invariant (plan §2 decision 2): refuses to start unless the
configured EMBED_MODEL is present in both document_chunk.model and
document_summary.embedding_model. Note the latter: document_summary.model is
the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the
embedder — checked live against kb-postgres@PIHA before writing this, see
app/startup.py's docstring. Verified end-to-end with a live docker run: the
invariant crash-loops on a mismatched EMBED_MODEL and passes through to a
real /search hit against the live corpus with a correct model.
Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local
embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and
a failure surfaces as 503, not a crash.
Also: scripts/deploy/deploy.sh's gate now builds each service via
`docker compose build` instead of a raw `docker build <svc_dir>`, so a
service whose docker-compose.yml declares a repo-root build context (needed
here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention
already documented in CLAUDE.md) resolves the same way in the gate as it
does at real deploy time (deploy-node.sh's `docker compose ... up --build`).
No behavior change for existing single-context services — verified against
llm-gateway's compose file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yaml.safe_load has no constructor for HA's `!include`/`!secret`/etc. tags
and raised ConstructorError on any real configuration.yaml. Add a
multi_constructor on `!` that generically wraps scalar/sequence/mapping
payloads in a TaggedValue, and a matching representer that re-emits the
same tag + payload — normalization stays idempotent across passes and key
sorting is unaffected by tagged values.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- node-agent: service_healthy emitowany tylko przy przejsciu w stan zdrowy
(per-service in-memory state), nie co cykl dla kazdego zdrowego serwisu.
To samo dla control-plane HTTP probe. healthcheck_failed/containers_not_running/
incydenty pozostaja emitowane bez zmian (realne sygnaly).
- node-agent: naprawiono _cleanup_control_plane_fs — istniejaca retencja
eventow byla martwa od migracji observer_checkpoint.json na epoch-int
(str(path) <= int rzucal TypeError, lapane cicho przez broad except).
Teraz porownuje epoch-do-epoch i czysci wylacznie service_healthy/node_health
starsze niz checkpoint + 3-dniowy bufor; healthcheck_failed/incydenty/ha_*
zachowane bezterminowo.
- scripts/maintenance/cleanup_event_backlog.py: jednorazowy skrypt do
bezpiecznego czyszczenia backlogu na VPS (dry-run domyslnie, --apply
do usuniecia). Ten sam warunek: typ szumu + starsze niz checkpoint + 1h bufor.
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
token/list-hosts/list-certs/set-cert/create-host, dry-run domyslny dla
zmian (--apply wymagane), stdlib urllib (zero-dep). Adresy npm@VPS
przez Tailscale (100.95.58.48:81), NIE public IP.
+ docs/backlog.md: npm@VPS admin panel :81 publicznie osiagalny —
brak override ograniczajacego bind do mesh/localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy.sh vps iterates hosts/vps/services.yaml in deploy-node.sh, which
includes control-plane. But control-plane also has a dedicated deploy path
(deploy.sh control-plane → deploy-control-plane.sh → deploy-local.sh) that
runs compose from services/control-plane/. The node loop runs compose from
${REPO_PATH}, so on Compose versions that derive the project name from cwd
the two paths own the containers under different project names. The loop's
`up -d --remove-orphans` then Recreates and tears down the running brain
(observer/supervisor/executor/ui), aborting the loop under set -e. This
wiped the VPS control-plane on 2026-06-25.
Generic guard: skip any service that ships its own services/<svc>/deploy-local.sh.
control-plane stays in services.yaml so the gate (pytest+build) still covers it;
only the destructive loop deploy is skipped. Protects future services with a
dedicated deploy path too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Without --env-file, docker compose resolved variables from the repo root
(cwd), not from services/<service>/.env where the file actually lives.
This caused ${TAILSCALE_BIND_IP} to expand to empty string, binding
fleet-prometheus on 0.0.0.0:9090 instead of the Tailscale-only IP —
a security hole on the public VPS.
Guard mirrors the existing override-file pattern: only add --env-file
when the file exists, so services without .env continue to work as
before. Flag is injected into COMPOSE_CMD (before the `up` subcommand)
so docker compose sees it as a global option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the brittle `hosts/$(hostname|lower)` assumption with a scan
of hosts/*/host.yaml for a matching os_hostname field. This fixes VPS
where the OS hostname (ubuntu-4gb-hel1-1) never matched the repo
directory (hosts/vps/), causing a silent "No services found" false-green.
Fallback to lowercase-hostname dir preserved for nodes that haven't yet
received the os_hostname field; exits 1 with a clear message if neither
match nor fallback directory exists.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nodes that crash or lose connectivity without emitting node_offline
stay online in world state indefinitely. _prune_stale_world() now
flips any online node to offline if its last_seen is older than
NODE_OFFLINE_TTL_SECS (default 300 s = 5× the 60 s heartbeat interval).
Nodes with last_seen=None (never reported) and already-offline nodes
are left unchanged. Five new tests cover all branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recovery from bad merge of task/observer-poison-quarantine (c255a02)
which carried false deletes from a stale branch base. Re-applies only
the genuine observer changes on top of correct master state.
When an event file fails to parse (malformed JSON, truncated, corrupted),
the observer previously kept retrying on every cycle while the node's
checkpoint stayed pinned — all subsequent good events for that node lost.
Now: first parse failure -> atomic os.replace to STATE_DIR/observer_failed_events/<node>/
with collision handling. Checkpoint advances, downstream events flow.
Move failures are logged but don't crash the loop.
Complementary to the atomic_write_json fix on state files; this addresses
the same race-pattern on event files instead.
Regression test asserts: bad event quarantined to failed_events dir,
removed from hot path, subsequent good event processed (node online),
checkpoint moves to good event.
rsync fails with "No such file or directory" when intermediate dirs
don't exist. /opt/homelab/deploy/ is not created by 20-base.sh.
Add rrun mkdir -p before rsync_dir; pi owns /opt/homelab so no sudo.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Push-based deploy step for LUSTRO (git_control=false): rsync
services/node-agent/ and the host override to /opt/homelab/deploy/node-agent/
on the remote, then docker compose up --build via SSH.
Guard by effect: skip push+build+up if node-agent container already running
(docker ps filter, not command -v). Verify: container running + events appear
in /opt/homelab/events/lustro/ within 90 s (confirms agent write path).
Override (hosts/lustro/runtime/node-agent/docker-compose.override.yml):
- group_add: ["991"] (docker GID on LUSTRO; 999 from base concatenated — harmless)
- mem_limit: 256m (MagicMirror ~1.9 GiB; agent must be bounded)
- /home/pi/.ssh:/root/.ssh:ro (not /home/oskar/.ssh — pi user)
- /opt/homelab/deploy/node-agent:/repo:ro (no repo checkout on push-based node)
- NODE_NAME=lustro, NODE_TYPE=sd_card, VPS_EVENTS_HOST=100.95.58.48
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stary guard porównywał literał konfigu (SIZE=) zamiast sprawdzać efekt.
Ręcznie postawiony zram był pomijany (dpkg -l vs command -v) i config
był nadpisywany niepotrzebnie.
- Guard by effect: sudo swapon --show | grep /dev/zram + dphys nieaktywny
→ cała sekcja skip bez wchodzenia w substages
- Detekcja pakietu przez dpkg -l zram-tools (nie command -v zramswap — PATH)
- Config: PERCENT=50 (skaluje z RAM) zamiast SIZE=; printf '%s\n' | sudo tee
- Wszystkie weryfikacje zram przez sudo swapon --show (nie zramctl)
- Usuń parsowanie hardware.swap.mb (nieużywane po przejściu na PERCENT)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three idempotent stages with guards (probe-before-mutate), rrun() for all
remote mutations, rprobe() for unconditional state queries. Reads
hardware.swap.mb from node.yaml (default 2048 MB). Adds swap.mb: 2048
to hosts/lustro/node.yaml so the value is declarative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scripts/onboard/README.md (new):
- Tool purpose and --node/--step/--from/--dry-run usage
- Full node.yaml field schema with annotations (ssh_user uid-1000
gotcha, first_contact IP vs .local, deploy_autonomy/git_control gates)
- Step status table (00-access DONE, 00-preflight SCAFFOLD, 10-50 TODO)
- lib/ architecture: run() dry-run convention, yaml_get fallback caveats
- Gotchas/Learnings table from session
CLAUDE.md:
- Node Onboarding section: onboard.sh commands, pointer to README
- Multi-agent worktree mode: add explicit DISCIPLINE RULE — feature
work must happen in agent.sh worktrees, not the main checkout;
references the 2026-06-08 session that violated this
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On first SSH to a new mesh hostname, OpenSSH emits
"Warning: Permanently added 'lustro' to the list of known hosts"
on stderr. The previous code used 2>&1, merging it into the captured
arch variable, which caused the arch assertion to fail with
arch="Warning:Permanentlyadded...".
Fix:
- Add dedicated _TS_SSH opts array with -o LogLevel=ERROR, which
suppresses INFO-level messages (known-hosts, banner) at source
- Remove 2>&1 — stderr is no longer merged into the captured value
- Run only `uname -m` instead of `echo ok && uname -m`; take the last
non-empty stdout line to be robust against any remaining preamble
- Change arch mismatch from warn to die in live mode (warn in dry-run)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs in the grep+sed fallback (triggered when yq is unavailable):
1. Greedy colon match: `s/.*: *//` consumed the *last* `: ` in the line, so
values containing a colon (e.g. `systemd:magicmirror.service`) were
silently truncated to the portion after the last colon.
Fix: `s/^[[:space:]]*[^:]*:[[:space:]]*//' — anchored at line start,
key chars are `[^:]*` (no colons), so only the first `: ` separator is removed.
2. Inline YAML comment not stripped: `first_contact: pi@pimirror2.local # ...`
returned the full tail including `#`, breaking callers like ssh-copy-id.
Fix: add `s/[[:space:]]\+#.*$//` — requires at least one space before `#`
to preserve bare `#` characters inside a value.
Also add leading/trailing whitespace trim as a separate pass.
Both bugs affect any node.yaml field that has an inline comment or a colon
in its value; all ten fields in hosts/lustro/node.yaml now parse correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DRY_RUN now uses 1/0 instead of "true"/"false" across all onboard scripts.
common.sh: add run() — wraps mutations; prints "[dry-run] would: ..." when
DRY_RUN=1. Exported via `export -f run` so child bash processes inherit it.
onboard.sh: remove the `--dry-run → dryrun "Would execute" → continue` bypass.
Steps now always execute; DRY_RUN=1 is exported so each step's own run()
calls handle simulation. The orchestrator no longer needs to know step internals.
remote.sh: update DRY_RUN checks to [ "${DRY_RUN:-0}" = 1 ] for consistency.
00-access.sh: remove all if/else DRY_RUN blocks; replace with:
- Mutations (ssh-copy-id, curl install, tailscale up) wrapped in run()
- Probes (SSH BatchMode test, command -v, _ts_state) run unconditionally
so dry-run reports real current state ("key present → skip" vs "would: ...")
- Stage 3 verify runs always; SSH failure is die in live mode, warn in
dry-run (Tailscale not yet joined is expected on a fresh node)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
00-access.sh implements a 3-stage idempotent access bootstrap:
1. ensure_ssh_key — ssh-copy-id to first_contact (pi@pimirror2.local),
skips if BatchMode key-auth already passes
2. ensure_tailscale — install via install.sh if missing, then tailscale up
--hostname=lustro; prints interactive auth URL to operator, blocks until
authenticated; skips if BackendState already Running
3. verify — SSH over Tailscale to pi@lustro, asserts 'ok' + arch=aarch64
Reads first_contact and tailscale.hostname from node.yaml.
Respects --dry-run. No NOPASSWD or /opt/homelab mutations.
hosts/lustro/node.yaml: fill known hardware facts (arm64, 4096 MB RAM,
zram swap, docker_present, mm_runtime=systemd:magicmirror.service),
add ssh_user=pi, first_contact=pi@pimirror2.local,
services.node-agent.runtime engine=docker mem_limit=256m.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
grep -cv (and grep -v) return exit code 1 when there are zero matches.
With set -euo pipefail this silently aborted the script before count
was returned — causing 'agent.sh new' to fail on a fresh repo with no
existing worktrees.
Fix: move the grep -v into worktree_paths with '|| true' so the
function always exits 0, then derive worktree_count via wc -l.
Refactor [ test ] && prefail pattern to if/then/fi — set -euo pipefail
was silently exiting after the loop because the failing-test compound
propagated exit code 1 through the function return.
Add ERR trap so future silent fails get diagnosed at the source.
Oskar Kapala