kb/services/paperless.md (Stack, OIDC, Storage i backup, RAM na PIHA)
kb/decisions/paperless-split-ocr.md (split OCR serwis@PIHA + worker@SOLARIA)
kb/runbooks/paperless-cutover.md (cutover checklist)
Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kb/services/nextcloud.md (Stack, WebDAV dla ingestu, Storage i backup)
kb/decisions/nextcloud-host-piha.md (decyzja: host = PIHA)
kb/runbooks/nextcloud-cutover.md (OIDC przez Forgejo + cutover checklist)
Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kb/services/gokapi.md (Stack, Backup, Rejestracja w repo)
kb/decisions/gokapi-storage-e2e-siec.md (storage lokalny zamiast S3,
szyfrowanie E2E, bind tylko na Tailscale)
kb/runbooks/gokapi-cutover.md (cutover checklist)
Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kb/services/fleet-prometheus.md (Stack, Placement & exposure, Data, Next steps)
kb/decisions/fleet-prometheus-osobny-od-prom.md ("Why separate from the home prom")
kb/runbooks/fleet-prometheus-deploy.md (Configuration, Verify)
Wzajemne links. Tresc sekcji nietknieta; kontrola: multizbior niepustych
linii czesci == oryginal z HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wzorzec mechaniczny: sekcje deploy/verify/install/testy wycinane do
kb/runbooks/<serwis>-*.md, reszta zostaje dokumentem type: service.
Wzajemne `links` w obie strony. Tresc sekcji nietknieta — przenoszone
doslownie, dodany wylacznie naglowek H1 nowego runbooka.
kb-query, paperless-worker, planner-agent, ha-diag-agent, ollama-piha,
narty27, home-assistant, ha-mcp, job-gmail-header-backfill, job-mail-body-ingest.
Weryfikacja: dla kazdego pliku multizbior niepustych linii
(main + runbook) == oryginal z HEAD. Zero zgubionych, zero dodanych.
Recon szacowal 13 splitow service+runbook; faktycznie 2-typowych jest 10,
pozostale 5 (paperless, nextcloud, gokapi, fleet-prometheus, deploy-runner)
sa 3-typowe i ida osobno jako splity wielotypowe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rozstrzygniecie 5: katalogi z kodem, ktore nie mialy README, dostaja stub
(frontmatter + stub: true + opis jednozdaniowy). NIE pisana pelna dokumentacja.
kb/services/: control-plane, node-agent, brain-watchdog, node-exporter,
job-gmail-bulk-import, pkg-kb-mail, pkg-kb-retrieval.
Kazdy stub podaje zrodlo opisu (service.yaml / docstring / CLAUDE.md /
pyproject.toml). Dla gmail-bulk-import i kb-retrieval zrodla brak — stub
mowi o tym wprost zamiast zmyslac opis.
Dodatkowo kb/subsystems/repo-operating-contract.md — wskaznik na CLAUDE.md
(rozstrzygniecie 4). CLAUDE.md zostaje w korzeniu jako zywa konfiguracja
narzedzia; kb-doc niesie pole contradicts: brak katalogow services/joplin,
services/outline, services/ai-cluster deklarowanych w sekcji
"Repo-managed services on VPS". Kontekst PR2 feat/vps-service-migration,
swiadomie nienaprawiane.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nowy typ `audit` (rozstrzygniecie 2) — migawka stanu z pola `as_of`,
nie opis stanu biezacego.
monitoring-coverage-2026-07-14, ha-automatyzacje-2026-07-23.
Pozostale 5 audytow/reconow jest wielotypowych — wychodza w grupie SPLIT-ow.
git mv + frontmatter, tresc nietknieta.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Jedyny jawny incident-doc w repo. Pozostale incydenty siedza wtopione
w docs/backlog.md, services/home-assistant/DESIGN.md i lustro-shipping-recon
— wychodza w grupie SPLIT-ow.
git mv + frontmatter, tresc nietknieta.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zaadaptowany z ~/narty-2027/saalbach-kb/check_okf.py. Tamten sprawdzal
wylacznie obecnosc frontmattera i niepuste `type`. Tutaj dochodza reguly
tego repo: okf przypiete do "0.1", type/visibility/status z zamknietych
list, updated/as_of jako YYYY-MM-DD, as_of wymagane wylacznie dla
type: audit, superseded_by wymagane wylacznie dla status: deprecated,
stub jako bool, links rozwiazywalne wzgledem katalogu dokumentu.
Zakres walidacji: kb/ + docs/sessions/. Reszta repo (CLAUDE.md, README.md,
.claude/skills/) lezy poza baza wiedzy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
R1 (critical): _prune_stopped_containers no longer calls containers.prune().
The unfiltered API call removes EVERY non-running container, ignoring
RestartPolicy and compose labels — that is what deleted ollama@solaria 19 s
after an operator `docker stop`. No filter argument can fix it (`until` filters
on creation time, not stop time). Replaced with explicit enumeration over
containers.list(all=True, filters={"status": "exited"}), skipping anything with
restart policy unless-stopped/always/on-failure or a com.docker.compose.project
label. A stopped managed service is recorded operator intent and now belongs to
the module's NEVER TOUCHED list; only one-off leftovers are removed. Dangling
image and build cache prune are unchanged.
R2 (high): _sd_card_rate_ok → _cleanup_rate_ok, applied to every cleanup-
eligible node type. ai_node and standard pruned every 60 s (1440/day, "0 MB
reclaimed" in practically every cycle) with no rate limit at all; they now share
sd_card's CLEANUP_INTERVAL_SECS (24 h) and mark the cleanup timestamp.
R3 (medium): removals are named in the log — WARNING with the container names
when the list is non-empty, INFO otherwise. The old line reported megabytes
only, which is why ollama's deletion looked identical to every no-op cycle.
Tests: services/node-agent/tests/test_safe_cleanup.py (22 cases) — the ollama
regression (exited + unless-stopped survives, prune() never called), all three
restart policies, compose-label protection, disposable-leftover removal, sweep
continues past a failed remove, WARNING-level naming, and the rate limit for
ai_node/standard/sd_card plus lte_node still doing nothing.
Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 R1-R3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
node-agent calls docker containers.prune() with no filters every
CHECK_INTERVAL (60 s on VPS). The Docker API removes EVERY non-running
container regardless of RestartPolicy or compose labels — this destroyed
ollama@solaria on 2026-07-30 (19 s after an operator `docker stop`).
On VPS the blast radius is worse: humanai-mailer and humanai-landing have
no compose definition in this repo (recreated by hand from `docker inspect`),
so a pruned container there is an irreversible loss of the only config source.
self.node_type is read only by run_safe_cleanup() (node_agent.py:648,654) and
two log lines (250, 1103). Verified additionally for VPS: the control-plane
filesystem rotation and health probe in run_once() are gated on
`node_name == VPS_NODE_NAME`, not node_type — so they keep running. Monitoring,
event shipping and action dispatch are likewise unaffected.
Temporary — remove once R1 (explicit-enumeration prune) is deployed.
Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 M1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mitygacja tymczasowa z incydentu 2026-07-30-ollama-solaria-vanish (§7, M1),
na czas backfillu embed (faza mailowa KB). Na SOLARII node-agent robił
niefiltrowany `docker container prune` co cykl (60 s), kasując również
kontenery z `restart: unless-stopped` zatrzymane świadomie przez operatora.
Zweryfikowane w kodzie (services/node-agent/src/node_agent.py, linie zgodne
z incydentem): `self.node_type` jest czytane wyłącznie w run_safe_cleanup()
(648, 654) i w dwóch liniach logu (250, 1103). `lte_node` daje wczesny return
w run_safe_cleanup — monitoring, eventy, dispatch akcji bez zmian.
_cleanup_control_plane_fs jest bramkowane node_name == VPS, nie node_type.
stability-agent nie prune'uje — node-agent był jedynym źródłem.
Ścieżka deployu zweryfikowana: deploy-service.sh:100 składa
${HOST_DIR}/runtime/${SERVICE}/docker-compose.override.yml, a HOST_DIR to
hosts/<node> w obu wywołaniach (deploy-node.sh:102 operatorskie,
deploy-runner.sh:170 agentowe). Dowód, że plik nie jest martwy: działający
kontener na SOLARII ma NODE_TYPE=ai_node, co występuje wyłącznie w tym pliku.
`docker compose config` na złożeniu daje NODE_TYPE=lte_node, group_add 999+996
zachowane, projekt "node-agent" (bez zmiany nazwy projektu).
Skutek uboczny: lte_node pomija CAŁY cleanup, więc dangling images i build
cache też nie są sprzątane — pilnować miejsca na dysku SOLARII.
Zdjąć po wdrożeniu R1 na nodzie → przywrócić NODE_TYPE=ai_node.
Nie zdeployowane — deploy po stronie operatora.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Trzeci wpis ze znalezisk sesji 2026-07-31: services/narty27/service.yaml
i README deklaruja exposure private ("no npm vhost, no public ingress"),
a narty27.kapala.org jest publiczne (NPM VPS host #14, cert LE #38).
Pole exposure czytaja agenci — rozjazd kontrakt<->rzeczywistosc.
Fix jako osobny task: najpierw weryfikacja konsumentow pola exposure
(observer/supervisor/deploy), dopiero potem zmiana service.yaml + README.
Pliki serwisu swiadomie nietkniete w tym commicie.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dwa wpisy do sekcji Aktywne, oba ze zrodlem w sesji 2026-07-31:
- upstream GoogleCloudPlatform/knowledge-catalog: generator grafu pomija
linki od "/" wbrew §5.1 wlasnej spec; obejscie = wlasny generator
w pilocie narty27, fix = issue/PR do upstreamu
- hosts/solaria/runtime/ollama/docker-compose.override.yml brak w repo,
mimo ze ollama jest w hosts/solaria/services.yaml — konfiguracja
zywego kontenera niewersjonowana (wzorzec: hosts/piha/runtime/ollama-piha)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Uzupelnia log o fakty brakujace w pierwszej wersji:
- OKF v0.2: migracja = jedna zamiana pola (timestamp -> generated {by,at})
- upstream bug: GoogleCloudPlatform/knowledge-catalog, generator grafu
pomija linki od / wbrew §5.1 wlasnej spec
- landing w lancuchu generatorow (bylo: "strona linkow")
- nowa sekcja "Otwarte po sesji": expected_envelope w mail_queries,
brak hosts/solaria/runtime/ollama/docker-compose.override.yml,
R1-R3 node-agent (incydent 2026-07-30-ollama-solaria-vanish)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Executor odpalal scripts/deploy/deploy-node.sh <node> <service> wewnatrz
swojego kontenera: skrypt ignoruje oba argumenty i wymaga repo w
${HOME}/homelab-codex-ws (w kontenerze HOME=/home/homelab) -> exit 1 w 18.
linii. Za tym brak git, brak klienta docker w obrazie, a gdyby przeszedl —
deploy calego zestawu uslug hosta executora zamiast wezla z akcji. Kazdy
redeploy padal (recon D14/D15; 18 pending / 0 completed).
Redeploy idzie teraz ta sama sciezka pull co container_restart — VPS nigdy
nie inicjuje polaczenia do wezla:
executor -> actions/deploy/<node>/<id>.json
-> deploy-runner (systemd na hoscie) rsync-pull, walidacja, deploy
-> action_result event -> executor rozlicza completed/failed
- scripts/deploy/deploy-service.sh: deploy jednej uslugi, wspoldzielony z
deploy-node.sh, wiec inwokacja compose (a przez to nazwa projektu) jest
identyczna jak przy deployu recznym
- jobs/deploy-runner/: host-level, nie kontener — compose rozwiazuje
wzgledne bindy i nazwe projektu tak jak przy deployu czlowieka;
niezalezny od node-agenta, wiec potrafi zredeployowac takze jego
- walidacja: tylko typ redeploy, node musi sie zgadzac, usluga musi byc w
hosts/<node>/services.yaml, zadna tresc z payloadu nie trafia do shella
- --force-recreate bez --build i bez --remove-orphans: redeploy to
rekoncyliacja, nie wysylka kodu
- executor: REDEPLOY_TIMEOUT_SECS=900, /repo zjechany do :ro (nieuzywany)
248 testow zielonych; deploy-node.sh przecwiczony na atrapie dockera —
argv compose bez zmian. Instalacja unitow na wezlach i E2E: backlog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brakowalo certu dla narty27.kapala.org (host 14 na npm@VPS) — 443 zwracalo
tlsv1 unrecognized name. Proby POST /api/nginx/certificates z meta
{letsencrypt_email, letsencrypt_agree} konczyly sie 400 "data/meta must NOT
have additional properties".
Zweryfikowane w GET /api/schema oraz w /app/internal/certificate.js kontenera
(NPM 2.14.0): meta ma additionalProperties:false i nie zna pol email/agree.
Certbot dostaje --agree-tos na sztywno, a -m <email> z konta uzytkownika NPM.
Poprawny payload to {provider, domain_names, meta:{dns_challenge:false}}.
- create-cert --domain (powtarzalne), HTTP-01, dry-run domyslnie + --apply
- POST /nginx/certificates jest synchroniczne (reload + certbot + reload),
wiec ma wlasny timeout 120 s zamiast globalnych 15 s; po nim polling
GET /nginx/certificates/<id> az do expires_on, z czytelnym bledem po czasie
- gdy POST padnie na timeoucie klienta, cert jest odszukiwany po domenach
(certbot moze wciaz pracowac po stronie serwera)
- set-cert: przyjmuje takze --host-id/--cert-id obok pozycyjnych i umie
ustawic --ssl-forced przy tej samej zmianie
- README: schemat payloadu, skad bierze sie email, przyklad dla narty27
Cert #38 wystawiony (expires 2026-11-01), podpiety pod host 14 z ssl_forced;
https://narty27.kapala.org/viz.html -> 200, http -> 301.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nowy serwis services/narty27 — nginx:alpine serwujacy jeden self-contained
plik viz.html z named volume narty27_content (:ro). Port 8240 to nastepny
wolny w bloku 82x0 na PIHA (8210 paperless, 8220 nextcloud, 8230 kb-query).
Content jest personal: zyje wylacznie w volume narty27_narty27_content i w
zrodle na SOLARII (~/narty-2027/saalbach-kb/viz.html). Nigdy w repo, bez
backup joba, bez bindu pod /opt/homelab/data.
Procedura aktualizacji w README uzywa kontenera-pomocnika (alpine z volume
rw), NIE `docker cp` — przy mount :ro docker cp zwraca "mounted volume is
marked read-only", zarowno dla dzialajacego jak i zatrzymanego kontenera
(zweryfikowane empirycznie 2026-07-31). Zapisywane sa dwie kopie tego samego
pliku: viz.html (nazwa kanoniczna) + index.html (zeby golny root dzialal).
Walidacja: docker compose config OK (volume rozwija sie do
narty27_narty27_content), YAML parse OK, bash -n healthcheck.sh OK.
Bez zywego deployu.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Kontener nie padł ani nie zniknął przy starcie: usunął go własny node-agent.
19 s po operatorskim `docker stop ollama` cykl cleanupu wywołał
`containers.prune()` bez filtrów — Docker kasuje każdy kontener nie-running,
ignorując restart policy i labele compose.
Dowód: ten sam cykl logu node-agent, dwie kolejne linie —
16:45:33,528 WARNING Container exited: ollama (restart=unless-stopped)
16:45:33,563 INFO Pruned stopped containers (57 MB reclaimed)
57 MB to jedyna niezerowa wartość SpaceReclaimed w całym dniu (reszta 0 MB).
Mina uzbroiła się dzień wcześniej: prune istnieje od 01b7758, ale na SOLARII
node-agent nie miał dostępu do docker.sock (GID 996 vs 999) do czasu ddae57c.
Wykluczone: remediation pipeline (dispatch/solaria pusty, whitelist tylko
container_restart), stability-agent (zero ścieżek usuwających), config compose
(AutoRemove=false, brak --rm, brak compose down), cron/systemd (brak prune).
To NIE jest ollama-solaria-start-race — tamte dotyczyły startu kontenera.
Tu problem jest w cleanupie node-agenta i dotyczy każdego serwisu: ai_node
(solaria) i standard (vps) prune'ują bez rate-limitu, sd_card raz na 24 h.
`docker stop` jest obecnie na 4 z 6 nodów operacją destrukcyjną.
Fix świadomie niezaimplementowany — rekomendacje R1-R4 + mitygacja M1 w §7.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decide-support recon for the two per-node sensors. Finding-first, sections 1-5
per brief, ends with DECISION NEEDED (no recommendation).
Headline: stability-agent's filesystem event channel is structurally unreadable
by the control plane, not merely mis-tagged. Recon D15 named one break; there
are five, stacked, any one fatal:
1. wrong subtree — events/<date>/<node>/events.jsonl vs events/<node>/
2. wrong extension — observer.py:912 / supervisor.py:605 glob **/*.json
3. wrong format — observer json.load()s the whole file; JSONL is not one
4. never shipped — node_agent.py:788 rsyncs only events/<node>/
5. details vs payload key, plus service=None (D15)
D15 is stale on master but still true in production: fix f92e161 (2026-07-29) is
merged, the running solaria image was built 2026-05-17, and 100% of the 5,056
deployed events sampled carry service=None. Moot regardless — breaks 1-4 mean
the observer never opens the file. All six event types are inert, not just
containers_not_running.
Counterweight: stability-agent is the sole writer of the Redis keys the
agent-system WebUI renders (4 node, 130 service, 31,458-entry stream, refreshed
during the recon). node-agent has no Redis code. Retiring it blanks that UI.
Runtime evidence over the retained window:
node-agent 50,773 events -> 1 incident
stability-agent 7,317 events -> 0 incidents (structurally zero since
first deploy 2026-05-17)
Also found: deploy-local.sh validates 'chelsty' but the repo has
hosts/chelsty-infra/, so the only override carrying MQTT_HOST has never been
applied (stale homelab:nodes:chelsty key from 2026-06-02); scripts/lib/events.sh
writes to the same dead path; CLAUDE.md:100 documents the dead format as
canonical; recon B7 is stale — stability-agent IS in hosts/{solaria,vps}
services.yaml, but deploy-node.sh:93 skips it.
Read-only recon: no runtime mutations, no config changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Own minimal MCP server exposing the live state of the HA instances in
services/home-assistant/instances.yaml to Claude Code over stdio — the
phase-2 "MCP read-only" gate in services/home-assistant/DESIGN.md.
Operator decision 2026-07-30: build our own rather than adopt hass-mcp,
so the tools reuse scripts/ha/lib/{ha_api,ha_ws}.py (one token-handling
story for the whole HA toolchain) and can answer from the repo and from
instances.yaml, which a generic server cannot.
Seven tools, all read-only, default instance `ken`: list_entities,
get_state, get_areas, find_entities_by_description, read_automation,
list_automations, instance_status.
Read-only by construction, not by policy: REST goes through ha_api.Client
(get/get_raw_text only — no POST method exists), WebSocket commands are
checked against a three-entry *_list allowlist before being sent, and
read_automation reads services/home-assistant/config/<instance>/ rather
than /api/config. Tests assert all three, including a grep guard that
fails if requests.post/call_service ever appears in the package. The
write path stays repo + scripts/ha/deploy.sh.
Details that follow from how this instance actually behaves:
- unavailable is never silent — every entity view carries unavailable +
unavailable_since, every list a count. The 2026-07-23 audit traced ~15
silently dead automations to conditions sitting on dead sensors.
- chelsty-ha (status: offline in instances.yaml) is answered from the
file, never dialed — no 5s timeout for a known-offline LTE site.
- areas come from the WS registries (entity area_id > device area_id) with
a storage-export fallback; area_source/area_note say which was used and
what the offline export cannot resolve.
- PL->EN fuzzy matching, since the house is Polish and the entity_ids are
transliterated English: "czujnik temperatury salon" ->
sensor.thsalon_temperature, each hit explaining why it matched.
- 5s timeouts and errors returned as {"error": ...} inside a normal tool
result — a missing token or an unreachable instance never crashes the
server or hangs the agent.
Registered for Claude Code in the repo-root .mcp.json (new file) as `ha`,
via services/ha-mcp/run.sh (prefers the venv, falls back to system
python3). The mcp SDK lives in services/ha-mcp/.venv — rationale for venv
over --break-system-packages is in the README.
Tests: 42 offline (no network, no HA, no token) + a live read-only smoke
against ken — HA 2026.7.2, 1647 entities, 377 unavailable, 115
automations, 13 areas.
T1: one-shot switch przy timeoutcie mid-embed (ścieżka asyncio.wait_for,
dotąd nieprzetestowana — scenariusz 'SOLARIA wisi'). T2: breaker zostaje
'down' po mid-embed failure — kolejne requesty w oknie TTL idą prosto na
fallback bez probe'a. T3: noga fallbacku nie dziedziczy twardego timeoutu
primary. T4 pominięty (semantyka granicy TTL identyczna, wg raportu).
Pytest kb-query: 42/42 PASS.
S4: pomiar kalibracji 2026-07-27 (peak ~983 MiB, GO) dopisany do override'u
i sekcji Calibration w README — master mówił dotąd 'Confirm/trim after live
calibration'; konfiguracja kontenera identyczna z mierzoną, pomiar się
przenosi. Raport dedup: status zaktualizowany na 'salvage wykonany'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implementacja kodu z tej sesji porzucona na rzecz e7625cd (dopisek redakcyjny
na górze pliku), ale log jest jedynym zapisem faktów operacyjnych: osierocony
natywny ollama.service na PIHA wyłączony 2026-07-27 (backlog odinstalowania
≈2026-08-10), kalibracja live ollama-piha z werdyktem GO (peak ~983 MiB,
~4.2–5.3 s/embed) i baseline bramki §9 (HTTP-equivalence 0 rozbieżności,
sol-down Δ~3e-4 — do powtórki na masterze, raport dedup follow-up (b)).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cherry-pick 1:1 z porzucanego brancha task/kb-f4-fallback (3d4ee38) wg decyzji
z raportu dedup (docs/kb/modules/05-fallback-dedup-raport.md): plan §2 D6/§9
wymaga trybu HTTP-equivalence, a master go nie miał — e7625cd nie tknął tego
pliku, patch aplikuje się czysto i woła wyłącznie GET /search (pola
envelope_id/dist/source zgodne z odpowiedzią mastera). Live-PASS 2026-07-27
na kodzie brancha; smoke na masterze: CLI + pełny przebieg http przeciwko
stubowi /search (raport i werdykt bramki generują się poprawnie).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recon read-only przed salvage (moduł 5 faza 4 §2/§5): embed_router.py na
masterze pokrywa 100% zachowań fallback.py z porzucanego brancha i dodaje
weryfikację modelu per backend, mapowanie 503/500 i tryb bez fallbacku.
Do uratowania: --transport http w retrieval_eval.py (plan §2 D6/§9, czysty
cherry-pick), session log 2026-07-27 (kalibracja GO, osierocony ollama.service),
luki testowe T1/T2, wynik kalibracji do override/README ollama-piha.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The group_add fix itself already landed on master (ddae57c) but the running
container still has GroupAdd=[999] — deploy pending from main checkout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RECON appendix summarizes the whole etap-0 cleanup (topology dormant status,
hosts reconciliation, dead-code removal, legacy decision), carries the
runbook line for the stale chelsty pending action
(rm /opt/homelab/actions/pending/redeploy-chelsty-infra-ha-diag-agent.json —
verified the only chelsty-targeted pending file 2026-07-30), and lists
discrepancies found but out of scope: lustro watchtower, stale solaria/chelsty
world-state keys, piha naming-drift duplicate keys, gokapi still down with a
stuck pending redeploy, outline/joplin/umami still lacking services/ dirs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Etap 0 truth cleanup (recon B5/B6/F20), all facts re-verified 2026-07-30 via
read-only ssh (docker ps, world/services.json, events store):
- vps: add stability-agent, npm, outline, joplin-server, umami (all running
and healthy in world state). joplin keyed joplin-server to match the
world-state key — an aggregate 'joplin' entry would drift as
missing_service forever. humanai-mailer/humanai-landing documented as
unmanaged: true + monitor: false (hand-run, no compose labels — supervisor
must not queue dead-end redeploys for them). ai-cluster deliberately gets
NO entry: legacy, retiring — comment points at ai-cluster-LEGACY.md.
- piha: comment block enumerating the ~28 known unmanaged containers plus the
host systemd mosquitto; bringing them in is a later stage (recon OQ5).
- saturn: create services.yaml with explicit empty list (dev workstation /
orchestrator, nothing monitored — closes recon F20.4).
- lustro: add node-exporter (hyphenated key = actual container name) and
piper-tts (runs locally on the Pi, no services/ dir); watchtower noted as
deliberately unmanaged.
All YAMLs parse (yaml.safe_load); control-plane suite 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified 2026-07-30 in the vps event store: lustro's node_offline events fire
once a day, ~24 h apart (~23:30 local), matching solaria's scheduled power-off
pattern. Documented so daily stale/offline/online cycles read as expected
behavior, not faults.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
healthcheck_failed incidents fell through to redeploy, which is broken as
wired (executor calls deploy-node.sh with arguments it ignores, at a path
that does not exist in the container) — so 3376 healthcheck_failed events
dead-ended with no working remediation (recon D14/D15). A container restart
plausibly heals a failing healthcheck and rides the executor path that
actually works; redeploy returns to the map once etap 2 fixes the executor.
service_unhealthy / deployment_failed / missing_service stay on redeploy —
theoretical until etap 2, kept so drift remains visible in pending actions
(noted in comments). CLAUDE.md routing table updated to match; stale
mqtt_unreachable example in the observer's trigger_type comment refreshed.
Tests: trigger-type recognition and the end-to-end observer→supervisor
reconcile test parametrized over both container_restart triggers, with an
assertion that no redeploy action is also generated. Full control-plane
suite: 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The aggregate containers_not_running event carried service=None, which the
observer skips when building service state and incidents — stability-agent's
flagship signal never opened an incident (recon D15). Emit one event per
non-running container instead, tagged with the compose service name from the
com.docker.compose.service label (same pattern as node-agent's
_canonical_container_name fix from May), falling back to the container name
with Docker's stale-state hash prefix stripped; never crashes on unlabeled
containers. 'created' compose tracking artifacts are skipped — they are not
running services and would open fake incidents now that the event is
actionable.
Adds the service's first test suite covering the label-extraction helper.
Smoke-run performed with runtime paths redirected (no docker build, authoring
only): main loop runs, service names resolve on live solaria containers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base compose assumes Debian-default docker gid 999; on SOLARIA the docker
group is 996, so node-agent hit 'Docker unavailable: Permission denied' on
the socket and reported no containers (recon A2/E19). Same per-host
override pattern as piha (123) and lustro (991). Verified read-only on the
host: getent group docker -> docker❌996:oskar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follows the supervisor change: mqtt_unreachable never becomes an
incident (recon D15), so the documented container_restart routing was
false. Table now matches code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bus codex/* dead since 2026-06-09 (recon C9). Records what survives into
subsystem B (patterns only: task bus, worker roles, enforced command
allowlists, preview mode), that task/ai-cluster-solaria stays unmerged
as documentation, and that the vps containers get stopped in a separate
supervised session.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The observer never creates incidents with trigger_type=mqtt_unreachable,
so the CONTAINER_RESTART_TRIGGERS branch for it could never fire (recon
D15). stability-agent keeps emitting the event; it just never becomes an
incident. Tests: 145 passed.
SERVICE_NAMES in ai-cluster's service_ops_worker.py (the other dead
constant from the plan) is NOT touched: that code is legacy-frozen in
the unmerged task/ai-cluster-solaria worktree and nothing on this branch
references it (verified by grep — only the recon and plan docs mention it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The script composes from roles/$ROLE/compose/docker-compose.yml, but no
roles/ directory exists anywhere in the repo (recon B7) and nothing
references the script. Dead since the roles concept was abandoned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oskar Kapala