Read-only import of the pre-migration homeassistant5 container (piha),
per DESIGN.md phase-0 scope: 60 automations (split one-file-per-id),
5 scenes, 1 script, plus curated .storage/* export (area/entity
registries, input_boolean/input_text, lovelace dashboards).
No token at ~/.config/ha-deploy/ken-legacy.token — fixtures step
fail-softed as designed, no repo import.sh changes needed. Verified:
gitignore-excluded paths (secrets.yaml, *.db*, logs, .storage/auth*,
core.restore_state, tts/, deps/, backups/, .cloud/) correctly stripped
from the pull; three consecutive import.sh runs produced byte-identical
output under config/ken-legacy/ and storage-export/ken-legacy/
(idempotent); automation count matches the 2026-07-22 recon estimate
(~60) exactly. README.md only touched to note the import date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Native ESPHome gree climate component over the API, no MQTT/Broadlink/Tuya.
HA on piha runs as a plain Docker container (not HAOS), so ESPHome is a
manual docker compose run --rm invocation, not a managed service — device
is deliberately not registered in hosts/piha/services.yaml.
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
instances.yaml pointed "ken" at the piha container (homeassistant5), a
pre-migration instance that never actually stopped running — it kept
firing automations in parallel with the real home instance (HAOS on a
dedicated RPi4, 192.168.31.7), causing duplicate triggers on shared
buttons/scenes. Repoint ken to the real instance (api adapter),
demote the container to ken-legacy (archived, import-only, never a
deploy target), and log the incident in DESIGN.md.
Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first
user-facing HTTP entry point to the KB. FastAPI wrapping
kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via
Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source
links) and GET /healthz. Search API only, no answer synthesis (phase 5) and
no server-side dist filtering — the 0.45/0.55 colour thresholds are a
frontend concern (plan §7, a later step).
Hard startup invariant (plan §2 decision 2): refuses to start unless the
configured EMBED_MODEL is present in both document_chunk.model and
document_summary.embedding_model. Note the latter: document_summary.model is
the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the
embedder — checked live against kb-postgres@PIHA before writing this, see
app/startup.py's docstring. Verified end-to-end with a live docker run: the
invariant crash-loops on a mismatched EMBED_MODEL and passes through to a
real /search hit against the live corpus with a correct model.
Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local
embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and
a failure surfaces as 503, not a crash.
Also: scripts/deploy/deploy.sh's gate now builds each service via
`docker compose build` instead of a raw `docker build <svc_dir>`, so a
service whose docker-compose.yml declares a repo-root build context (needed
here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention
already documented in CLAUDE.md) resolves the same way in the gate as it
does at real deploy time (deploy-node.sh's `docker compose ... up --build`).
No behavior change for existing single-context services — verified against
llm-gateway's compose file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 4 step 0 (docs/kb/modules/05-faza4-plan.md, §3, decision 1):
kb-query is a long-lived Docker service, documents-ingest is a venv job with
an `anthropic` dependency and CLI scripts it doesn't need. Move
embed_chunk/_vector_literal/cascade_query/flat_query into a shared package
with minimal deps (asyncpg, aiohttp only) so both can depend on the same
tested retrieval code without the service image pulling in the job's extras.
documents_ingest.chunk_embed/retrieval keep thin re-exports so nothing
importing the old paths breaks. Pure refactor: retrieval_eval.py run live
against kb-postgres@PIHA + Ollama@SOLARIA before/after gives byte-identical
`dist`/hit@3/gate results (still PASS) — zero behavior change in the cascade.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yaml.safe_load has no constructor for HA's `!include`/`!secret`/etc. tags
and raised ConstructorError on any real configuration.yaml. Add a
multi_constructor on `!` that generically wraps scalar/sequence/mapping
payloads in a TaggedValue, and a matching representer that re-emits the
same tag + payload — normalization stays idempotent across passes and key
sorting is unaffected by tagged values.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py):
orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic,
claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA
(availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu
etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na
kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał
tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla
cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem.
Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors,
llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie
blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis,
last_success_timestamp trzymany z poprzedniego pliku przy failu) do
/opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu).
systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer
(OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User
oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do
/opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w
README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge.
fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical)
+ KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem
brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml).
node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override
(--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) +
topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter
owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent
zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA.
Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run
złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym
taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń,
0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY
dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany).
Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge),
zdecydowanie czy stability-agent też idzie na per-host przy okazji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause rozjazdu panelu agents.okit.pl vs services.json NIE lezal w
observer.py/operator_ui.py (te sa poprawne: ghost-prune dziala in-memory
i na plik co cykl, operator_ui czyta services.json na zywo bez cache).
Prawdziwy lancuch: npm proxy_host agents.okit.pl -> 100.108.208.3:18180
(Tailscale IP PIHA, nie VPS!) -> legacy kontener agent-system-webui
(services/agent-system/, sprzed migracji do control-plane/, brak
service.yaml, nieobecny w hosts/piha/services.yaml i topology.yaml).
Jego runtime-materializer czytal z Redis (homelab:services:*), do ktorego
NIC juz nie pisze w obecnej architekturze -- stad 27 martwych/ghost
wpisow (w tym hash-prefixed) i inna liczba serwisow (110 vs 117 w
services.json z VPS).
Fix na to juz istnial w repo od 2026-05-27 (7277bdc): materializer.py ma
materialize_from_api() ktory mirroruje czysty output observera z VPS przez
CONTROL_PLANE_URL, a hosts/piha/runtime/agent-system/docker-compose.override.yml
ustawia ta zmienna. Nigdy sie jednak nie aktywowal, bo
services/agent-system/deploy.sh (jedyna sciezka deployu tego serwisu)
wolal `docker compose up` uzywajac WYLACZNIE docker-compose.yml, bez
dolaczania override'u z hosts/ -- w odroznieniu od control-plane/
deploy-local.sh i stability-agent/deploy-local.sh, ktore ten wzorzec juz
stosuja.
Fix: deploy.sh dolacza teraz hosts/piha/runtime/agent-system/docker-compose.override.yml
(ten sam wzorzec co control-plane, ktory hardkoduje vps). Po nastepnym
`services/agent-system/deploy.sh` na PIHA runtime-materializer zacznie
mirrorowac /nodes /services /summary itd. z control-plane API zamiast
Redis -- panel bedzie pokazywal to samo co services.json.
Docker socket PIHA (DOCKER_API_ERROR 2026-07-17T00:00:40Z): NIE regres
group_add/gid (node-agent ma poprawne grupy 999/123, gid docker.sock=123
sie zgadza). Wszystkie ~40 kontenerow na PIHA wystartowaly jednoczesnie o
00:00:26 UTC -- to byl automatyczny apt-get upgrade docker-ce
29.6.1->29.6.2 (apt/history.log, Start-Date 02:00:08 CEST), ktory
zrestartowal Docker Engine. stability-agent (root) trafil na gniazdo w
~1-sekundowym oknie zanim daemon w pelni wstal (docker.service
ActiveEnterTimestamp 02:00:41 CEST). Jednorazowy, samo-naprawiony,
zdarzenie sie nie powtorzylo. Brak zmiany kodu.
Testy: ast.parse (observer.py, operator_ui.py, materializer.py) OK,
bash -n deploy.sh OK, docker compose config (merge z override) poprawnie
wstrzykuje CONTROL_PLANE_URL, pytest services/control-plane/tests 114
passed, pytest services/agent-system/telegram-bot/tests 8 passed.
Co NIE zrobiono (poza zakresem/deploy nalezy do operatora): rzeczywisty
redeploy agent-system na PIHA + weryfikacja panelu na zywo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents_ingest.retrieval: flat_query (baseline) i cascade_query (stage1
document_summary model='claude-haiku-4-5' -> stage2 document_chunk), jedno
dzielone wywołanie embeddingu bge-m3 per zapytanie, tylko +1 SQL na kaskadę.
Czyste query_text -> wyniki(dist, source) pod przyszłe kb-query fazy 4.
166/166 testów (10 nowych, mocki: stage1->stage2, koperta bez chunków,
N > liczba kopert, no-summaries short-circuit).
Eval-set utrwalony 1:1 z pilota (docs/kb/eval/retrieval-pilot-2026-07-16.md,
nietknięty) w eval/queries.yaml + skrypt bramki eval/retrieval_eval.py
(integracyjny, read-only, poza pytest).
Wynik bramki (żywa baza, N=10 k=5): kryterium 1 (brak degradacji) PASS,
kryterium 2 (hit@3 kaskada=5/5 vs płaski=5/5) PASS, kryterium 3 (kontrole
negatywne 0.644/0.553 > 0.55 w obu torach) PASS. Sweep N∈{1,2,3,5,10,20}:
N=5 to zmierzony próg bezpieczny (N<5 degraduje zapytania 3-4), N=10 ma
2x margines — potwierdza domyślną wartość z planu zamiast przyjmować ją
z założenia. Kaskada nie poprawia jakości na 186-dok. korpusie (dystanse
identyczne z płaskim przy N≥5) — zgodnie z przewidywaniem planu: to test
architektury pod skalę mailową, nie optymalizacja pilota.
Decyzja: kaskada (N=10, k=5, claude-haiku-4-5) = domyślna ścieżka retrievalu.
Plan-doc §6.3 zaktualizowany wynikiem; §2 D3 zamknięte rozstrzygnięciem
Oskara (tor kompilacyjny=claude-haiku-4-5, gemma3:12b w odwodzie, decyzja
mailowa odłożona do reconu z flagą prywatności/kosztu).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migracja 004: document_summary (envelope_id, summary, tags JSONB, model, embedding
VECTOR(1024) + HNSW cosine, embedding_model, UNIQUE(envelope_id, model) od razu — wzorzec
002/003). Zastosowana na żywej bazie kb-postgres@PIHA.
Job documents-ingest-summarize: wejście = document_chunk.text WHERE excluded_reason IS NULL
per koperta source='paperless' (duplikaty przez entities[duplicate_of] pomijane w całości),
wymuszony JSON {summary, tags} przez --backend ollama|anthropic, słownik tagów kontrolowany
(tags-vocab.yaml) z max 3 free-form, map-reduce dla dokumentów >200k znaków (grupy ~20
chunków), bilans + idempotencja + izolacja błędów per wiersz wg wzorców rodziny jobów.
Osobny --embed-summaries (bge-m3, reużywa chunk_embed.embed_chunk).
Bug znaleziony i naprawiony w trakcie pilota: brak options.num_ctx w wywołaniach Ollamy
powodował, że gemma3:12b używał domyślnego runtime kontekstu (~2048 tok), nie
zadeklarowanego 128k — dla 71/157 dokumentów (45%, >8k znaków) treść była cicho ucinana
(zweryfikowane: prompt_eval_count=2051 dla dokumentu 93k znaków). Naprawa: compute_num_ctx()
liczy num_ctx z długości promptu (~3 znaki/token, cap 131072). Cały tor lokalny przeliczony
od zera po naprawie.
Wynik pilota (186 dok. paperless, 3 duplikaty, 26 bez aktywnych chunków → 157 oczekiwanych
na tor): tor lokalny (gemma3:12b) 155/157 (2 izolowane błędy JSON po retry: paperless:24,
paperless:61), tor referencyjny (claude-haiku-4-5) 157/157, 0 błędów JSON, 0 tagów
ucinanych — słownik przestrzegany w 100% przypadków. Oba komplety zembedowane (bge-m3).
Znaleziony przy okazji: prompt do tagów wymagał dopracowania — pierwsza wersja pozwalała
modelowi zwracać tagi po angielsku spoza słownika; wzmocniona instrukcja (słownik w
pierwszej kolejności, "nigdy po angielsku") poprawiła zgodność w 2/3 przypadków testowych.
Porównanie A/B (~15 dok.) i weryfikacja końcowa (bilans, sanity SQL, retrieval po summary)
odłożone do następnej sesji.
Testy: 157 (152 nowe/summarize.py + istniejące), mocki API/Ollama/DB, bilans, idempotencja,
regresja num_ctx.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (sposób podania
klucza API, wybór modelu lokalnego gemma3:12b, naprawa+przeliczenie całego toru lokalnego po
odkryciu buga num_ctx) — udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- node-agent: service_healthy emitowany tylko przy przejsciu w stan zdrowy
(per-service in-memory state), nie co cykl dla kazdego zdrowego serwisu.
To samo dla control-plane HTTP probe. healthcheck_failed/containers_not_running/
incydenty pozostaja emitowane bez zmian (realne sygnaly).
- node-agent: naprawiono _cleanup_control_plane_fs — istniejaca retencja
eventow byla martwa od migracji observer_checkpoint.json na epoch-int
(str(path) <= int rzucal TypeError, lapane cicho przez broad except).
Teraz porownuje epoch-do-epoch i czysci wylacznie service_healthy/node_health
starsze niz checkpoint + 3-dniowy bufor; healthcheck_failed/incydenty/ha_*
zachowane bezterminowo.
- scripts/maintenance/cleanup_event_backlog.py: jednorazowy skrypt do
bezpiecznego czyszczenia backlogu na VPS (dry-run domyslnie, --apply
do usuniecia). Ten sam warunek: typ szumu + starsze niz checkpoint + 1h bufor.
Migracja 003 (UNIQUE+model, excluded_reason) zastosowana na żywej bazie kb-postgres@PIHA
(2683 chunki, bez DELETE). Kalibracja heurystyki ocr_junk (3 sygnały z planu §3.1) ujawniła
realną sprzeczność z planem: sygnał 1 (dowolny znak kontrolny) fałszywie łapał paperless:119
(wymagany aktywny) i legalny angielski tekst — próg doprecyzowany do >=5 wystąpień na
podstawie rozkładu na korpusie. 8 chunków oflagowanych ocr_junk po odjęciu potwierdzonych
fałszywych alarmów (kalendarz, mikro-fragmenty referencyjne).
Dedup: SQL z planu (exact content hash) znalazł 2 pary, ale nie wykrył znanego z pilota
duplikatu paperless:14≡74 (różne OCR, 99.2% chunków identycznych treściowo) — dodany fuzzy
check na potwierdzenie. Odrzucono 3 kandydatury o wysokim nakładaniu jako różne
wersje/typy dokumentów dzielące boilerplate PZU, nie duplikaty. 130 chunków oflagowanych
duplicate + entities[duplicate_of] na 3 kopertach.
chunk_embed.py: heurystyka is_ocr_junk() przed embedem (junk -> insert bez wywołania Ollamy,
embedding=NULL), ON CONFLICT rozszerzony o model, nowy licznik chunks_junk_flagged w bilansie,
testy (kody kreskowe, mojibake nie-junk, dot-leader, idempotencja, model w kluczu konfliktu).
Weryfikacja: 7 zapytań eval-setu z WHERE excluded_reason IS NULL — żadne trafienie nie
degraduje, kontrole negatywne bez zmian (>0.55), śmieć zniknął z top-5 zapytania 2,
paperless:119 pozostał aktywnym trafieniem. Bilans: 2545 aktywne / 130 duplicate / 8 ocr_junk.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (kalibracja
progu sygnału 1, dołączenie 14≡74 mimo braku exact-hash matcha, odrzucenie 3 fałszywych
kandydatur dedup) — plan przewidywał, że heurystyka będzie się mylić; wszystkie decyzje
udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-embed: chunks source='paperless' envelope content
(paragraph-preferring, ~600 tok/chunk, ~150 tok overlap, hard char-fallback
for oversized paragraphs per plan §2 decision 3), embeds each chunk via
Ollama (bge-m3, dim validated against document_chunk's VECTOR(1024) on
every response) and inserts into document_chunk. Lives in documents-ingest
per the plan's own recommendation (§6 step 6) rather than a new package —
reuses the job family's existing idempotency/stats-balance/dry-run
conventions (paperless_adapter.py, gmail-header-backfill).
A 5-angle multi-agent code review of the initial implementation surfaced
three real bugs, fixed here: hard_split() could infinite-loop if
--chunk-overlap >= --chunk-size (now guarded in both hard_split() and
main()); insert_chunk() wasn't error-isolated like embed_chunk(), so a DB
write failure would crash the whole run instead of being counted and
skipped; and ON CONFLICT DO NOTHING's outcome was discarded, so a silently
skipped row (the known gap where document_chunk's UNIQUE constraint
doesn't include `model`) would have been miscounted as a successful insert
- now tracked separately as chunks_conflict_skipped and treated as a
run failure.
Smoke-tested and run to completion live on SOLARIA against the real Ollama
instance and kb-postgres@PIHA: dry-run matched the known phase-2-step-5
figures exactly (186 fetched, 26 empty_content, 2684 chunks planned), a
--limit 10 apply + idempotent re-run + DB/distance sanity checks all
passed, and the full 186-document run inserted 2683/2684 chunks (1 isolated
error - Ollama's runtime context window rejected one pathological
dot-leader table-of-contents chunk that tokenized far more densely than
estimated; documented as a known limitation, not fixed here given it's a
single-chunk edge case). Timing: ~0.79s/chunk average on CPU (SOLARIA's
Ollama runs GPU-less per the recent GPU-reservation-disabled fix), ~35 min
wall-clock for the full pilot - the real input for scoping the later
mail-corpus embedding phase (plan §7's GPU-based estimate doesn't hold
here).
pytest: 101 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sesja: analiza Etapu 2 (Fable) rekomenduje GO dla wszystkich 4 węzłów; 165
mismatchy solaria/lustro wyjaśnione jako detection-lag TTL vs scrape, nie bug.
Trwały log SHADOW_LIVENESS_MISMATCH zweryfikowany testem kontrolowanym na
lustro. Backlog: ha-diag-agent node=unknown oznaczone ZROBIONE (f2ba81b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An "unknown"/empty node_name silently produced evt-unknown-* event files. One
landed in events/piha/ and, being lexically greater than every evt-piha-* name,
pinned the observer's (then lexical) checkpoint and blocked PIHA for ~34 days.
Defense in depth, two layers, both refuse rather than emit poison:
- config.py: node_name field_validator rejects ""/"unknown" (case-insensitive,
trimmed); validate_default=True so the "unknown" default itself is rejected
when NODE_NAME never reaches the process. main() catches ValidationError and
exits 1 with a clear FATAL message instead of a raw traceback.
- event_emitter.py: EventEmitter.__init__ guards node_name at the exact poison
site (node_name is embedded in the filename), so no future call path can
reintroduce evt-unknown-*.
Precedence unchanged and correct: Settings.load() uses os.environ.setdefault, so
env NODE_NAME wins over YAML; YAML only supplies node_name when env is absent.
api.py never writes event files (only /health, /trigger), so its _node_name
global cannot produce poison — verified, left as-is.
Tests (services/ha-diag-agent/tests/): new test_config.py (env load, env>YAML
precedence, fail-fast on unknown/empty/whitespace/default/YAML-unknown) and
EventEmitter guard tests. Full unit suite: 139 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live cutover 2026-07-15 odkrył: nvidia-smi not found, brak libnvidia-ml
w ldcache — host nigdy nie miał sterownika, natywna Ollama mieliła na CPU.
Sekcja deploy.resources (nvidia) zakomentowana do czasu instalacji sterownika
(backlog). nvidia-container-toolkit doinstalowany na hoście (prerequisite,
brakował w runbooku). Kontener zweryfikowany: 5/5 modeli z bind-mounta,
embed bge-m3 dim=1024, reachable z PIHA przez tailnet.
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-paperless: paginated GET /api/documents/, maps each
doc to a source='paperless' envelope per plan §4.2-4.3, reusing
kb_mail.Envelope/insert_envelope unchanged (packages/kb-mail not touched).
Cross-source link (source_mail entity) is a deterministic join of
original_file_name against the phase-1 registry.json consume_name index —
no heuristics, no correspondent guessing (plan decision 4). Stats always
balance (fetched = already_in_db + inserted + errors) and main() now also
exits non-zero on imbalance, not just on errors>0, matching the exit-code
convention already established in gmail-bulk-import.
Verified live on PIHA (rsync to /tmp, ~/kb/venv, PIHA checkout untouched):
dry-run then --apply inserted 186/186 paperless envelopes (0 errors,
180 source_mail links), a second --apply reported inserted=0/already_in_db=186
(idempotent), gmail rows stayed at 225030 and document_chunk stayed empty.
Rotated the kb-ingest Paperless API token after it was accidentally
partially echoed during recon (old token now dead).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wszystkie mismatche = detection-lag przy planowych power-offach (prom szybszy
o ~9.5 min od TTL eventowego); zero fałszywych prom=down. vps/piha 100% zgodne.
Rekomendacja: GO dla etapu 3 per-node, mapping timestamp(up)->compute_liveness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four audit findings (2026-07-14), each reproduced on crafted mboxes; the
Takeout corpus has proven 8-bit header bytes, so all are real re-import risks.
1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an
email.header.Header (not str) for raw 8-bit bytes; the old
Header.strip() raised AttributeError. _message_id ran BEFORE the
per-message try, so one bad header killed the entire import.
Fix: str() + sanitize_surrogates() before strip; and move
_message_id/_parse_date/_parse_attachments INSIDE the per-message try —
a broken message is now errors += 1, never run death.
2. Poison batch. pending.clear() ran only AFTER a successful insert, so a
failed flush (DB down / bad row) left pending intact and every later
message re-flushed the doomed batch; the final flush sat in try/finally
with no except and propagated out, losing all stats. Fix: _flush always
clears pending and counts a failed insert as db_insert_failed; the run
always reaches import_complete.
3. Stats didn't reconcile with the DB. imported counts archive writes, not
DB rows, so a partial-insert drift was invisible. Fix: separate
db_inserted/db_insert_failed counters; main() exits non-zero on any
error, DB drift, or a processed = imported + skipped + errors imbalance.
4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header)
raised even when str(header) parses fine. Fix: str() before the epoch
fallback.
Shared helper: _sanitize moved from gmail-header-backfill into
packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs;
gmail-header-backfill now depends on kb-mail.
Tests: regression coverage for all four findings in gmail-bulk-import
(8-bit id/date, per-message guard, failed-insert non-poisoning, stats
balance) plus kb_mail.text unit tests. Full suites green:
kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
check_containers() classified only exited/dead, running+unhealthy and running.
Docker's "restarting", "paused", "removing" and any future state matched no
branch, so a crash-looping container (restart policy + continuous crash) emitted
ZERO events and world-state kept showing it "healthy" (observed live:
lustro/pi-watchtower-1). Every Docker state is now handled:
- restarting + RestartCount >= CRASH_LOOP_RESTART_THRESHOLD (default 3):
reuse containers_not_running (high, crash_loop=true) — parity with exited/dead,
rides the existing supervisor-wired remediation path.
- restarting below threshold: new observational container_restarting (low),
visible but non-actionable so benign post-deploy restarts don't alarm.
- paused / unknown-or-future state: new diagnostic container_state_unexpected
(medium) — no more silent fall-through; new Docker states become visible.
- removing: conscious documented skip (ephemeral teardown).
- created: unchanged skip (compose tracking artifact).
RestartCount (top-level inspect field) distinguishes a crash-loop from a one-off
restart. Adds services/node-agent/tests/test_check_containers.py pinning the full
state table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diagnosis of the 5008 header-less envelopes after the full 225 030-row run
(read-only, on PIHA, 2026-07-14):
- 4999 = one contiguous block at ORDER BY id positions 70001-74999: the
--offset 70000 slice died mid-run. Root cause reproduced: a Date: header
with raw 8-bit bytes makes compat32 .get() return email.header.Header
(not str), and json.dumps([headers]) — which sat OUTSIDE the per-row
try/except — raised TypeError and killed the process, silently losing
the rest of the slice.
- 9 = genuine typed-parse failures: 7x RFC 2047 encoded-word decoding to
CR/LF inside a display name (ValueError in headerregistry), 1x RFC 5322
group syntax in To: ("unlisted-recipients:;"), 1x CPython
_header_value_parser bug on a malformed display name (fixed upstream,
present on PIHA's 3.11).
- 0 missing .eml files.
Fixes:
- date_raw: str() + surrogate sanitization on the compat32 value — the
crash cause, now also covered by a regression test.
- json.dumps moved inside the per-row try: a non-serializable value counts
as that row's parse_error instead of crashing the slice.
- parse_headers_fallback(): on typed-parse failure retry with a pure
compat32 parse — getaddresses over raw header text, raw-string values,
same §4.1 entity shape. Counted separately as parsed_fallback (labeled
subset of updated), logged per row with the original typed error.
- missing_file counter + skip.missing_file info log (id, expected path);
run_complete now balances: scanned = updated + already_has_headers +
parse_errors + read_errors + missing_file. Non-zero missing_file also
fails the exit code.
Verified: 43/43 pytest locally (3.13) and on PIHA (3.11); read-only dry
runs on PIHA — all 9 parse failures recover via fallback, the lost slice
completes scanned=5000 updated=4999 already_has_headers=1 with zero errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Module 5 phase 2 (docs/kb/modules/05-faza2-plan.md, §5). 225,030 existing
source='gmail' envelope rows carry only an attachment manifest — no
from/to/cc/delivered_to/subject anywhere in the DB, which blocks answering
"is this on me or my wife" / distinguishing aliases (plan §1.8). Separate
job from gmail-bulk-import per plan §2 decision 7: UPDATE semantics against
production data is a different risk profile than the historical INSERT job.
Parses headers only (no MIME-walk of attachments) from the archived .eml
files, appends {"type": "headers", ...} (plan §4.1) via the idempotent
UPDATE ... WHERE NOT EXISTS from §5.2, batched via executemany. --limit/
--offset (ORDER BY id) give stable, deterministic partitioning so the full
backfill can run and be verified in slices instead of one unattended pass.
Plain CLI (pip install -e), no Dockerfile — same convention as
gmail-bulk-import/documents-ingest, which run directly on PIHA for local
filesystem access to the .eml archive.
Verified against kb-postgres@PIHA (100-row dry-run + apply, re-run proved
idempotent no-op, 1000-row timed slice): 1096/225030 rows backfilled,
996/1000 succeeded on the timed slice (4 parse_errors — a 2014 spam message
with an RFC 2047 encoded-word decoding to an embedded newline in the From
display name, correctly caught and skipped rather than crashing the batch).
Extrapolated full-run time ~16 minutes. Full 225,030-row run is out of
scope for this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
