Root cause rozjazdu panelu agents.okit.pl vs services.json NIE lezal w
observer.py/operator_ui.py (te sa poprawne: ghost-prune dziala in-memory
i na plik co cykl, operator_ui czyta services.json na zywo bez cache).
Prawdziwy lancuch: npm proxy_host agents.okit.pl -> 100.108.208.3:18180
(Tailscale IP PIHA, nie VPS!) -> legacy kontener agent-system-webui
(services/agent-system/, sprzed migracji do control-plane/, brak
service.yaml, nieobecny w hosts/piha/services.yaml i topology.yaml).
Jego runtime-materializer czytal z Redis (homelab:services:*), do ktorego
NIC juz nie pisze w obecnej architekturze -- stad 27 martwych/ghost
wpisow (w tym hash-prefixed) i inna liczba serwisow (110 vs 117 w
services.json z VPS).
Fix na to juz istnial w repo od 2026-05-27 (7277bdc): materializer.py ma
materialize_from_api() ktory mirroruje czysty output observera z VPS przez
CONTROL_PLANE_URL, a hosts/piha/runtime/agent-system/docker-compose.override.yml
ustawia ta zmienna. Nigdy sie jednak nie aktywowal, bo
services/agent-system/deploy.sh (jedyna sciezka deployu tego serwisu)
wolal `docker compose up` uzywajac WYLACZNIE docker-compose.yml, bez
dolaczania override'u z hosts/ -- w odroznieniu od control-plane/
deploy-local.sh i stability-agent/deploy-local.sh, ktore ten wzorzec juz
stosuja.
Fix: deploy.sh dolacza teraz hosts/piha/runtime/agent-system/docker-compose.override.yml
(ten sam wzorzec co control-plane, ktory hardkoduje vps). Po nastepnym
`services/agent-system/deploy.sh` na PIHA runtime-materializer zacznie
mirrorowac /nodes /services /summary itd. z control-plane API zamiast
Redis -- panel bedzie pokazywal to samo co services.json.
Docker socket PIHA (DOCKER_API_ERROR 2026-07-17T00:00:40Z): NIE regres
group_add/gid (node-agent ma poprawne grupy 999/123, gid docker.sock=123
sie zgadza). Wszystkie ~40 kontenerow na PIHA wystartowaly jednoczesnie o
00:00:26 UTC -- to byl automatyczny apt-get upgrade docker-ce
29.6.1->29.6.2 (apt/history.log, Start-Date 02:00:08 CEST), ktory
zrestartowal Docker Engine. stability-agent (root) trafil na gniazdo w
~1-sekundowym oknie zanim daemon w pelni wstal (docker.service
ActiveEnterTimestamp 02:00:41 CEST). Jednorazowy, samo-naprawiony,
zdarzenie sie nie powtorzylo. Brak zmiany kodu.
Testy: ast.parse (observer.py, operator_ui.py, materializer.py) OK,
bash -n deploy.sh OK, docker compose config (merge z override) poprawnie
wstrzykuje CONTROL_PLANE_URL, pytest services/control-plane/tests 114
passed, pytest services/agent-system/telegram-bot/tests 8 passed.
Co NIE zrobiono (poza zakresem/deploy nalezy do operatora): rzeczywisty
redeploy agent-system na PIHA + weryfikacja panelu na zywo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents_ingest.retrieval: flat_query (baseline) i cascade_query (stage1
document_summary model='claude-haiku-4-5' -> stage2 document_chunk), jedno
dzielone wywołanie embeddingu bge-m3 per zapytanie, tylko +1 SQL na kaskadę.
Czyste query_text -> wyniki(dist, source) pod przyszłe kb-query fazy 4.
166/166 testów (10 nowych, mocki: stage1->stage2, koperta bez chunków,
N > liczba kopert, no-summaries short-circuit).
Eval-set utrwalony 1:1 z pilota (docs/kb/eval/retrieval-pilot-2026-07-16.md,
nietknięty) w eval/queries.yaml + skrypt bramki eval/retrieval_eval.py
(integracyjny, read-only, poza pytest).
Wynik bramki (żywa baza, N=10 k=5): kryterium 1 (brak degradacji) PASS,
kryterium 2 (hit@3 kaskada=5/5 vs płaski=5/5) PASS, kryterium 3 (kontrole
negatywne 0.644/0.553 > 0.55 w obu torach) PASS. Sweep N∈{1,2,3,5,10,20}:
N=5 to zmierzony próg bezpieczny (N<5 degraduje zapytania 3-4), N=10 ma
2x margines — potwierdza domyślną wartość z planu zamiast przyjmować ją
z założenia. Kaskada nie poprawia jakości na 186-dok. korpusie (dystanse
identyczne z płaskim przy N≥5) — zgodnie z przewidywaniem planu: to test
architektury pod skalę mailową, nie optymalizacja pilota.
Decyzja: kaskada (N=10, k=5, claude-haiku-4-5) = domyślna ścieżka retrievalu.
Plan-doc §6.3 zaktualizowany wynikiem; §2 D3 zamknięte rozstrzygnięciem
Oskara (tor kompilacyjny=claude-haiku-4-5, gemma3:12b w odwodzie, decyzja
mailowa odłożona do reconu z flagą prywatności/kosztu).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migracja 004: document_summary (envelope_id, summary, tags JSONB, model, embedding
VECTOR(1024) + HNSW cosine, embedding_model, UNIQUE(envelope_id, model) od razu — wzorzec
002/003). Zastosowana na żywej bazie kb-postgres@PIHA.
Job documents-ingest-summarize: wejście = document_chunk.text WHERE excluded_reason IS NULL
per koperta source='paperless' (duplikaty przez entities[duplicate_of] pomijane w całości),
wymuszony JSON {summary, tags} przez --backend ollama|anthropic, słownik tagów kontrolowany
(tags-vocab.yaml) z max 3 free-form, map-reduce dla dokumentów >200k znaków (grupy ~20
chunków), bilans + idempotencja + izolacja błędów per wiersz wg wzorców rodziny jobów.
Osobny --embed-summaries (bge-m3, reużywa chunk_embed.embed_chunk).
Bug znaleziony i naprawiony w trakcie pilota: brak options.num_ctx w wywołaniach Ollamy
powodował, że gemma3:12b używał domyślnego runtime kontekstu (~2048 tok), nie
zadeklarowanego 128k — dla 71/157 dokumentów (45%, >8k znaków) treść była cicho ucinana
(zweryfikowane: prompt_eval_count=2051 dla dokumentu 93k znaków). Naprawa: compute_num_ctx()
liczy num_ctx z długości promptu (~3 znaki/token, cap 131072). Cały tor lokalny przeliczony
od zera po naprawie.
Wynik pilota (186 dok. paperless, 3 duplikaty, 26 bez aktywnych chunków → 157 oczekiwanych
na tor): tor lokalny (gemma3:12b) 155/157 (2 izolowane błędy JSON po retry: paperless:24,
paperless:61), tor referencyjny (claude-haiku-4-5) 157/157, 0 błędów JSON, 0 tagów
ucinanych — słownik przestrzegany w 100% przypadków. Oba komplety zembedowane (bge-m3).
Znaleziony przy okazji: prompt do tagów wymagał dopracowania — pierwsza wersja pozwalała
modelowi zwracać tagi po angielsku spoza słownika; wzmocniona instrukcja (słownik w
pierwszej kolejności, "nigdy po angielsku") poprawiła zgodność w 2/3 przypadków testowych.
Porównanie A/B (~15 dok.) i weryfikacja końcowa (bilans, sanity SQL, retrieval po summary)
odłożone do następnej sesji.
Testy: 157 (152 nowe/summarize.py + istniejące), mocki API/Ollama/DB, bilans, idempotencja,
regresja num_ctx.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (sposób podania
klucza API, wybór modelu lokalnego gemma3:12b, naprawa+przeliczenie całego toru lokalnego po
odkryciu buga num_ctx) — udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- node-agent: service_healthy emitowany tylko przy przejsciu w stan zdrowy
(per-service in-memory state), nie co cykl dla kazdego zdrowego serwisu.
To samo dla control-plane HTTP probe. healthcheck_failed/containers_not_running/
incydenty pozostaja emitowane bez zmian (realne sygnaly).
- node-agent: naprawiono _cleanup_control_plane_fs — istniejaca retencja
eventow byla martwa od migracji observer_checkpoint.json na epoch-int
(str(path) <= int rzucal TypeError, lapane cicho przez broad except).
Teraz porownuje epoch-do-epoch i czysci wylacznie service_healthy/node_health
starsze niz checkpoint + 3-dniowy bufor; healthcheck_failed/incydenty/ha_*
zachowane bezterminowo.
- scripts/maintenance/cleanup_event_backlog.py: jednorazowy skrypt do
bezpiecznego czyszczenia backlogu na VPS (dry-run domyslnie, --apply
do usuniecia). Ten sam warunek: typ szumu + starsze niz checkpoint + 1h bufor.
Migracja 003 (UNIQUE+model, excluded_reason) zastosowana na żywej bazie kb-postgres@PIHA
(2683 chunki, bez DELETE). Kalibracja heurystyki ocr_junk (3 sygnały z planu §3.1) ujawniła
realną sprzeczność z planem: sygnał 1 (dowolny znak kontrolny) fałszywie łapał paperless:119
(wymagany aktywny) i legalny angielski tekst — próg doprecyzowany do >=5 wystąpień na
podstawie rozkładu na korpusie. 8 chunków oflagowanych ocr_junk po odjęciu potwierdzonych
fałszywych alarmów (kalendarz, mikro-fragmenty referencyjne).
Dedup: SQL z planu (exact content hash) znalazł 2 pary, ale nie wykrył znanego z pilota
duplikatu paperless:14≡74 (różne OCR, 99.2% chunków identycznych treściowo) — dodany fuzzy
check na potwierdzenie. Odrzucono 3 kandydatury o wysokim nakładaniu jako różne
wersje/typy dokumentów dzielące boilerplate PZU, nie duplikaty. 130 chunków oflagowanych
duplicate + entities[duplicate_of] na 3 kopertach.
chunk_embed.py: heurystyka is_ocr_junk() przed embedem (junk -> insert bez wywołania Ollamy,
embedding=NULL), ON CONFLICT rozszerzony o model, nowy licznik chunks_junk_flagged w bilansie,
testy (kody kreskowe, mojibake nie-junk, dot-leader, idempotencja, model w kluczu konfliktu).
Weryfikacja: 7 zapytań eval-setu z WHERE excluded_reason IS NULL — żadne trafienie nie
degraduje, kontrole negatywne bez zmian (>0.55), śmieć zniknął z top-5 zapytania 2,
paperless:119 pozostał aktywnym trafieniem. Bilans: 2545 aktywne / 130 duplicate / 8 ocr_junk.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (kalibracja
progu sygnału 1, dołączenie 14≡74 mimo braku exact-hash matcha, odrzucenie 3 fałszywych
kandydatur dedup) — plan przewidywał, że heurystyka będzie się mylić; wszystkie decyzje
udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-embed: chunks source='paperless' envelope content
(paragraph-preferring, ~600 tok/chunk, ~150 tok overlap, hard char-fallback
for oversized paragraphs per plan §2 decision 3), embeds each chunk via
Ollama (bge-m3, dim validated against document_chunk's VECTOR(1024) on
every response) and inserts into document_chunk. Lives in documents-ingest
per the plan's own recommendation (§6 step 6) rather than a new package —
reuses the job family's existing idempotency/stats-balance/dry-run
conventions (paperless_adapter.py, gmail-header-backfill).
A 5-angle multi-agent code review of the initial implementation surfaced
three real bugs, fixed here: hard_split() could infinite-loop if
--chunk-overlap >= --chunk-size (now guarded in both hard_split() and
main()); insert_chunk() wasn't error-isolated like embed_chunk(), so a DB
write failure would crash the whole run instead of being counted and
skipped; and ON CONFLICT DO NOTHING's outcome was discarded, so a silently
skipped row (the known gap where document_chunk's UNIQUE constraint
doesn't include `model`) would have been miscounted as a successful insert
- now tracked separately as chunks_conflict_skipped and treated as a
run failure.
Smoke-tested and run to completion live on SOLARIA against the real Ollama
instance and kb-postgres@PIHA: dry-run matched the known phase-2-step-5
figures exactly (186 fetched, 26 empty_content, 2684 chunks planned), a
--limit 10 apply + idempotent re-run + DB/distance sanity checks all
passed, and the full 186-document run inserted 2683/2684 chunks (1 isolated
error - Ollama's runtime context window rejected one pathological
dot-leader table-of-contents chunk that tokenized far more densely than
estimated; documented as a known limitation, not fixed here given it's a
single-chunk edge case). Timing: ~0.79s/chunk average on CPU (SOLARIA's
Ollama runs GPU-less per the recent GPU-reservation-disabled fix), ~35 min
wall-clock for the full pilot - the real input for scoping the later
mail-corpus embedding phase (plan §7's GPU-based estimate doesn't hold
here).
pytest: 101 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sesja: analiza Etapu 2 (Fable) rekomenduje GO dla wszystkich 4 węzłów; 165
mismatchy solaria/lustro wyjaśnione jako detection-lag TTL vs scrape, nie bug.
Trwały log SHADOW_LIVENESS_MISMATCH zweryfikowany testem kontrolowanym na
lustro. Backlog: ha-diag-agent node=unknown oznaczone ZROBIONE (f2ba81b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An "unknown"/empty node_name silently produced evt-unknown-* event files. One
landed in events/piha/ and, being lexically greater than every evt-piha-* name,
pinned the observer's (then lexical) checkpoint and blocked PIHA for ~34 days.
Defense in depth, two layers, both refuse rather than emit poison:
- config.py: node_name field_validator rejects ""/"unknown" (case-insensitive,
trimmed); validate_default=True so the "unknown" default itself is rejected
when NODE_NAME never reaches the process. main() catches ValidationError and
exits 1 with a clear FATAL message instead of a raw traceback.
- event_emitter.py: EventEmitter.__init__ guards node_name at the exact poison
site (node_name is embedded in the filename), so no future call path can
reintroduce evt-unknown-*.
Precedence unchanged and correct: Settings.load() uses os.environ.setdefault, so
env NODE_NAME wins over YAML; YAML only supplies node_name when env is absent.
api.py never writes event files (only /health, /trigger), so its _node_name
global cannot produce poison — verified, left as-is.
Tests (services/ha-diag-agent/tests/): new test_config.py (env load, env>YAML
precedence, fail-fast on unknown/empty/whitespace/default/YAML-unknown) and
EventEmitter guard tests. Full unit suite: 139 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live cutover 2026-07-15 odkrył: nvidia-smi not found, brak libnvidia-ml
w ldcache — host nigdy nie miał sterownika, natywna Ollama mieliła na CPU.
Sekcja deploy.resources (nvidia) zakomentowana do czasu instalacji sterownika
(backlog). nvidia-container-toolkit doinstalowany na hoście (prerequisite,
brakował w runbooku). Kontener zweryfikowany: 5/5 modeli z bind-mounta,
embed bge-m3 dim=1024, reachable z PIHA przez tailnet.
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-paperless: paginated GET /api/documents/, maps each
doc to a source='paperless' envelope per plan §4.2-4.3, reusing
kb_mail.Envelope/insert_envelope unchanged (packages/kb-mail not touched).
Cross-source link (source_mail entity) is a deterministic join of
original_file_name against the phase-1 registry.json consume_name index —
no heuristics, no correspondent guessing (plan decision 4). Stats always
balance (fetched = already_in_db + inserted + errors) and main() now also
exits non-zero on imbalance, not just on errors>0, matching the exit-code
convention already established in gmail-bulk-import.
Verified live on PIHA (rsync to /tmp, ~/kb/venv, PIHA checkout untouched):
dry-run then --apply inserted 186/186 paperless envelopes (0 errors,
180 source_mail links), a second --apply reported inserted=0/already_in_db=186
(idempotent), gmail rows stayed at 225030 and document_chunk stayed empty.
Rotated the kb-ingest Paperless API token after it was accidentally
partially echoed during recon (old token now dead).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wszystkie mismatche = detection-lag przy planowych power-offach (prom szybszy
o ~9.5 min od TTL eventowego); zero fałszywych prom=down. vps/piha 100% zgodne.
Rekomendacja: GO dla etapu 3 per-node, mapping timestamp(up)->compute_liveness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four audit findings (2026-07-14), each reproduced on crafted mboxes; the
Takeout corpus has proven 8-bit header bytes, so all are real re-import risks.
1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an
email.header.Header (not str) for raw 8-bit bytes; the old
Header.strip() raised AttributeError. _message_id ran BEFORE the
per-message try, so one bad header killed the entire import.
Fix: str() + sanitize_surrogates() before strip; and move
_message_id/_parse_date/_parse_attachments INSIDE the per-message try —
a broken message is now errors += 1, never run death.
2. Poison batch. pending.clear() ran only AFTER a successful insert, so a
failed flush (DB down / bad row) left pending intact and every later
message re-flushed the doomed batch; the final flush sat in try/finally
with no except and propagated out, losing all stats. Fix: _flush always
clears pending and counts a failed insert as db_insert_failed; the run
always reaches import_complete.
3. Stats didn't reconcile with the DB. imported counts archive writes, not
DB rows, so a partial-insert drift was invisible. Fix: separate
db_inserted/db_insert_failed counters; main() exits non-zero on any
error, DB drift, or a processed = imported + skipped + errors imbalance.
4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header)
raised even when str(header) parses fine. Fix: str() before the epoch
fallback.
Shared helper: _sanitize moved from gmail-header-backfill into
packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs;
gmail-header-backfill now depends on kb-mail.
Tests: regression coverage for all four findings in gmail-bulk-import
(8-bit id/date, per-message guard, failed-insert non-poisoning, stats
balance) plus kb_mail.text unit tests. Full suites green:
kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
check_containers() classified only exited/dead, running+unhealthy and running.
Docker's "restarting", "paused", "removing" and any future state matched no
branch, so a crash-looping container (restart policy + continuous crash) emitted
ZERO events and world-state kept showing it "healthy" (observed live:
lustro/pi-watchtower-1). Every Docker state is now handled:
- restarting + RestartCount >= CRASH_LOOP_RESTART_THRESHOLD (default 3):
reuse containers_not_running (high, crash_loop=true) — parity with exited/dead,
rides the existing supervisor-wired remediation path.
- restarting below threshold: new observational container_restarting (low),
visible but non-actionable so benign post-deploy restarts don't alarm.
- paused / unknown-or-future state: new diagnostic container_state_unexpected
(medium) — no more silent fall-through; new Docker states become visible.
- removing: conscious documented skip (ephemeral teardown).
- created: unchanged skip (compose tracking artifact).
RestartCount (top-level inspect field) distinguishes a crash-loop from a one-off
restart. Adds services/node-agent/tests/test_check_containers.py pinning the full
state table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diagnosis of the 5008 header-less envelopes after the full 225 030-row run
(read-only, on PIHA, 2026-07-14):
- 4999 = one contiguous block at ORDER BY id positions 70001-74999: the
--offset 70000 slice died mid-run. Root cause reproduced: a Date: header
with raw 8-bit bytes makes compat32 .get() return email.header.Header
(not str), and json.dumps([headers]) — which sat OUTSIDE the per-row
try/except — raised TypeError and killed the process, silently losing
the rest of the slice.
- 9 = genuine typed-parse failures: 7x RFC 2047 encoded-word decoding to
CR/LF inside a display name (ValueError in headerregistry), 1x RFC 5322
group syntax in To: ("unlisted-recipients:;"), 1x CPython
_header_value_parser bug on a malformed display name (fixed upstream,
present on PIHA's 3.11).
- 0 missing .eml files.
Fixes:
- date_raw: str() + surrogate sanitization on the compat32 value — the
crash cause, now also covered by a regression test.
- json.dumps moved inside the per-row try: a non-serializable value counts
as that row's parse_error instead of crashing the slice.
- parse_headers_fallback(): on typed-parse failure retry with a pure
compat32 parse — getaddresses over raw header text, raw-string values,
same §4.1 entity shape. Counted separately as parsed_fallback (labeled
subset of updated), logged per row with the original typed error.
- missing_file counter + skip.missing_file info log (id, expected path);
run_complete now balances: scanned = updated + already_has_headers +
parse_errors + read_errors + missing_file. Non-zero missing_file also
fails the exit code.
Verified: 43/43 pytest locally (3.13) and on PIHA (3.11); read-only dry
runs on PIHA — all 9 parse failures recover via fallback, the lost slice
completes scanned=5000 updated=4999 already_has_headers=1 with zero errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Module 5 phase 2 (docs/kb/modules/05-faza2-plan.md, §5). 225,030 existing
source='gmail' envelope rows carry only an attachment manifest — no
from/to/cc/delivered_to/subject anywhere in the DB, which blocks answering
"is this on me or my wife" / distinguishing aliases (plan §1.8). Separate
job from gmail-bulk-import per plan §2 decision 7: UPDATE semantics against
production data is a different risk profile than the historical INSERT job.
Parses headers only (no MIME-walk of attachments) from the archived .eml
files, appends {"type": "headers", ...} (plan §4.1) via the idempotent
UPDATE ... WHERE NOT EXISTS from §5.2, batched via executemany. --limit/
--offset (ORDER BY id) give stable, deterministic partitioning so the full
backfill can run and be verified in slices instead of one unattended pass.
Plain CLI (pip install -e), no Dockerfile — same convention as
gmail-bulk-import/documents-ingest, which run directly on PIHA for local
filesystem access to the .eml archive.
Verified against kb-postgres@PIHA (100-row dry-run + apply, re-run proved
idempotent no-op, 1000-row timed slice): 1096/225030 rows backfilled,
996/1000 succeeded on the timed slice (4 parse_errors — a 2014 spam message
with an RFC 2047 encoded-word decoding to an embedded newline in the From
display name, correctly caught and skipped rather than crashing the batch).
Extrapolated full-run time ~16 minutes. Full 225,030-row run is out of
scope for this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Krok 1 planu docs/kb/modules/05-faza2-plan.md §3/§6 (chunk-level embeddings,
1:N do envelope). Addytywna — 001_envelope.sql nietknięta (zweryfikowane
\d envelope po migracji: identyczny schemat + FK jako "Referenced by").
Schemat wg rekomendacji recon (§2 decyzja 1+2): osobna tabela (nie kolumna
w envelope, bo N-wartościowy chunking jest obowiązkowy przy dokumentach
>8k tokenów), embedding VECTOR(1024) pod bge-m3 (dense), HNSW cosine index,
kolumna `model` do trywialnego re-indexu przy zmianie modelu (kb-00 zasada
#1: indeks odtwarzalny). Idempotentna (CREATE TABLE/INDEX IF NOT EXISTS,
zweryfikowane podwójnym uruchomieniem na kb-postgres@PIHA — drugi run same
NOTICE "already exists, skipping").
Zastosowana na żywej bazie: ssh piha docker exec kb-postgres psql, po
potwierdzeniu SQL przez Oskara. \dt + \d document_chunk + \d envelope
zweryfikowane po migracji.
Testy: dopisane sanity-testy 002 do packages/kb-mail/tests/test_migration.py
(wzorzec 001 — statyczne assercje na treści SQL, bez DB), 13/13 zielone.
Co NIE jest częścią tego kroku (§3 planu, odłożone): entity/entity_link
(graf encji) — szkic na przyszłość, nie blokuje domknięcia modułu 5.
Co dalej (plan §6, poza zakresem tego kroku): ollama pull bge-m3 na SOLARII,
token API Paperless, jobs/gmail-header-backfill/, adapter Paperless→koperta,
chunking+embed job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of
PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive
and drops them into Paperless consume/ for OCR, so the RAG layer has real
documents to work with before the full Paperless/Nextcloud envelope adapter
is built. sha256-first attachment matching (manifest filenames can carry raw
RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default
decodes today — confirmed against live data, ~10% of candidates were affected).
Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write.
Verified end-to-end on PIHA: dry-run + --apply both run against live
kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run
duplicates correctly deduped), Paperless picked them up and started OCR
immediately.
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
token/list-hosts/list-certs/set-cert/create-host, dry-run domyslny dla
zmian (--apply wymagane), stdlib urllib (zero-dep). Adresy npm@VPS
przez Tailscale (100.95.58.48:81), NIE public IP.
+ docs/backlog.md: npm@VPS admin panel :81 publicznie osiagalny —
brak override ograniczajacego bind do mesh/localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nextcloud zostaje prywatny (mesh/kapala.org); Gokapi to osobny publiczny
serwis do wysylania linkow do plikow na zewnatrz (Firefox Send alt).
owner_node=vps, storage lokalny dysk (nie S3), E2E encryption ON, port
53842 bindowany tylko na TAILSCALE_BIND_IP (hairpin NAT przez npm@VPS,
nigdy 0.0.0.0). Cutover checklist w README (DNS, wildcard *.okit.pl na
npm@VPS, setup wizard) — nie zdeployowane w tym commicie.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co zrobione:
- Nextcloud host = PIHA (always-on dla aktywnego uzycia), owner_node +
LAN_BIND_IP/TRUSTED_PROXIES w .env, README zaktualizowane
- Redis brokera Paperlessa: requirepass, PAPERLESS_REDIS_PASSWORD w .env
po obu stronach (PIHA + worker@SOLARIA), healthchecki z auth
- Domeny potwierdzone: paper.kapala.org, cloud.kapala.org (Cloudflare
DNS-only -> Tailscale PIHA, wildcard cert juz pokrywa) — udokumentowane,
nic nie utworzone
- Backup Paperlessa zatwierdzony: document_exporter + rsync/borg -> SOLARIA,
retencja 7/4/6, offsite jako future-note
- Nextcloud pin: 34-apache (zweryfikowany aktualny stable, endoflife.date)
- Whoosh fallback-worker: zaakceptowane bez zmian
- Porty/wylaczenie local login/sizing OCR-workera: przeniesione z "decyzji"
na "TODO przy deployu"
- DECYZJE-do-podjecia.md zaktualizowane: wszystko poza portami/loginem/
sizingiem przeniesione do "Rozstrzygniete"
Tylko edycja configow w repo — nic nie zdeployowane, zadne kontenery nie
byly ruszane, DNS/vhosty nie utworzone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oskar Kapala