Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Module 5 phase 2 (docs/kb/modules/05-faza2-plan.md, §5). 225,030 existing
source='gmail' envelope rows carry only an attachment manifest — no
from/to/cc/delivered_to/subject anywhere in the DB, which blocks answering
"is this on me or my wife" / distinguishing aliases (plan §1.8). Separate
job from gmail-bulk-import per plan §2 decision 7: UPDATE semantics against
production data is a different risk profile than the historical INSERT job.
Parses headers only (no MIME-walk of attachments) from the archived .eml
files, appends {"type": "headers", ...} (plan §4.1) via the idempotent
UPDATE ... WHERE NOT EXISTS from §5.2, batched via executemany. --limit/
--offset (ORDER BY id) give stable, deterministic partitioning so the full
backfill can run and be verified in slices instead of one unattended pass.
Plain CLI (pip install -e), no Dockerfile — same convention as
gmail-bulk-import/documents-ingest, which run directly on PIHA for local
filesystem access to the .eml archive.
Verified against kb-postgres@PIHA (100-row dry-run + apply, re-run proved
idempotent no-op, 1000-row timed slice): 1096/225030 rows backfilled,
996/1000 succeeded on the timed slice (4 parse_errors — a 2014 spam message
with an RFC 2047 encoded-word decoding to an embedded newline in the From
display name, correctly caught and skipped rather than crashing the batch).
Extrapolated full-run time ~16 minutes. Full 225,030-row run is out of
scope for this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Krok 1 planu docs/kb/modules/05-faza2-plan.md §3/§6 (chunk-level embeddings,
1:N do envelope). Addytywna — 001_envelope.sql nietknięta (zweryfikowane
\d envelope po migracji: identyczny schemat + FK jako "Referenced by").
Schemat wg rekomendacji recon (§2 decyzja 1+2): osobna tabela (nie kolumna
w envelope, bo N-wartościowy chunking jest obowiązkowy przy dokumentach
>8k tokenów), embedding VECTOR(1024) pod bge-m3 (dense), HNSW cosine index,
kolumna `model` do trywialnego re-indexu przy zmianie modelu (kb-00 zasada
#1: indeks odtwarzalny). Idempotentna (CREATE TABLE/INDEX IF NOT EXISTS,
zweryfikowane podwójnym uruchomieniem na kb-postgres@PIHA — drugi run same
NOTICE "already exists, skipping").
Zastosowana na żywej bazie: ssh piha docker exec kb-postgres psql, po
potwierdzeniu SQL przez Oskara. \dt + \d document_chunk + \d envelope
zweryfikowane po migracji.
Testy: dopisane sanity-testy 002 do packages/kb-mail/tests/test_migration.py
(wzorzec 001 — statyczne assercje na treści SQL, bez DB), 13/13 zielone.
Co NIE jest częścią tego kroku (§3 planu, odłożone): entity/entity_link
(graf encji) — szkic na przyszłość, nie blokuje domknięcia modułu 5.
Co dalej (plan §6, poza zakresem tego kroku): ollama pull bge-m3 na SOLARII,
token API Paperless, jobs/gmail-header-backfill/, adapter Paperless→koperta,
chunking+embed job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of
PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive
and drops them into Paperless consume/ for OCR, so the RAG layer has real
documents to work with before the full Paperless/Nextcloud envelope adapter
is built. sha256-first attachment matching (manifest filenames can carry raw
RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default
decodes today — confirmed against live data, ~10% of candidates were affected).
Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write.
Verified end-to-end on PIHA: dry-run + --apply both run against live
kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run
duplicates correctly deduped), Paperless picked them up and started OCR
immediately.
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
token/list-hosts/list-certs/set-cert/create-host, dry-run domyslny dla
zmian (--apply wymagane), stdlib urllib (zero-dep). Adresy npm@VPS
przez Tailscale (100.95.58.48:81), NIE public IP.
+ docs/backlog.md: npm@VPS admin panel :81 publicznie osiagalny —
brak override ograniczajacego bind do mesh/localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nextcloud zostaje prywatny (mesh/kapala.org); Gokapi to osobny publiczny
serwis do wysylania linkow do plikow na zewnatrz (Firefox Send alt).
owner_node=vps, storage lokalny dysk (nie S3), E2E encryption ON, port
53842 bindowany tylko na TAILSCALE_BIND_IP (hairpin NAT przez npm@VPS,
nigdy 0.0.0.0). Cutover checklist w README (DNS, wildcard *.okit.pl na
npm@VPS, setup wizard) — nie zdeployowane w tym commicie.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co zrobione:
- Nextcloud host = PIHA (always-on dla aktywnego uzycia), owner_node +
LAN_BIND_IP/TRUSTED_PROXIES w .env, README zaktualizowane
- Redis brokera Paperlessa: requirepass, PAPERLESS_REDIS_PASSWORD w .env
po obu stronach (PIHA + worker@SOLARIA), healthchecki z auth
- Domeny potwierdzone: paper.kapala.org, cloud.kapala.org (Cloudflare
DNS-only -> Tailscale PIHA, wildcard cert juz pokrywa) — udokumentowane,
nic nie utworzone
- Backup Paperlessa zatwierdzony: document_exporter + rsync/borg -> SOLARIA,
retencja 7/4/6, offsite jako future-note
- Nextcloud pin: 34-apache (zweryfikowany aktualny stable, endoflife.date)
- Whoosh fallback-worker: zaakceptowane bez zmian
- Porty/wylaczenie local login/sizing OCR-workera: przeniesione z "decyzji"
na "TODO przy deployu"
- DECYZJE-do-podjecia.md zaktualizowane: wszystko poza portami/loginem/
sizingiem przeniesione do "Rozstrzygniete"
Tylko edycja configow w repo — nic nie zdeployowane, zadne kontenery nie
byly ruszane, DNS/vhosty nie utworzone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kod zyl tylko na dysku PIHA w /opt/llm-gateway (bez gita). Przeniesiony do
services/llm-gateway/ pod pelny wzorzec homelaba:
- app/main.py: kod 1:1 z PIHA + OLLAMA_URL/CHAT_MODEL/CODE_MODEL
nadpisywalne przez env (defaulty bez zmian)
- docker-compose.yml: bind TYLKO do Tailscale IP (${TAILSCALE_BIND_IP},
wzorzec fleet-prometheus), nie 0.0.0.0 jak w starym compose
- service.yaml, env.example (bez sekretow), healthcheck.sh, README, testy
- hosts/piha/runtime/llm-gateway: mem_limit 256m (PIHA jest RAM-bound)
- rejestracja w hosts/piha/services.yaml i inventory/topology.yaml
Zepsuty /opt/llm-gateway/docker-compose.yml (zduplikowany klucz ports)
celowo NIE przeniesiony.
DoD: pytest 4 passed; docker build + smoke run OK (GET / -> gateway ok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faza 2 modułu 0 po review Oskara: elasticsearch+diskover usunięte (compose down,
dane esdata zostawione na dysku), available 2.8Gi -> 3.8Gi, kryterium >=1.5Gi
spełnione. llm-gateway udokumentowany (własny router LLM -> Ollama@SOLARIA,
źródło tylko w /opt/llm-gateway — archiwizacja w backlogu); immich zostaje na
PIHA na stałe (24/7, SOLARIA sesyjna).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- targety node_exporter floty (piha/solaria/lustro) wdrożone (7d4014e)
- fix deploy.sh vps↔control-plane potwierdzony w boju (3b71707)
- PENDING: health-verify targetów w /api/v1/targets nie potwierdzony
- lekcja: nie commitować na master równolegle gdy CC pracuje na wątku (potrójny rebase)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oskar Kapala