Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sesja: analiza Etapu 2 (Fable) rekomenduje GO dla wszystkich 4 węzłów; 165
mismatchy solaria/lustro wyjaśnione jako detection-lag TTL vs scrape, nie bug.
Trwały log SHADOW_LIVENESS_MISMATCH zweryfikowany testem kontrolowanym na
lustro. Backlog: ha-diag-agent node=unknown oznaczone ZROBIONE (f2ba81b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wszystkie mismatche = detection-lag przy planowych power-offach (prom szybszy
o ~9.5 min od TTL eventowego); zero fałszywych prom=down. vps/piha 100% zgodne.
Rekomendacja: GO dla etapu 3 per-node, mapping timestamp(up)->compute_liveness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
token/list-hosts/list-certs/set-cert/create-host, dry-run domyslny dla
zmian (--apply wymagane), stdlib urllib (zero-dep). Adresy npm@VPS
przez Tailscale (100.95.58.48:81), NIE public IP.
+ docs/backlog.md: npm@VPS admin panel :81 publicznie osiagalny —
brak override ograniczajacego bind do mesh/localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co zrobione:
- Nextcloud host = PIHA (always-on dla aktywnego uzycia), owner_node +
LAN_BIND_IP/TRUSTED_PROXIES w .env, README zaktualizowane
- Redis brokera Paperlessa: requirepass, PAPERLESS_REDIS_PASSWORD w .env
po obu stronach (PIHA + worker@SOLARIA), healthchecki z auth
- Domeny potwierdzone: paper.kapala.org, cloud.kapala.org (Cloudflare
DNS-only -> Tailscale PIHA, wildcard cert juz pokrywa) — udokumentowane,
nic nie utworzone
- Backup Paperlessa zatwierdzony: document_exporter + rsync/borg -> SOLARIA,
retencja 7/4/6, offsite jako future-note
- Nextcloud pin: 34-apache (zweryfikowany aktualny stable, endoflife.date)
- Whoosh fallback-worker: zaakceptowane bez zmian
- Porty/wylaczenie local login/sizing OCR-workera: przeniesione z "decyzji"
na "TODO przy deployu"
- DECYZJE-do-podjecia.md zaktualizowane: wszystko poza portami/loginem/
sizingiem przeniesione do "Rozstrzygniete"
Tylko edycja configow w repo — nic nie zdeployowane, zadne kontenery nie
byly ruszane, DNS/vhosty nie utworzone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Faza 2 modułu 0 po review Oskara: elasticsearch+diskover usunięte (compose down,
dane esdata zostawione na dysku), available 2.8Gi -> 3.8Gi, kryterium >=1.5Gi
spełnione. llm-gateway udokumentowany (własny router LLM -> Ollama@SOLARIA,
źródło tylko w /opt/llm-gateway — archiwizacja w backlogu); immich zostaje na
PIHA na stałe (24/7, SOLARIA sesyjna).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- targety node_exporter floty (piha/solaria/lustro) wdrożone (7d4014e)
- fix deploy.sh vps↔control-plane potwierdzony w boju (3b71707)
- PENDING: health-verify targetów w /api/v1/targets nie potwierdzony
- lekcja: nie commitować na master równolegle gdy CC pracuje na wątku (potrójny rebase)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swap 2-4GB na VPS -> Zamkniete (2026-06-22). Nowa sekcja planu monitoringu
floty (Prometheus pull up{}, osobny instance, bez Alertmanagera) z krokami.
Zastepuje szkic blackbox+Alertmanager z 2026-06-17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the onboarding session for LUSTRO (RPi4, KEN site):
node facts from preflight, key decisions (user pi/uid-1000, IP
over mDNS, zram target), 00-access status, tool bugs fixed
(dry-run propagation, yaml_get greedy-colon + inline comment,
ssh known-hosts in verify), open items for next session
(worktree hygiene first, bootstrap-runtime, node-agent, register,
verify, mm-watch).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala