Operator added the A-record manually; verified against Cloudflare's own
authoritative NS (which briefly still returned NXDOMAIN — not a propagation
lag, the record simply wasn't saved yet on first check) plus 8.8.8.8/1.1.1.1.
Both DNS layers (Cloudflare public/Tailscale + Pi-hole LAN override) and the
vhost now confirmed working end-to-end.
Runtime steps (not in Git, logged here): npm@PIHA proxy host #35
(kb.kapala.org -> 192.168.31.5:8230, cert #49 *.kapala.org wildcard, same
pattern as paper./vikunja.kapala.org); Pi-hole custom.list split-horizon
entry added and verified (first kapala.org entry in that file — the other
kapala.org vhosts turned out to have no LAN override at all, a plan
assumption that didn't hold). Cloudflare A-record left for the operator (no
API token available here). OIDC intentionally not built: confirmed no
forward-auth pattern exists anywhere in this repo, and building authlib
OIDC into kb-query is real service code out of scope for an infra-only
task — operator decided to leave kb.kapala.org without auth for now.
Zmiany wyłącznie w repo (config/ken/ automations), deploy LIVE robi operator.
- 1752086407230 "dry in Kuchnia": trigger moist -> not_moist (kopiuj-wklej z
"moist in Kuchnia", "sucho" nigdy nie przyjdzie).
- 1752085965483 "dry in Lazienka": literówka mesaage -> message (skrypt
notify_email_ntfy dostawał niezdefiniowane message, akcja się wykładała).
- 1764190493305 "after 15 minutes anyway": dodany warunek stanu
automation.turn_off_lights_in_kuchania == on, żeby respektować ręczne
"Disable AUTO off" (double-click) zamiast unieważniać je po 15 minutach.
- 1700832676138 "unconditionally at 3am": time_pattern hours: '3' (60
odpaleń/noc) -> punktowy trigger time: 03:00:00, zgodnie z intencją aliasu.
- 1784804668795 "Klima salon: wyłącz...": dodany warunek
input_boolean.klima_salon_auto == on dla gałęzi sunset/balkon (automat
ubija tylko chłodzenie, którym sam zarządza — ręczne chłodzenie przy
zgaszonym auto zostaje nietknięte). Trigger auto_off (zgaszenie
klima_salon_auto) ma osobną gałąź choose bez tego warunku — graceful
shutdown suszenia parownika działa nawet gdy przełącznik właśnie zgasł.
Zweryfikowano na żywym ken: helpery klimy istnieją (input_boolean.
klima_salon_auto=off, input_number.klima_salon_temp_docelowa=24.0,
input_number.klima_salon_tolerancja=3.0).
DESIGN.md: nowa sekcja "Decyzje operatora po audycie 2026-07-23" — co
zaimplementowane, co świadomie odłożone (enforcer sleep mode + konsolidacja
nocnych wyłączników -> projekt night_mode; OwnTracks / Leave batch 02 / alerty
on-leave -> bez zmian).
docs/backlog.md: cztery nowe wpisy w Aktywne — guard TRV przed sezonem
grzewczym, przycisk graceful shutdown klimy na dashboard, diagnoza wspólnej
awarii sprzętowej 2026-07-17 (czujniki ruchu + pilot 4button + xiaomi_miot),
projekt architektura night_mode.
Walidacja: round-trip scripts/ha/lib/normalize.py na wszystkich 5 zmienionych
plików (stabilny), 4 zestawy testów offline scripts/ha/tests/*.sh (wszystkie
PASS), dry-run scripts/ha/deploy.sh ken --dry-run per plik na żywym ken
(wszystkie: check_config valid, brak driftu, brak abortu).
Choinkowe automatyzacje (audyt 4.2) celowo nietknięte w plikach — disable to
stan runtime, nie config; operator wyłączy je przez UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bug: hit_at_3 returned None for kind=mail_hit rows (expected_envelope is
always null for them -- operator supplies query text, not a Message-ID), so
criterion 4 could never count a hit and read 0/5 despite hybrid distances of
0.25-0.42. Fixed with mail_hit_at_3: hit iff the top-3 distinct hybrid
envelopes include a mail-sourced one (envelope.source lookup via
fetch_envelope_sources, since hybrid_retrieve overwrites source to "hybrid"
on merge) under HIT_THRESHOLD. Also added a per-query no_answer_threshold
override in queries.yaml for criterion 3.
N2 ("piaskownica plastikowa") investigation: after Etap A added ~34k mail
chunks, N2's top-1 neighbor dropped to dist 0.5298 (< the 0.55 bar). Content
check showed it's a ski-school reservation newsletter (Rossignol ski sizes)
-- a semantic false-positive collision, not a real corpus match. M5, which
the operator had added assuming a genuine piaskownica mail existed, itself
misses (dist 0.5585) -- confirming there's no such mail in the corpus. M5
dropped; N2's pass bar lowered to 0.50 with a note documenting the collision.
Gate result on the live DB post-Etap A (Etap A: 13 009 mails scanned -> 33 871
new gmail chunks, 6 398 embedded / 27 473 newsletter-flagged, balanced +
idempotent on rerun; Ollama incident #4 during the run required a compose
force-recreate, not just restart -- root-cause task ollama-solaria-start-race
stays in backlog): all 4 criteria PASS (5/5 flat hits held, cascade hit@3 5/5
vs flat 4/5, negative controls above their bars, mail hit@3 4/4 after
dropping M5). Plan doc updated with the numbers and verdict table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Zamknięte: publiczny bind operator_ui:18180 (9a5c160), remediacja floty
bez SSH (2dac154, E2E potwierdzone), uprawnienia actions/ na PIHA.
Aktywne (nowe): retry zepsutego JSON w approved/, uprawnienia actions/
niezweryfikowane poza PIHA, brak checka .env/TAILSCALE_BIND_IP w
deploy-local.sh, brak autoryzacji w operator_ui.py, alert_only zapycha
approval queue, crash-loop bez container_restart, brak Telegram yes/no
dla pending, homeassistant5 Exited(0), node-agent repo-less na lustro.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Zamknięcie wątku "czemu mózg nie leczy floty": fix publicznego bindu 18180
bez autoryzacji (9a5c160), remediacja pull-based przez node-agent bez SSH
(2dac154), fix uprawnień actions/ na PIHA, pierwszy udany cykl E2E (31s).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ken is HAOS (no SSH/docker exec path), so it needs a REST/WebSocket-only
adapter: automations/scripts/scenes via one GET per
/api/config/<domain>/config/<id>, dashboards/area+entity registries/
input_* helpers via the HA WebSocket API (read-only commands only).
scripts/ha/lib/ha_api.py and ha_ws.py never take a token as a value —
only a token_path, read from disk in-process — so the bearer token never
touches a subprocess argv or a log line. ha_ws.py depends on the optional
websocket-client package and raises a clear, actionable ImportError if
it's missing rather than a raw traceback; import.sh still completes the
REST-only part of the import in that case.
Automations/scripts/scenes reuse split.write_split() so both adapters
produce byte-identical config/<instance>/ layouts and the same
idempotent stale-file cleanup on re-run.
docker-exec adapter logic is untouched.
instances.yaml pointed "ken" at the piha container (homeassistant5), a
pre-migration instance that never actually stopped running — it kept
firing automations in parallel with the real home instance (HAOS on a
dedicated RPi4, 192.168.31.7), causing duplicate triggers on shared
buttons/scenes. Repoint ken to the real instance (api adapter),
demote the container to ken-legacy (archived, import-only, never a
deploy target), and log the incident in DESIGN.md.
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents_ingest.retrieval: flat_query (baseline) i cascade_query (stage1
document_summary model='claude-haiku-4-5' -> stage2 document_chunk), jedno
dzielone wywołanie embeddingu bge-m3 per zapytanie, tylko +1 SQL na kaskadę.
Czyste query_text -> wyniki(dist, source) pod przyszłe kb-query fazy 4.
166/166 testów (10 nowych, mocki: stage1->stage2, koperta bez chunków,
N > liczba kopert, no-summaries short-circuit).
Eval-set utrwalony 1:1 z pilota (docs/kb/eval/retrieval-pilot-2026-07-16.md,
nietknięty) w eval/queries.yaml + skrypt bramki eval/retrieval_eval.py
(integracyjny, read-only, poza pytest).
Wynik bramki (żywa baza, N=10 k=5): kryterium 1 (brak degradacji) PASS,
kryterium 2 (hit@3 kaskada=5/5 vs płaski=5/5) PASS, kryterium 3 (kontrole
negatywne 0.644/0.553 > 0.55 w obu torach) PASS. Sweep N∈{1,2,3,5,10,20}:
N=5 to zmierzony próg bezpieczny (N<5 degraduje zapytania 3-4), N=10 ma
2x margines — potwierdza domyślną wartość z planu zamiast przyjmować ją
z założenia. Kaskada nie poprawia jakości na 186-dok. korpusie (dystanse
identyczne z płaskim przy N≥5) — zgodnie z przewidywaniem planu: to test
architektury pod skalę mailową, nie optymalizacja pilota.
Decyzja: kaskada (N=10, k=5, claude-haiku-4-5) = domyślna ścieżka retrievalu.
Plan-doc §6.3 zaktualizowany wynikiem; §2 D3 zamknięte rozstrzygnięciem
Oskara (tor kompilacyjny=claude-haiku-4-5, gemma3:12b w odwodzie, decyzja
mailowa odłożona do reconu z flagą prywatności/kosztu).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sesja: analiza Etapu 2 (Fable) rekomenduje GO dla wszystkich 4 węzłów; 165
mismatchy solaria/lustro wyjaśnione jako detection-lag TTL vs scrape, nie bug.
Trwały log SHADOW_LIVENESS_MISMATCH zweryfikowany testem kontrolowanym na
lustro. Backlog: ha-diag-agent node=unknown oznaczone ZROBIONE (f2ba81b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wszystkie mismatche = detection-lag przy planowych power-offach (prom szybszy
o ~9.5 min od TTL eventowego); zero fałszywych prom=down. vps/piha 100% zgodne.
Rekomendacja: GO dla etapu 3 per-node, mapping timestamp(up)->compute_liveness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
token/list-hosts/list-certs/set-cert/create-host, dry-run domyslny dla
zmian (--apply wymagane), stdlib urllib (zero-dep). Adresy npm@VPS
przez Tailscale (100.95.58.48:81), NIE public IP.
+ docs/backlog.md: npm@VPS admin panel :81 publicznie osiagalny —
brak override ograniczajacego bind do mesh/localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co zrobione:
- Nextcloud host = PIHA (always-on dla aktywnego uzycia), owner_node +
LAN_BIND_IP/TRUSTED_PROXIES w .env, README zaktualizowane
- Redis brokera Paperlessa: requirepass, PAPERLESS_REDIS_PASSWORD w .env
po obu stronach (PIHA + worker@SOLARIA), healthchecki z auth
- Domeny potwierdzone: paper.kapala.org, cloud.kapala.org (Cloudflare
DNS-only -> Tailscale PIHA, wildcard cert juz pokrywa) — udokumentowane,
nic nie utworzone
- Backup Paperlessa zatwierdzony: document_exporter + rsync/borg -> SOLARIA,
retencja 7/4/6, offsite jako future-note
- Nextcloud pin: 34-apache (zweryfikowany aktualny stable, endoflife.date)
- Whoosh fallback-worker: zaakceptowane bez zmian
- Porty/wylaczenie local login/sizing OCR-workera: przeniesione z "decyzji"
na "TODO przy deployu"
- DECYZJE-do-podjecia.md zaktualizowane: wszystko poza portami/loginem/
sizingiem przeniesione do "Rozstrzygniete"
Tylko edycja configow w repo — nic nie zdeployowane, zadne kontenery nie
byly ruszane, DNS/vhosty nie utworzone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Faza 2 modułu 0 po review Oskara: elasticsearch+diskover usunięte (compose down,
dane esdata zostawione na dysku), available 2.8Gi -> 3.8Gi, kryterium >=1.5Gi
spełnione. llm-gateway udokumentowany (własny router LLM -> Ollama@SOLARIA,
źródło tylko w /opt/llm-gateway — archiwizacja w backlogu); immich zostaje na
PIHA na stałe (24/7, SOLARIA sesyjna).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- targety node_exporter floty (piha/solaria/lustro) wdrożone (7d4014e)
- fix deploy.sh vps↔control-plane potwierdzony w boju (3b71707)
- PENDING: health-verify targetów w /api/v1/targets nie potwierdzony
- lekcja: nie commitować na master równolegle gdy CC pracuje na wątku (potrójny rebase)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swap 2-4GB na VPS -> Zamkniete (2026-06-22). Nowa sekcja planu monitoringu
floty (Prometheus pull up{}, osobny instance, bez Alertmanagera) z krokami.
Zastepuje szkic blackbox+Alertmanager z 2026-06-17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records the onboarding session for LUSTRO (RPi4, KEN site):
node facts from preflight, key decisions (user pi/uid-1000, IP
over mDNS, zram target), 00-access status, tool bugs fixed
(dry-run propagation, yaml_get greedy-colon + inline comment,
ssh known-hosts in verify), open items for next session
(worktree hygiene first, bootstrap-runtime, node-agent, register,
verify, mm-watch).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala