The group_add fix itself already landed on master (ddae57c) but the running
container still has GroupAdd=[999] — deploy pending from main checkout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RECON appendix summarizes the whole etap-0 cleanup (topology dormant status,
hosts reconciliation, dead-code removal, legacy decision), carries the
runbook line for the stale chelsty pending action
(rm /opt/homelab/actions/pending/redeploy-chelsty-infra-ha-diag-agent.json —
verified the only chelsty-targeted pending file 2026-07-30), and lists
discrepancies found but out of scope: lustro watchtower, stale solaria/chelsty
world-state keys, piha naming-drift duplicate keys, gokapi still down with a
stuck pending redeploy, outline/joplin/umami still lacking services/ dirs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bus codex/* dead since 2026-06-09 (recon C9). Records what survives into
subsystem B (patterns only: task bus, worker roles, enforced command
allowlists, preview mode), that task/ai-cluster-solaria stays unmerged
as documentation, and that the vps containers get stopped in a separate
supervised session.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One page of binding architecture decisions per the 2026-07-27 recon:
two subsystems (A keep-it-running / B do-the-work), node roles incl.
solaria's by-design duty cycle and dormant chelsty site, ai-cluster
declared legacy (bus dead since 2026-06-09), approvals stay HITL with
Telegram as the etap-2 channel.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Odtworzone graceful shutdown pimirror (audyt pkt 14, jedyna realna strata
migracji): nowa automatyzacja "Pimirror: graceful shutdown przed odcieciem
zasilania" (1785164185794.yaml) wciska przycisk shutdown przed twardym
cieciem zasilania o 23:35, dajac istniejacemu ACK-flow ("Magic Mirror OFF on
ACK") szanse zamknac Pi grzecznie zamiast zawsze odcinac prad na sile.
Zaadaptowane wzgledem legacy: entity_id zamiast device automation, oryginalna
encja button.rpi_pimirror_* jest dziś unavailable — uzyto aktualnej
button.rpi_pimirror2_* (potwierdzone w storage-export/ken).
Spisana konwencja na przyszlosc (audyt pkt 17) w DESIGN.md: entity_id zamiast
device automations, alias PL z prefiksem funkcjonalnym, description z
"managed-by: repo" — bez hurtowej migracji istniejacych automatyzacji.
Backlog: deploy.sh --delete (kasacje wspolnym torem dry-run/LIVE zamiast
recznych curl DELETE) i brakujacy trigger na zmiane input_number.klima_salon_tolerancja
w automatyzacji ON klimy.
Walidacja: normalizator round-trip (byte-identical), 4 pakiety testow offline
(normalize/split/deploy_api/import_api) pass, dry-run deploy.sh na zywym ken
(check_config valid, brak driftu) — bez LIVE deployu.
Operator added the A-record manually; verified against Cloudflare's own
authoritative NS (which briefly still returned NXDOMAIN — not a propagation
lag, the record simply wasn't saved yet on first check) plus 8.8.8.8/1.1.1.1.
Both DNS layers (Cloudflare public/Tailscale + Pi-hole LAN override) and the
vhost now confirmed working end-to-end.
Runtime steps (not in Git, logged here): npm@PIHA proxy host #35
(kb.kapala.org -> 192.168.31.5:8230, cert #49 *.kapala.org wildcard, same
pattern as paper./vikunja.kapala.org); Pi-hole custom.list split-horizon
entry added and verified (first kapala.org entry in that file — the other
kapala.org vhosts turned out to have no LAN override at all, a plan
assumption that didn't hold). Cloudflare A-record left for the operator (no
API token available here). OIDC intentionally not built: confirmed no
forward-auth pattern exists anywhere in this repo, and building authlib
OIDC into kb-query is real service code out of scope for an infra-only
task — operator decided to leave kb.kapala.org without auth for now.
Zmiany wyłącznie w repo (config/ken/ automations), deploy LIVE robi operator.
- 1752086407230 "dry in Kuchnia": trigger moist -> not_moist (kopiuj-wklej z
"moist in Kuchnia", "sucho" nigdy nie przyjdzie).
- 1752085965483 "dry in Lazienka": literówka mesaage -> message (skrypt
notify_email_ntfy dostawał niezdefiniowane message, akcja się wykładała).
- 1764190493305 "after 15 minutes anyway": dodany warunek stanu
automation.turn_off_lights_in_kuchania == on, żeby respektować ręczne
"Disable AUTO off" (double-click) zamiast unieważniać je po 15 minutach.
- 1700832676138 "unconditionally at 3am": time_pattern hours: '3' (60
odpaleń/noc) -> punktowy trigger time: 03:00:00, zgodnie z intencją aliasu.
- 1784804668795 "Klima salon: wyłącz...": dodany warunek
input_boolean.klima_salon_auto == on dla gałęzi sunset/balkon (automat
ubija tylko chłodzenie, którym sam zarządza — ręczne chłodzenie przy
zgaszonym auto zostaje nietknięte). Trigger auto_off (zgaszenie
klima_salon_auto) ma osobną gałąź choose bez tego warunku — graceful
shutdown suszenia parownika działa nawet gdy przełącznik właśnie zgasł.
Zweryfikowano na żywym ken: helpery klimy istnieją (input_boolean.
klima_salon_auto=off, input_number.klima_salon_temp_docelowa=24.0,
input_number.klima_salon_tolerancja=3.0).
DESIGN.md: nowa sekcja "Decyzje operatora po audycie 2026-07-23" — co
zaimplementowane, co świadomie odłożone (enforcer sleep mode + konsolidacja
nocnych wyłączników -> projekt night_mode; OwnTracks / Leave batch 02 / alerty
on-leave -> bez zmian).
docs/backlog.md: cztery nowe wpisy w Aktywne — guard TRV przed sezonem
grzewczym, przycisk graceful shutdown klimy na dashboard, diagnoza wspólnej
awarii sprzętowej 2026-07-17 (czujniki ruchu + pilot 4button + xiaomi_miot),
projekt architektura night_mode.
Walidacja: round-trip scripts/ha/lib/normalize.py na wszystkich 5 zmienionych
plików (stabilny), 4 zestawy testów offline scripts/ha/tests/*.sh (wszystkie
PASS), dry-run scripts/ha/deploy.sh ken --dry-run per plik na żywym ken
(wszystkie: check_config valid, brak driftu, brak abortu).
Choinkowe automatyzacje (audyt 4.2) celowo nietknięte w plikach — disable to
stan runtime, nie config; operator wyłączy je przez UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bug: hit_at_3 returned None for kind=mail_hit rows (expected_envelope is
always null for them -- operator supplies query text, not a Message-ID), so
criterion 4 could never count a hit and read 0/5 despite hybrid distances of
0.25-0.42. Fixed with mail_hit_at_3: hit iff the top-3 distinct hybrid
envelopes include a mail-sourced one (envelope.source lookup via
fetch_envelope_sources, since hybrid_retrieve overwrites source to "hybrid"
on merge) under HIT_THRESHOLD. Also added a per-query no_answer_threshold
override in queries.yaml for criterion 3.
N2 ("piaskownica plastikowa") investigation: after Etap A added ~34k mail
chunks, N2's top-1 neighbor dropped to dist 0.5298 (< the 0.55 bar). Content
check showed it's a ski-school reservation newsletter (Rossignol ski sizes)
-- a semantic false-positive collision, not a real corpus match. M5, which
the operator had added assuming a genuine piaskownica mail existed, itself
misses (dist 0.5585) -- confirming there's no such mail in the corpus. M5
dropped; N2's pass bar lowered to 0.50 with a note documenting the collision.
Gate result on the live DB post-Etap A (Etap A: 13 009 mails scanned -> 33 871
new gmail chunks, 6 398 embedded / 27 473 newsletter-flagged, balanced +
idempotent on rerun; Ollama incident #4 during the run required a compose
force-recreate, not just restart -- root-cause task ollama-solaria-start-race
stays in backlog): all 4 criteria PASS (5/5 flat hits held, cascade hit@3 5/5
vs flat 4/5, negative controls above their bars, mail hit@3 4/4 after
dropping M5). Plan doc updated with the numbers and verdict table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Zamknięte: publiczny bind operator_ui:18180 (9a5c160), remediacja floty
bez SSH (2dac154, E2E potwierdzone), uprawnienia actions/ na PIHA.
Aktywne (nowe): retry zepsutego JSON w approved/, uprawnienia actions/
niezweryfikowane poza PIHA, brak checka .env/TAILSCALE_BIND_IP w
deploy-local.sh, brak autoryzacji w operator_ui.py, alert_only zapycha
approval queue, crash-loop bez container_restart, brak Telegram yes/no
dla pending, homeassistant5 Exited(0), node-agent repo-less na lustro.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Zamknięcie wątku "czemu mózg nie leczy floty": fix publicznego bindu 18180
bez autoryzacji (9a5c160), remediacja pull-based przez node-agent bez SSH
(2dac154), fix uprawnień actions/ na PIHA, pierwszy udany cykl E2E (31s).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ken is HAOS (no SSH/docker exec path), so it needs a REST/WebSocket-only
adapter: automations/scripts/scenes via one GET per
/api/config/<domain>/config/<id>, dashboards/area+entity registries/
input_* helpers via the HA WebSocket API (read-only commands only).
scripts/ha/lib/ha_api.py and ha_ws.py never take a token as a value —
only a token_path, read from disk in-process — so the bearer token never
touches a subprocess argv or a log line. ha_ws.py depends on the optional
websocket-client package and raises a clear, actionable ImportError if
it's missing rather than a raw traceback; import.sh still completes the
REST-only part of the import in that case.
Automations/scripts/scenes reuse split.write_split() so both adapters
produce byte-identical config/<instance>/ layouts and the same
idempotent stale-file cleanup on re-run.
docker-exec adapter logic is untouched.
instances.yaml pointed "ken" at the piha container (homeassistant5), a
pre-migration instance that never actually stopped running — it kept
firing automations in parallel with the real home instance (HAOS on a
dedicated RPi4, 192.168.31.7), causing duplicate triggers on shared
buttons/scenes. Repoint ken to the real instance (api adapter),
demote the container to ken-legacy (archived, import-only, never a
deploy target), and log the incident in DESIGN.md.
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents_ingest.retrieval: flat_query (baseline) i cascade_query (stage1
document_summary model='claude-haiku-4-5' -> stage2 document_chunk), jedno
dzielone wywołanie embeddingu bge-m3 per zapytanie, tylko +1 SQL na kaskadę.
Czyste query_text -> wyniki(dist, source) pod przyszłe kb-query fazy 4.
166/166 testów (10 nowych, mocki: stage1->stage2, koperta bez chunków,
N > liczba kopert, no-summaries short-circuit).
Eval-set utrwalony 1:1 z pilota (docs/kb/eval/retrieval-pilot-2026-07-16.md,
nietknięty) w eval/queries.yaml + skrypt bramki eval/retrieval_eval.py
(integracyjny, read-only, poza pytest).
Wynik bramki (żywa baza, N=10 k=5): kryterium 1 (brak degradacji) PASS,
kryterium 2 (hit@3 kaskada=5/5 vs płaski=5/5) PASS, kryterium 3 (kontrole
negatywne 0.644/0.553 > 0.55 w obu torach) PASS. Sweep N∈{1,2,3,5,10,20}:
N=5 to zmierzony próg bezpieczny (N<5 degraduje zapytania 3-4), N=10 ma
2x margines — potwierdza domyślną wartość z planu zamiast przyjmować ją
z założenia. Kaskada nie poprawia jakości na 186-dok. korpusie (dystanse
identyczne z płaskim przy N≥5) — zgodnie z przewidywaniem planu: to test
architektury pod skalę mailową, nie optymalizacja pilota.
Decyzja: kaskada (N=10, k=5, claude-haiku-4-5) = domyślna ścieżka retrievalu.
Plan-doc §6.3 zaktualizowany wynikiem; §2 D3 zamknięte rozstrzygnięciem
Oskara (tor kompilacyjny=claude-haiku-4-5, gemma3:12b w odwodzie, decyzja
mailowa odłożona do reconu z flagą prywatności/kosztu).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sesja: analiza Etapu 2 (Fable) rekomenduje GO dla wszystkich 4 węzłów; 165
mismatchy solaria/lustro wyjaśnione jako detection-lag TTL vs scrape, nie bug.
Trwały log SHADOW_LIVENESS_MISMATCH zweryfikowany testem kontrolowanym na
lustro. Backlog: ha-diag-agent node=unknown oznaczone ZROBIONE (f2ba81b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wszystkie mismatche = detection-lag przy planowych power-offach (prom szybszy
o ~9.5 min od TTL eventowego); zero fałszywych prom=down. vps/piha 100% zgodne.
Rekomendacja: GO dla etapu 3 per-node, mapping timestamp(up)->compute_liveness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
