Executor nie ma klienta ssh ani klucza do floty (uid 1000 homelab, brak
~/.ssh, brak resolucji nazw wezlow) — container_restart przez subprocess ssh
failowal w 6ms na kazdej probie. Zamiast dodawac SSH do executora, kierunek
jest odwrocony: executor zapisuje zlecenie do
/opt/homelab/actions/dispatch/<node>/<action_id>.json, a node-agent na
docelowym wezle (ktory ma dzialajacy docker.sock i juz ma klucz SSH do VPS
uzywany do shippingu eventow) sam je odbiera i wykonuje lokalnie.
- executor: _dispatch_container_restart pisze zlecenie zamiast ssh;
_reconcile_running_actions konsumuje zwrotne action_result eventy i
timeoutuje akcje bez odpowiedzi (ACTION_TIMEOUT_SECS, domyslnie 300s).
redeploy/disk_cleanup/alert_only bez zmian.
- node-agent: nowy krok w petli — rsync-pull wlasnej podkatalogu dispatch z
VPS (ten sam klucz co _ship_events_to_vps, w przeciwnym kierunku; no-op na
VPS, gdzie katalog jest lokalny), walidacja (node_name, whitelist tylko
container_restart, odmowa restartu wlasnego kontenera), wykonanie przez
docker SDK, raport jako event action_result (istniejacy kanal shippingu).
Idempotencja przez znacznik w /opt/homelab/state/processed-actions/.
- 26 nowych testow (10 executor, 16 node-agent), pelny suite obu serwisow
183/183 zielony.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First production run of the api adapter against ken (192.168.31.7,
HAOS): 118 automations, 5 scripts, 3 scenes, 7 dashboards (default + 6
named — one registered dashboard was never configured, config_not_found,
reported not fatal), area/entity registries, and 5 input_* helper
domains present on this instance.
ken is HAOS (no SSH/docker exec path), so it needs a REST/WebSocket-only
adapter: automations/scripts/scenes via one GET per
/api/config/<domain>/config/<id>, dashboards/area+entity registries/
input_* helpers via the HA WebSocket API (read-only commands only).
scripts/ha/lib/ha_api.py and ha_ws.py never take a token as a value —
only a token_path, read from disk in-process — so the bearer token never
touches a subprocess argv or a log line. ha_ws.py depends on the optional
websocket-client package and raises a clear, actionable ImportError if
it's missing rather than a raw traceback; import.sh still completes the
REST-only part of the import in that case.
Automations/scripts/scenes reuse split.write_split() so both adapters
produce byte-identical config/<instance>/ layouts and the same
idempotent stale-file cleanup on re-run.
docker-exec adapter logic is untouched.
Co: operator-ui (18180:8080) bindowal na 0.0.0.0, wiec /action/mutate
(brak autoryzacji, moze przenosic akcje do "approved") byl osiagalny z
publicznego internetu. Fix stosuje istniejacy wzorzec repo
(TAILSCALE_BIND_IP env var, patrz fleet-prometheus/llm-gateway/gokapi) +
dual-bind jak w ollama (127.0.0.1 obok TAILSCALE_BIND_IP), bo node-agent
na VPS laczy sie z network_mode: host przez localhost:18180/summary.
Nie ruszono operator_ui.py / mutate_action — auth to osobny temat.
Krok 4 of the phase-4 plan done ahead of the local-embed-fallback step
(Krok 2, deliberately deferred -- embed stays a plain SOLARIA call, per
task instruction): one FastAPI process now serves both the /search API
and the UI, no separate frontend build (plan §2 decision 4).
- GET / renders a Jinja2 shell; app/static/app.js (vanilla, no build) and
style.css are the whole client. Query -> /search, results grouped by
envelope_id client-side (chunks sorted by dist, <details> fragments).
- Colour thresholds per plan §7: dist<0.45 green, 0.45-0.55 yellow (still
shown with a warning), >0.55 never rendered as an individual result; if
a query ends up with nothing renderable, one "Brak odpowiedzi w KB"
message replaces the list, carrying the best observed dist.
- Paperless hits link out; gmail hits get a "kopiuj Message-ID" button
(there's nothing to link to yet, plan §2 decision 3) plus header
metadata. Cascade/flat toggle defaults to cascade. Footer shows
sol_status, refreshed from /healthz on load and after each search.
- /search gained additive summary/summary_tags fields (document_summary,
haiku track) so the UI can show a document summary as each result
group's header -- non-breaking, existing response shape untouched.
- Tests: app/db.py + app/search.py unit tests (mocked DB/HTTP, no live
deps) cover the new summary join; tests/test_frontend.py drives GET /
and /static/* via TestClient without running the DB-requiring lifespan;
tests/frontend/app.test.js (Node's built-in test runner, no framework)
covers query-URL encoding, threshold colouring, and envelope grouping.
- Verified live: docker build + container against kb-postgres@PIHA over
LAN and Ollama@SOLARIA over Tailscale -- GET / (HTML), /static/app.js,
/healthz, and /search (cascade + flat) all round-tripped correctly,
including real summary/summary_tags data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Read-only import of the pre-migration homeassistant5 container (piha),
per DESIGN.md phase-0 scope: 60 automations (split one-file-per-id),
5 scenes, 1 script, plus curated .storage/* export (area/entity
registries, input_boolean/input_text, lovelace dashboards).
No token at ~/.config/ha-deploy/ken-legacy.token — fixtures step
fail-softed as designed, no repo import.sh changes needed. Verified:
gitignore-excluded paths (secrets.yaml, *.db*, logs, .storage/auth*,
core.restore_state, tts/, deps/, backups/, .cloud/) correctly stripped
from the pull; three consecutive import.sh runs produced byte-identical
output under config/ken-legacy/ and storage-export/ken-legacy/
(idempotent); automation count matches the 2026-07-22 recon estimate
(~60) exactly. README.md only touched to note the import date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
instances.yaml pointed "ken" at the piha container (homeassistant5), a
pre-migration instance that never actually stopped running — it kept
firing automations in parallel with the real home instance (HAOS on a
dedicated RPi4, 192.168.31.7), causing duplicate triggers on shared
buttons/scenes. Repoint ken to the real instance (api adapter),
demote the container to ken-legacy (archived, import-only, never a
deploy target), and log the incident in DESIGN.md.
Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first
user-facing HTTP entry point to the KB. FastAPI wrapping
kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via
Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source
links) and GET /healthz. Search API only, no answer synthesis (phase 5) and
no server-side dist filtering — the 0.45/0.55 colour thresholds are a
frontend concern (plan §7, a later step).
Hard startup invariant (plan §2 decision 2): refuses to start unless the
configured EMBED_MODEL is present in both document_chunk.model and
document_summary.embedding_model. Note the latter: document_summary.model is
the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the
embedder — checked live against kb-postgres@PIHA before writing this, see
app/startup.py's docstring. Verified end-to-end with a live docker run: the
invariant crash-loops on a mismatched EMBED_MODEL and passes through to a
real /search hit against the live corpus with a correct model.
Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local
embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and
a failure surfaces as 503, not a crash.
Also: scripts/deploy/deploy.sh's gate now builds each service via
`docker compose build` instead of a raw `docker build <svc_dir>`, so a
service whose docker-compose.yml declares a repo-root build context (needed
here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention
already documented in CLAUDE.md) resolves the same way in the gate as it
does at real deploy time (deploy-node.sh's `docker compose ... up --build`).
No behavior change for existing single-context services — verified against
llm-gateway's compose file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py):
orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic,
claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA
(availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu
etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na
kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał
tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla
cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem.
Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors,
llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie
blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis,
last_success_timestamp trzymany z poprzedniego pliku przy failu) do
/opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu).
systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer
(OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User
oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do
/opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w
README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge.
fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical)
+ KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem
brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml).
node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override
(--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) +
topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter
owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent
zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA.
Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run
złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym
taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń,
0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY
dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany).
Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge),
zdecydowanie czy stability-agent też idzie na per-host przy okazji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause rozjazdu panelu agents.okit.pl vs services.json NIE lezal w
observer.py/operator_ui.py (te sa poprawne: ghost-prune dziala in-memory
i na plik co cykl, operator_ui czyta services.json na zywo bez cache).
Prawdziwy lancuch: npm proxy_host agents.okit.pl -> 100.108.208.3:18180
(Tailscale IP PIHA, nie VPS!) -> legacy kontener agent-system-webui
(services/agent-system/, sprzed migracji do control-plane/, brak
service.yaml, nieobecny w hosts/piha/services.yaml i topology.yaml).
Jego runtime-materializer czytal z Redis (homelab:services:*), do ktorego
NIC juz nie pisze w obecnej architekturze -- stad 27 martwych/ghost
wpisow (w tym hash-prefixed) i inna liczba serwisow (110 vs 117 w
services.json z VPS).
Fix na to juz istnial w repo od 2026-05-27 (7277bdc): materializer.py ma
materialize_from_api() ktory mirroruje czysty output observera z VPS przez
CONTROL_PLANE_URL, a hosts/piha/runtime/agent-system/docker-compose.override.yml
ustawia ta zmienna. Nigdy sie jednak nie aktywowal, bo
services/agent-system/deploy.sh (jedyna sciezka deployu tego serwisu)
wolal `docker compose up` uzywajac WYLACZNIE docker-compose.yml, bez
dolaczania override'u z hosts/ -- w odroznieniu od control-plane/
deploy-local.sh i stability-agent/deploy-local.sh, ktore ten wzorzec juz
stosuja.
Fix: deploy.sh dolacza teraz hosts/piha/runtime/agent-system/docker-compose.override.yml
(ten sam wzorzec co control-plane, ktory hardkoduje vps). Po nastepnym
`services/agent-system/deploy.sh` na PIHA runtime-materializer zacznie
mirrorowac /nodes /services /summary itd. z control-plane API zamiast
Redis -- panel bedzie pokazywal to samo co services.json.
Docker socket PIHA (DOCKER_API_ERROR 2026-07-17T00:00:40Z): NIE regres
group_add/gid (node-agent ma poprawne grupy 999/123, gid docker.sock=123
sie zgadza). Wszystkie ~40 kontenerow na PIHA wystartowaly jednoczesnie o
00:00:26 UTC -- to byl automatyczny apt-get upgrade docker-ce
29.6.1->29.6.2 (apt/history.log, Start-Date 02:00:08 CEST), ktory
zrestartowal Docker Engine. stability-agent (root) trafil na gniazdo w
~1-sekundowym oknie zanim daemon w pelni wstal (docker.service
ActiveEnterTimestamp 02:00:41 CEST). Jednorazowy, samo-naprawiony,
zdarzenie sie nie powtorzylo. Brak zmiany kodu.
Testy: ast.parse (observer.py, operator_ui.py, materializer.py) OK,
bash -n deploy.sh OK, docker compose config (merge z override) poprawnie
wstrzykuje CONTROL_PLANE_URL, pytest services/control-plane/tests 114
passed, pytest services/agent-system/telegram-bot/tests 8 passed.
Co NIE zrobiono (poza zakresem/deploy nalezy do operatora): rzeczywisty
redeploy agent-system na PIHA + weryfikacja panelu na zywo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migracja 004: document_summary (envelope_id, summary, tags JSONB, model, embedding
VECTOR(1024) + HNSW cosine, embedding_model, UNIQUE(envelope_id, model) od razu — wzorzec
002/003). Zastosowana na żywej bazie kb-postgres@PIHA.
Job documents-ingest-summarize: wejście = document_chunk.text WHERE excluded_reason IS NULL
per koperta source='paperless' (duplikaty przez entities[duplicate_of] pomijane w całości),
wymuszony JSON {summary, tags} przez --backend ollama|anthropic, słownik tagów kontrolowany
(tags-vocab.yaml) z max 3 free-form, map-reduce dla dokumentów >200k znaków (grupy ~20
chunków), bilans + idempotencja + izolacja błędów per wiersz wg wzorców rodziny jobów.
Osobny --embed-summaries (bge-m3, reużywa chunk_embed.embed_chunk).
Bug znaleziony i naprawiony w trakcie pilota: brak options.num_ctx w wywołaniach Ollamy
powodował, że gemma3:12b używał domyślnego runtime kontekstu (~2048 tok), nie
zadeklarowanego 128k — dla 71/157 dokumentów (45%, >8k znaków) treść była cicho ucinana
(zweryfikowane: prompt_eval_count=2051 dla dokumentu 93k znaków). Naprawa: compute_num_ctx()
liczy num_ctx z długości promptu (~3 znaki/token, cap 131072). Cały tor lokalny przeliczony
od zera po naprawie.
Wynik pilota (186 dok. paperless, 3 duplikaty, 26 bez aktywnych chunków → 157 oczekiwanych
na tor): tor lokalny (gemma3:12b) 155/157 (2 izolowane błędy JSON po retry: paperless:24,
paperless:61), tor referencyjny (claude-haiku-4-5) 157/157, 0 błędów JSON, 0 tagów
ucinanych — słownik przestrzegany w 100% przypadków. Oba komplety zembedowane (bge-m3).
Znaleziony przy okazji: prompt do tagów wymagał dopracowania — pierwsza wersja pozwalała
modelowi zwracać tagi po angielsku spoza słownika; wzmocniona instrukcja (słownik w
pierwszej kolejności, "nigdy po angielsku") poprawiła zgodność w 2/3 przypadków testowych.
Porównanie A/B (~15 dok.) i weryfikacja końcowa (bilans, sanity SQL, retrieval po summary)
odłożone do następnej sesji.
Testy: 157 (152 nowe/summarize.py + istniejące), mocki API/Ollama/DB, bilans, idempotencja,
regresja num_ctx.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (sposób podania
klucza API, wybór modelu lokalnego gemma3:12b, naprawa+przeliczenie całego toru lokalnego po
odkryciu buga num_ctx) — udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- node-agent: service_healthy emitowany tylko przy przejsciu w stan zdrowy
(per-service in-memory state), nie co cykl dla kazdego zdrowego serwisu.
To samo dla control-plane HTTP probe. healthcheck_failed/containers_not_running/
incydenty pozostaja emitowane bez zmian (realne sygnaly).
- node-agent: naprawiono _cleanup_control_plane_fs — istniejaca retencja
eventow byla martwa od migracji observer_checkpoint.json na epoch-int
(str(path) <= int rzucal TypeError, lapane cicho przez broad except).
Teraz porownuje epoch-do-epoch i czysci wylacznie service_healthy/node_health
starsze niz checkpoint + 3-dniowy bufor; healthcheck_failed/incydenty/ha_*
zachowane bezterminowo.
- scripts/maintenance/cleanup_event_backlog.py: jednorazowy skrypt do
bezpiecznego czyszczenia backlogu na VPS (dry-run domyslnie, --apply
do usuniecia). Ten sam warunek: typ szumu + starsze niz checkpoint + 1h bufor.
Migracja 003 (UNIQUE+model, excluded_reason) zastosowana na żywej bazie kb-postgres@PIHA
(2683 chunki, bez DELETE). Kalibracja heurystyki ocr_junk (3 sygnały z planu §3.1) ujawniła
realną sprzeczność z planem: sygnał 1 (dowolny znak kontrolny) fałszywie łapał paperless:119
(wymagany aktywny) i legalny angielski tekst — próg doprecyzowany do >=5 wystąpień na
podstawie rozkładu na korpusie. 8 chunków oflagowanych ocr_junk po odjęciu potwierdzonych
fałszywych alarmów (kalendarz, mikro-fragmenty referencyjne).
Dedup: SQL z planu (exact content hash) znalazł 2 pary, ale nie wykrył znanego z pilota
duplikatu paperless:14≡74 (różne OCR, 99.2% chunków identycznych treściowo) — dodany fuzzy
check na potwierdzenie. Odrzucono 3 kandydatury o wysokim nakładaniu jako różne
wersje/typy dokumentów dzielące boilerplate PZU, nie duplikaty. 130 chunków oflagowanych
duplicate + entities[duplicate_of] na 3 kopertach.
chunk_embed.py: heurystyka is_ocr_junk() przed embedem (junk -> insert bez wywołania Ollamy,
embedding=NULL), ON CONFLICT rozszerzony o model, nowy licznik chunks_junk_flagged w bilansie,
testy (kody kreskowe, mojibake nie-junk, dot-leader, idempotencja, model w kluczu konfliktu).
Weryfikacja: 7 zapytań eval-setu z WHERE excluded_reason IS NULL — żadne trafienie nie
degraduje, kontrole negatywne bez zmian (>0.55), śmieć zniknął z top-5 zapytania 2,
paperless:119 pozostał aktywnym trafieniem. Bilans: 2545 aktywne / 130 duplicate / 8 ocr_junk.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (kalibracja
progu sygnału 1, dołączenie 14≡74 mimo braku exact-hash matcha, odrzucenie 3 fałszywych
kandydatur dedup) — plan przewidywał, że heurystyka będzie się mylić; wszystkie decyzje
udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An "unknown"/empty node_name silently produced evt-unknown-* event files. One
landed in events/piha/ and, being lexically greater than every evt-piha-* name,
pinned the observer's (then lexical) checkpoint and blocked PIHA for ~34 days.
Defense in depth, two layers, both refuse rather than emit poison:
- config.py: node_name field_validator rejects ""/"unknown" (case-insensitive,
trimmed); validate_default=True so the "unknown" default itself is rejected
when NODE_NAME never reaches the process. main() catches ValidationError and
exits 1 with a clear FATAL message instead of a raw traceback.
- event_emitter.py: EventEmitter.__init__ guards node_name at the exact poison
site (node_name is embedded in the filename), so no future call path can
reintroduce evt-unknown-*.
Precedence unchanged and correct: Settings.load() uses os.environ.setdefault, so
env NODE_NAME wins over YAML; YAML only supplies node_name when env is absent.
api.py never writes event files (only /health, /trigger), so its _node_name
global cannot produce poison — verified, left as-is.
Tests (services/ha-diag-agent/tests/): new test_config.py (env load, env>YAML
precedence, fail-fast on unknown/empty/whitespace/default/YAML-unknown) and
EventEmitter guard tests. Full unit suite: 139 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live cutover 2026-07-15 odkrył: nvidia-smi not found, brak libnvidia-ml
w ldcache — host nigdy nie miał sterownika, natywna Ollama mieliła na CPU.
Sekcja deploy.resources (nvidia) zakomentowana do czasu instalacji sterownika
(backlog). nvidia-container-toolkit doinstalowany na hoście (prerequisite,
brakował w runbooku). Kontener zweryfikowany: 5/5 modeli z bind-mounta,
embed bge-m3 dim=1024, reachable z PIHA przez tailnet.
Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.
Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.
Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
check_containers() classified only exited/dead, running+unhealthy and running.
Docker's "restarting", "paused", "removing" and any future state matched no
branch, so a crash-looping container (restart policy + continuous crash) emitted
ZERO events and world-state kept showing it "healthy" (observed live:
lustro/pi-watchtower-1). Every Docker state is now handled:
- restarting + RestartCount >= CRASH_LOOP_RESTART_THRESHOLD (default 3):
reuse containers_not_running (high, crash_loop=true) — parity with exited/dead,
rides the existing supervisor-wired remediation path.
- restarting below threshold: new observational container_restarting (low),
visible but non-actionable so benign post-deploy restarts don't alarm.
- paused / unknown-or-future state: new diagnostic container_state_unexpected
(medium) — no more silent fall-through; new Docker states become visible.
- removing: conscious documented skip (ephemeral teardown).
- created: unchanged skip (compose tracking artifact).
RestartCount (top-level inspect field) distinguishes a crash-loop from a one-off
restart. Adds services/node-agent/tests/test_check_containers.py pinning the full
state table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.
Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.
- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
checkpoints (and the older last_processed_file format) to int epochs;
unparseable → 0 (reprocess all — safe, process_event is idempotent on
last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
but-newer processed, unparseable name falls back to mtime (not wedged),
ts-not-path ordering, both checkpoint-format migrations, helper units.
Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Krok 1 planu docs/kb/modules/05-faza2-plan.md §3/§6 (chunk-level embeddings,
1:N do envelope). Addytywna — 001_envelope.sql nietknięta (zweryfikowane
\d envelope po migracji: identyczny schemat + FK jako "Referenced by").
Schemat wg rekomendacji recon (§2 decyzja 1+2): osobna tabela (nie kolumna
w envelope, bo N-wartościowy chunking jest obowiązkowy przy dokumentach
>8k tokenów), embedding VECTOR(1024) pod bge-m3 (dense), HNSW cosine index,
kolumna `model` do trywialnego re-indexu przy zmianie modelu (kb-00 zasada
#1: indeks odtwarzalny). Idempotentna (CREATE TABLE/INDEX IF NOT EXISTS,
zweryfikowane podwójnym uruchomieniem na kb-postgres@PIHA — drugi run same
NOTICE "already exists, skipping").
Zastosowana na żywej bazie: ssh piha docker exec kb-postgres psql, po
potwierdzeniu SQL przez Oskara. \dt + \d document_chunk + \d envelope
zweryfikowane po migracji.
Testy: dopisane sanity-testy 002 do packages/kb-mail/tests/test_migration.py
(wzorzec 001 — statyczne assercje na treści SQL, bez DB), 13/13 zielone.
Co NIE jest częścią tego kroku (§3 planu, odłożone): entity/entity_link
(graf encji) — szkic na przyszłość, nie blokuje domknięcia modułu 5.
Co dalej (plan §6, poza zakresem tego kroku): ollama pull bge-m3 na SOLARII,
token API Paperless, jobs/gmail-header-backfill/, adapter Paperless→koperta,
chunking+embed job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two config bugs found on the already-deployed split-host OCR worker
(module 3): (1) `command: celery ...` was routed through manage.py by
the image entrypoint because it didn't start with "/" — fixed with an
absolute gosu+celery path. (2) SCRATCH_DIR (/tmp/paperless) was not
shared over NFS like data/media/consume, so tasks picked up by
worker@SOLARIA instead of worker@PIHA failed with "File not found" —
fixed by adding a paperless_scratch NFS volume/bind mount on both
sides.
Verified live on PIHA + SOLARIA: test PDFs dropped into consume/ were
split across both workers, the SOLARIA-picked task completed OCR with
zero File not found errors, test documents cleaned up afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nextcloud zostaje prywatny (mesh/kapala.org); Gokapi to osobny publiczny
serwis do wysylania linkow do plikow na zewnatrz (Firefox Send alt).
owner_node=vps, storage lokalny dysk (nie S3), E2E encryption ON, port
53842 bindowany tylko na TAILSCALE_BIND_IP (hairpin NAT przez npm@VPS,
nigdy 0.0.0.0). Cutover checklist w README (DNS, wildcard *.okit.pl na
npm@VPS, setup wizard) — nie zdeployowane w tym commicie.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co zrobione:
- Nextcloud host = PIHA (always-on dla aktywnego uzycia), owner_node +
LAN_BIND_IP/TRUSTED_PROXIES w .env, README zaktualizowane
- Redis brokera Paperlessa: requirepass, PAPERLESS_REDIS_PASSWORD w .env
po obu stronach (PIHA + worker@SOLARIA), healthchecki z auth
- Domeny potwierdzone: paper.kapala.org, cloud.kapala.org (Cloudflare
DNS-only -> Tailscale PIHA, wildcard cert juz pokrywa) — udokumentowane,
nic nie utworzone
- Backup Paperlessa zatwierdzony: document_exporter + rsync/borg -> SOLARIA,
retencja 7/4/6, offsite jako future-note
- Nextcloud pin: 34-apache (zweryfikowany aktualny stable, endoflife.date)
- Whoosh fallback-worker: zaakceptowane bez zmian
- Porty/wylaczenie local login/sizing OCR-workera: przeniesione z "decyzji"
na "TODO przy deployu"
- DECYZJE-do-podjecia.md zaktualizowane: wszystko poza portami/loginem/
sizingiem przeniesione do "Rozstrzygniete"
Tylko edycja configow w repo — nic nie zdeployowane, zadne kontenery nie
byly ruszane, DNS/vhosty nie utworzone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kod zyl tylko na dysku PIHA w /opt/llm-gateway (bez gita). Przeniesiony do
services/llm-gateway/ pod pelny wzorzec homelaba:
- app/main.py: kod 1:1 z PIHA + OLLAMA_URL/CHAT_MODEL/CODE_MODEL
nadpisywalne przez env (defaulty bez zmian)
- docker-compose.yml: bind TYLKO do Tailscale IP (${TAILSCALE_BIND_IP},
wzorzec fleet-prometheus), nie 0.0.0.0 jak w starym compose
- service.yaml, env.example (bez sekretow), healthcheck.sh, README, testy
- hosts/piha/runtime/llm-gateway: mem_limit 256m (PIHA jest RAM-bound)
- rejestracja w hosts/piha/services.yaml i inventory/topology.yaml
Zepsuty /opt/llm-gateway/docker-compose.yml (zduplikowany klucz ports)
celowo NIE przeniesiony.
DoD: pytest 4 passed; docker build + smoke run OK (GET / -> gateway ok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test_run_once_* cases were flaky/order-dependent. Root cause: observer.observer
derives OBSERVER_STATE_FILE from STATE_DIR at import time. The helper patched
STATE_DIR but never OBSERVER_STATE_FILE, so run_once()/_save_checkpoint() wrote the
checkpoint to the real /opt/homelab/state/observer_checkpoint.json. Those node_checkpoints
(tmp paths tagged with a pytest run number) leaked across tests and across pytest runs;
run_once's `file_path > checkpoint` string compare then skipped/kept events based on
run-number ordering. The helper also never restored the module globals it overwrote.
Replace both ad-hoc helpers with an autouse monkeypatch fixture that redirects every
observer path — including OBSERVER_STATE_FILE — into the per-test tmp_path and reverts
them afterward. Tests no longer touch real disk and are deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously published "9090:9090" → Docker bound to 0.0.0.0 (all host interfaces,
including the public Hetzner IP), leaving tailscale-internal enforced only by the
VPS firewall. Now bind explicitly to the VPS Tailscale interface for
defense-in-depth: the port does not exist on the public IP at all.
- ports -> "${TAILSCALE_BIND_IP}:9090:9090"
- env.example: add TAILSCALE_BIND_IP (VPS Tailscale IP, verify via `tailscale ip -4`)
- README: deploy section — .env is mandatory; a missing .env makes Compose
silently bind 0.0.0.0 (warns, does not fail), so use --env-file and verify host_ip
Smoke: config with --env-file and with co-located .env both resolve
host_ip=100.95.58.48; with .env absent Compose warns and falls back to 0.0.0.0
(documented). .env is gitignored (global *.env rule).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clean scaffold of a Prometheus instance dedicated to fleet liveness, deliberately
separate from the home `prom` on PIHA. Scrapes only itself + the VPS-local
node_exporter for now.
- image pinned to prom/prometheus:v3.5.0 (LTS) — no :latest anti-pattern
- TSDB retention 15d / 2GB on the tight VPS; --web.enable-lifecycle for reloads
- mem_limit 512m, oom_score_adj +200 (sacrificial OOM victim before control-plane)
- tailscale-internal exposure via plain published 9090, mirroring control-plane
- node_exporter scraped via host.docker.internal:9100 (it runs network_mode: host)
- secret-free prometheus.yml; alerting/rule_files left as commented placeholders
- in-container healthcheck via busybox wget (present in the image; curl is not)
Smoke: docker compose config OK; promtool check config SUCCESS; up -d ->
/-/healthy + /-/ready OK; both targets (prometheus, fleet-node) report up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nodes that crash or lose connectivity without emitting node_offline
stay online in world state indefinitely. _prune_stale_world() now
flips any online node to offline if its last_seen is older than
NODE_OFFLINE_TTL_SECS (default 300 s = 5× the 60 s heartbeat interval).
Nodes with last_seen=None (never reported) and already-offline nodes
are left unchanged. Five new tests cover all branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses review findings — all reverting to the live config that worked:
- config.yml mounts at /app/vikunja/config.yml (live path), not /etc/vikunja.
- Remove the vikunja container healthcheck: the image ships no wget/curl, so
an in-container HTTP check is always unhealthy. Health stays on db
(pg_isready) + host-side healthcheck.sh (curl). Live had no app healthcheck.
- Secrets injected exclusively via env_file (.env) on BOTH services; dropped
the ${VAR:?} parse-time interpolation that depended on a .env in cwd. db now
also has env_file. Non-secret env stays inline.
- Rename .env.example -> env.example to match repo convention (forgejo).
Verified: `docker compose -f services/vikunja/docker-compose.yml config` passes;
services/vikunja/.env is gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wraps the existing manually-run Vikunja instance on PIHA in the standard
service layout. No deploy — definition only, for review.
- services/vikunja/: docker-compose.yml (vikunja + postgres:16-alpine),
service.yaml, README, healthcheck, config.yml (OIDC, secret-free),
.env.example. Real .env stays gitignored.
- Pins EXISTING named volumes (vikunja_vikunja_db, vikunja_vikunja_files)
so DB + uploaded files survive cutover.
- extra_hosts forgejo.okit.pl=192.168.31.5 (npm on PIHA) so OIDC discovery
resolves over LAN instead of flaky public DNS.
- OIDC client secret injected via env (VIKUNJA_AUTH_OPENID_PROVIDERS_
FORGEJO_CLIENTSECRET); config.yml keeps trailing-slash authurl/redirecturl.
- Registers vikunja in hosts/piha/services.yaml + inventory/topology.yaml,
plus hosts/piha/runtime/vikunja override (advisory mem_limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Recovery from bad merge of task/observer-poison-quarantine (c255a02)
which carried false deletes from a stale branch base. Re-applies only
the genuine observer changes on top of correct master state.
When an event file fails to parse (malformed JSON, truncated, corrupted),
the observer previously kept retrying on every cycle while the node's
checkpoint stayed pinned — all subsequent good events for that node lost.
Now: first parse failure -> atomic os.replace to STATE_DIR/observer_failed_events/<node>/
with collision handling. Checkpoint advances, downstream events flow.
Move failures are logged but don't crash the loop.
Complementary to the atomic_write_json fix on state files; this addresses
the same race-pattern on event files instead.
Regression test asserts: bad event quarantined to failed_events dir,
removed from hot path, subsequent good event processed (node online),
checkpoint moves to good event.
Port 8087 is no longer mapped to the host. Operator verify commands
that used curl http://localhost:8087/health now use docker exec with
Python's urllib (the image is python:3.11-slim, no curl binary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala