Bug: hit_at_3 returned None for kind=mail_hit rows (expected_envelope is
always null for them -- operator supplies query text, not a Message-ID), so
criterion 4 could never count a hit and read 0/5 despite hybrid distances of
0.25-0.42. Fixed with mail_hit_at_3: hit iff the top-3 distinct hybrid
envelopes include a mail-sourced one (envelope.source lookup via
fetch_envelope_sources, since hybrid_retrieve overwrites source to "hybrid"
on merge) under HIT_THRESHOLD. Also added a per-query no_answer_threshold
override in queries.yaml for criterion 3.
N2 ("piaskownica plastikowa") investigation: after Etap A added ~34k mail
chunks, N2's top-1 neighbor dropped to dist 0.5298 (< the 0.55 bar). Content
check showed it's a ski-school reservation newsletter (Rossignol ski sizes)
-- a semantic false-positive collision, not a real corpus match. M5, which
the operator had added assuming a genuine piaskownica mail existed, itself
misses (dist 0.5585) -- confirming there's no such mail in the corpus. M5
dropped; N2's pass bar lowered to 0.50 with a note documenting the collision.
Gate result on the live DB post-Etap A (Etap A: 13 009 mails scanned -> 33 871
new gmail chunks, 6 398 embedded / 27 473 newsletter-flagged, balanced +
idempotent on rerun; Ollama incident #4 during the run required a compose
force-recreate, not just restart -- root-cause task ollama-solaria-start-race
stays in backlog): all 4 criteria PASS (5/5 flat hits held, cascade hit@3 5/5
vs flat 4/5, negative controls above their bars, mail hit@3 4/4 after
dropping M5). Plan doc updated with the numbers and verdict table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The naive LIFO tag stack pushed every start tag and popped on any end tag by
position, not name. Real email HTML almost always writes void elements
(<meta>, <br>, <img>, ...) without a self-closing slash -- e.g. <head><meta
charset=...><meta name=viewport ...></head> pops a "meta" frame for the
literal </head>, leaving skip_depth stuck at 1 for the rest of the document.
Caught live during the faza-mailowa Etap A dry-run spot-check (plan §7,
Krok 4 calibration step) against real archived mail: a genuine HTML-only
promotional email extracted to '' entirely. Re-running the Etap A dry-run
after the fix dropped body_empty from 3253/13300 (24.5%) to 291/13300
(2.2%) -- the bug was silently discarding real content from a meaningful
slice of HTML-only mail.
Fix: void elements are never pushed onto the stack (so they can never
desync it); every other closing tag searches backward for its matching
open tag and truncates the stack from there, which also self-heals other
malformed nesting instead of only the void-tag case. 3 new regression
tests reproduce the exact <head><meta>...</head> pattern, br/hr line
breaks, and a stray unmatched closing tag.
retrieval_eval.py now runs every existing query through hybrid_query
alongside flat/cascade, and gate criteria 1/3 (no degradation, negative
controls) check hybrid too -- this is the phase's key regression test: does
adding mail chunks to the HNSW index degrade the 7 existing paperless
queries? Criterion 4 (mail hit@3 in hybrid) is wired but skipped with an
explicit report note when queries.yaml's new `mail_queries` placeholder is
empty, rather than silently PASSing/FAILing on absent data.
queries.yaml: added an empty `mail_queries: []` placeholder + inline format
docs. The existing 7-query `queries:` list is untouched.
REMINDER for the operator (per plan §8, Krok 5): 3-5 mail queries are still
needed -- things you know are in last year's mail, plus each one's expected
gmail Message-ID -- before criterion 4 and the full plan §8 gate can PASS.
Until then this only re-verifies the paperless regression check.
docs/kb/modules/05-faza-mailowa-plan.md, §8.
Mail (gmail) envelopes never get a document_summary (Decyzja 6 -- a mail
"summary" would usually be longer than the mail itself), so they're invisible
to the cascade's stage-1 pre-filter. hybrid_retrieve runs the existing
cascade for summarized sources (paperless) and, in parallel, a direct chunk
scan restricted to summaryless_sources (gmail), merging both by dist -- same
embedder/cosine space, so the merge is a plain sort, no re-normalization.
hybrid_query mirrors cascade_query (one shared query embed).
kb-query: mode pattern extended to ^(cascade|flat|hybrid)$, /search routes
"hybrid" to hybrid_query. Default mode stays "cascade" until the quality gate
(plan §8) PASSes on the full mail corpus -- flipping the default, and
deploying this to the running kb-query container, are separate follow-ups
for the operator; this task only adds the code path + tests
(docs/kb/modules/05-faza-mailowa-plan.md, §6, Krok 3).
Second full pass over the gmail .eml archive (gmail-bulk-import's first pass
skipped inline text/plain and text/html on purpose). Per envelope: typed
parse with compat32 fallback -> body extraction (inline text/plain preferred,
HTML->text via a small stdlib HTMLParser otherwise) -> quote-strip (reply
markers + `>`-quoted lines, EN/PL/Outlook patterns) -> newsletter
classification (List-Unsubscribe/List-Id/Precedence, chunked but not
embedded, excluded_reason='newsletter') -> Temat/Od/Data prefix from the
already-backfilled entities[type=headers] (zero header re-parse) -> chunk via
kb_mail.chunking -> batched embed_batch (64) -> INSERT document_chunk.
In-Reply-To/References are appended as entities[type=threading] during the
same read (idempotent WHERE NOT EXISTS append, 1:1 with
gmail-header-backfill) -- the only DB writes are document_chunk INSERTs and
an additive envelope.entities UPDATE; the .eml archive stays read-only.
--dsn/KB_DSN, --archive-root, --since/--limit/--offset, --batch-size,
--apply (dry-run default). Idempotency keys on (envelope_id, chunk_index)
pre-fetched scoped to --model, built correctly from the start per the
plan's flagged chunk_embed.py precedent. A failed embed batch is isolated
(chunks_errors, no abort) for Ollama's documented instability; a wrong
embedding dimension aborts the whole run.
48 tests, DoD smoke run against live kb-postgres@PIHA confirmed wiring
(archive not yet rsync'd to SOLARIA, so all 5 rows correctly reported
missing_file). docs/kb/modules/05-faza-mailowa-plan.md, §5.
Batched embedding client (input as a list, batch 64 measured live at ~8-18
ms/chunk vs ~150-200 ms/chunk sequential through embed_chunk's
/api/embeddings) for jobs/mail-body-ingest -- embed_chunk stays the
single-text path for kb-query and paperless cyclic ingest. Validates the
response embedding count and each item's dimension (1024), raising
EmbeddingDimensionError (abort-run) on mismatch so a bad batch never silently
writes a mismatched vector (docs/kb/modules/05-faza-mailowa-plan.md, §4,
decision 5).
chunk_text/hard_split/split_paragraphs + 2400/600-char constants move out of
documents_ingest.chunk_embed into kb_mail.chunking so the upcoming
jobs/mail-body-ingest shares the exact same chunker instead of a copy-pasted
drift (the phase-4 lesson for retrieval.py -> packages/kb-retrieval, applied
again). chunk_embed.py re-exports unchanged; zero behavior change, tests moved
1:1 alongside the code (docs/kb/modules/05-faza-mailowa-plan.md, §3).
Executor nie ma klienta ssh ani klucza do floty (uid 1000 homelab, brak
~/.ssh, brak resolucji nazw wezlow) — container_restart przez subprocess ssh
failowal w 6ms na kazdej probie. Zamiast dodawac SSH do executora, kierunek
jest odwrocony: executor zapisuje zlecenie do
/opt/homelab/actions/dispatch/<node>/<action_id>.json, a node-agent na
docelowym wezle (ktory ma dzialajacy docker.sock i juz ma klucz SSH do VPS
uzywany do shippingu eventow) sam je odbiera i wykonuje lokalnie.
- executor: _dispatch_container_restart pisze zlecenie zamiast ssh;
_reconcile_running_actions konsumuje zwrotne action_result eventy i
timeoutuje akcje bez odpowiedzi (ACTION_TIMEOUT_SECS, domyslnie 300s).
redeploy/disk_cleanup/alert_only bez zmian.
- node-agent: nowy krok w petli — rsync-pull wlasnej podkatalogu dispatch z
VPS (ten sam klucz co _ship_events_to_vps, w przeciwnym kierunku; no-op na
VPS, gdzie katalog jest lokalny), walidacja (node_name, whitelist tylko
container_restart, odmowa restartu wlasnego kontenera), wykonanie przez
docker SDK, raport jako event action_result (istniejacy kanal shippingu).
Idempotencja przez znacznik w /opt/homelab/state/processed-actions/.
- 26 nowych testow (10 executor, 16 node-agent), pelny suite obu serwisow
183/183 zielony.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First production run of the api adapter against ken (192.168.31.7,
HAOS): 118 automations, 5 scripts, 3 scenes, 7 dashboards (default + 6
named — one registered dashboard was never configured, config_not_found,
reported not fatal), area/entity registries, and 5 input_* helper
domains present on this instance.
ken is HAOS (no SSH/docker exec path), so it needs a REST/WebSocket-only
adapter: automations/scripts/scenes via one GET per
/api/config/<domain>/config/<id>, dashboards/area+entity registries/
input_* helpers via the HA WebSocket API (read-only commands only).
scripts/ha/lib/ha_api.py and ha_ws.py never take a token as a value —
only a token_path, read from disk in-process — so the bearer token never
touches a subprocess argv or a log line. ha_ws.py depends on the optional
websocket-client package and raises a clear, actionable ImportError if
it's missing rather than a raw traceback; import.sh still completes the
REST-only part of the import in that case.
Automations/scripts/scenes reuse split.write_split() so both adapters
produce byte-identical config/<instance>/ layouts and the same
idempotent stale-file cleanup on re-run.
docker-exec adapter logic is untouched.
Co: operator-ui (18180:8080) bindowal na 0.0.0.0, wiec /action/mutate
(brak autoryzacji, moze przenosic akcje do "approved") byl osiagalny z
publicznego internetu. Fix stosuje istniejacy wzorzec repo
(TAILSCALE_BIND_IP env var, patrz fleet-prometheus/llm-gateway/gokapi) +
dual-bind jak w ollama (127.0.0.1 obok TAILSCALE_BIND_IP), bo node-agent
na VPS laczy sie z network_mode: host przez localhost:18180/summary.
Nie ruszono operator_ui.py / mutate_action — auth to osobny temat.
Krok 4 of the phase-4 plan done ahead of the local-embed-fallback step
(Krok 2, deliberately deferred -- embed stays a plain SOLARIA call, per
task instruction): one FastAPI process now serves both the /search API
and the UI, no separate frontend build (plan §2 decision 4).
- GET / renders a Jinja2 shell; app/static/app.js (vanilla, no build) and
style.css are the whole client. Query -> /search, results grouped by
envelope_id client-side (chunks sorted by dist, <details> fragments).
- Colour thresholds per plan §7: dist<0.45 green, 0.45-0.55 yellow (still
shown with a warning), >0.55 never rendered as an individual result; if
a query ends up with nothing renderable, one "Brak odpowiedzi w KB"
message replaces the list, carrying the best observed dist.
- Paperless hits link out; gmail hits get a "kopiuj Message-ID" button
(there's nothing to link to yet, plan §2 decision 3) plus header
metadata. Cascade/flat toggle defaults to cascade. Footer shows
sol_status, refreshed from /healthz on load and after each search.
- /search gained additive summary/summary_tags fields (document_summary,
haiku track) so the UI can show a document summary as each result
group's header -- non-breaking, existing response shape untouched.
- Tests: app/db.py + app/search.py unit tests (mocked DB/HTTP, no live
deps) cover the new summary join; tests/test_frontend.py drives GET /
and /static/* via TestClient without running the DB-requiring lifespan;
tests/frontend/app.test.js (Node's built-in test runner, no framework)
covers query-URL encoding, threshold colouring, and envelope grouping.
- Verified live: docker build + container against kb-postgres@PIHA over
LAN and Ollama@SOLARIA over Tailscale -- GET / (HTML), /static/app.js,
/healthz, and /search (cascade + flat) all round-tripped correctly,
including real summary/summary_tags data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Read-only import of the pre-migration homeassistant5 container (piha),
per DESIGN.md phase-0 scope: 60 automations (split one-file-per-id),
5 scenes, 1 script, plus curated .storage/* export (area/entity
registries, input_boolean/input_text, lovelace dashboards).
No token at ~/.config/ha-deploy/ken-legacy.token — fixtures step
fail-softed as designed, no repo import.sh changes needed. Verified:
gitignore-excluded paths (secrets.yaml, *.db*, logs, .storage/auth*,
core.restore_state, tts/, deps/, backups/, .cloud/) correctly stripped
from the pull; three consecutive import.sh runs produced byte-identical
output under config/ken-legacy/ and storage-export/ken-legacy/
(idempotent); automation count matches the 2026-07-22 recon estimate
(~60) exactly. README.md only touched to note the import date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Native ESPHome gree climate component over the API, no MQTT/Broadlink/Tuya.
HA on piha runs as a plain Docker container (not HAOS), so ESPHome is a
manual docker compose run --rm invocation, not a managed service — device
is deliberately not registered in hosts/piha/services.yaml.
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
instances.yaml pointed "ken" at the piha container (homeassistant5), a
pre-migration instance that never actually stopped running — it kept
firing automations in parallel with the real home instance (HAOS on a
dedicated RPi4, 192.168.31.7), causing duplicate triggers on shared
buttons/scenes. Repoint ken to the real instance (api adapter),
demote the container to ken-legacy (archived, import-only, never a
deploy target), and log the incident in DESIGN.md.
Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first
user-facing HTTP entry point to the KB. FastAPI wrapping
kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via
Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source
links) and GET /healthz. Search API only, no answer synthesis (phase 5) and
no server-side dist filtering — the 0.45/0.55 colour thresholds are a
frontend concern (plan §7, a later step).
Hard startup invariant (plan §2 decision 2): refuses to start unless the
configured EMBED_MODEL is present in both document_chunk.model and
document_summary.embedding_model. Note the latter: document_summary.model is
the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the
embedder — checked live against kb-postgres@PIHA before writing this, see
app/startup.py's docstring. Verified end-to-end with a live docker run: the
invariant crash-loops on a mismatched EMBED_MODEL and passes through to a
real /search hit against the live corpus with a correct model.
Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local
embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and
a failure surfaces as 503, not a crash.
Also: scripts/deploy/deploy.sh's gate now builds each service via
`docker compose build` instead of a raw `docker build <svc_dir>`, so a
service whose docker-compose.yml declares a repo-root build context (needed
here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention
already documented in CLAUDE.md) resolves the same way in the gate as it
does at real deploy time (deploy-node.sh's `docker compose ... up --build`).
No behavior change for existing single-context services — verified against
llm-gateway's compose file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 4 step 0 (docs/kb/modules/05-faza4-plan.md, §3, decision 1):
kb-query is a long-lived Docker service, documents-ingest is a venv job with
an `anthropic` dependency and CLI scripts it doesn't need. Move
embed_chunk/_vector_literal/cascade_query/flat_query into a shared package
with minimal deps (asyncpg, aiohttp only) so both can depend on the same
tested retrieval code without the service image pulling in the job's extras.
documents_ingest.chunk_embed/retrieval keep thin re-exports so nothing
importing the old paths breaks. Pure refactor: retrieval_eval.py run live
against kb-postgres@PIHA + Ollama@SOLARIA before/after gives byte-identical
`dist`/hit@3/gate results (still PASS) — zero behavior change in the cascade.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yaml.safe_load has no constructor for HA's `!include`/`!secret`/etc. tags
and raised ConstructorError on any real configuration.yaml. Add a
multi_constructor on `!` that generically wraps scalar/sequence/mapping
payloads in a TaggedValue, and a matching representer that re-emits the
same tag + payload — normalization stays idempotent across passes and key
sorting is unaffected by tagged values.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Structure only, no deploy path: DESIGN.md decision registry (phasing,
per-instance adapter, bidirectional sync with drift-abort on deploy,
reload-vs-restart gating, token handling), instances.yaml (ken/piha via
docker-exec, chelsty-ha via api), config/storage-export/fixtures dirs,
and a read-only scripts/ha/import.sh (docker-exec adapter only) with a
canonical YAML normalize+split library and an offline determinism test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py):
orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic,
claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA
(availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu
etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na
kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał
tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla
cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem.
Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors,
llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie
blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis,
last_success_timestamp trzymany z poprzedniego pliku przy failu) do
/opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu).
systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer
(OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User
oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do
/opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w
README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge.
fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical)
+ KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem
brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml).
node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override
(--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) +
topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter
owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent
zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA.
Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run
złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym
taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń,
0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY
dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany).
Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge),
zdecydowanie czy stability-agent też idzie na per-host przy okazji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause rozjazdu panelu agents.okit.pl vs services.json NIE lezal w
observer.py/operator_ui.py (te sa poprawne: ghost-prune dziala in-memory
i na plik co cykl, operator_ui czyta services.json na zywo bez cache).
Prawdziwy lancuch: npm proxy_host agents.okit.pl -> 100.108.208.3:18180
(Tailscale IP PIHA, nie VPS!) -> legacy kontener agent-system-webui
(services/agent-system/, sprzed migracji do control-plane/, brak
service.yaml, nieobecny w hosts/piha/services.yaml i topology.yaml).
Jego runtime-materializer czytal z Redis (homelab:services:*), do ktorego
NIC juz nie pisze w obecnej architekturze -- stad 27 martwych/ghost
wpisow (w tym hash-prefixed) i inna liczba serwisow (110 vs 117 w
services.json z VPS).
Fix na to juz istnial w repo od 2026-05-27 (7277bdc): materializer.py ma
materialize_from_api() ktory mirroruje czysty output observera z VPS przez
CONTROL_PLANE_URL, a hosts/piha/runtime/agent-system/docker-compose.override.yml
ustawia ta zmienna. Nigdy sie jednak nie aktywowal, bo
services/agent-system/deploy.sh (jedyna sciezka deployu tego serwisu)
wolal `docker compose up` uzywajac WYLACZNIE docker-compose.yml, bez
dolaczania override'u z hosts/ -- w odroznieniu od control-plane/
deploy-local.sh i stability-agent/deploy-local.sh, ktore ten wzorzec juz
stosuja.
Fix: deploy.sh dolacza teraz hosts/piha/runtime/agent-system/docker-compose.override.yml
(ten sam wzorzec co control-plane, ktory hardkoduje vps). Po nastepnym
`services/agent-system/deploy.sh` na PIHA runtime-materializer zacznie
mirrorowac /nodes /services /summary itd. z control-plane API zamiast
Redis -- panel bedzie pokazywal to samo co services.json.
Docker socket PIHA (DOCKER_API_ERROR 2026-07-17T00:00:40Z): NIE regres
group_add/gid (node-agent ma poprawne grupy 999/123, gid docker.sock=123
sie zgadza). Wszystkie ~40 kontenerow na PIHA wystartowaly jednoczesnie o
00:00:26 UTC -- to byl automatyczny apt-get upgrade docker-ce
29.6.1->29.6.2 (apt/history.log, Start-Date 02:00:08 CEST), ktory
zrestartowal Docker Engine. stability-agent (root) trafil na gniazdo w
~1-sekundowym oknie zanim daemon w pelni wstal (docker.service
ActiveEnterTimestamp 02:00:41 CEST). Jednorazowy, samo-naprawiony,
zdarzenie sie nie powtorzylo. Brak zmiany kodu.
Testy: ast.parse (observer.py, operator_ui.py, materializer.py) OK,
bash -n deploy.sh OK, docker compose config (merge z override) poprawnie
wstrzykuje CONTROL_PLANE_URL, pytest services/control-plane/tests 114
passed, pytest services/agent-system/telegram-bot/tests 8 passed.
Co NIE zrobiono (poza zakresem/deploy nalezy do operatora): rzeczywisty
redeploy agent-system na PIHA + weryfikacja panelu na zywo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
documents_ingest.retrieval: flat_query (baseline) i cascade_query (stage1
document_summary model='claude-haiku-4-5' -> stage2 document_chunk), jedno
dzielone wywołanie embeddingu bge-m3 per zapytanie, tylko +1 SQL na kaskadę.
Czyste query_text -> wyniki(dist, source) pod przyszłe kb-query fazy 4.
166/166 testów (10 nowych, mocki: stage1->stage2, koperta bez chunków,
N > liczba kopert, no-summaries short-circuit).
Eval-set utrwalony 1:1 z pilota (docs/kb/eval/retrieval-pilot-2026-07-16.md,
nietknięty) w eval/queries.yaml + skrypt bramki eval/retrieval_eval.py
(integracyjny, read-only, poza pytest).
Wynik bramki (żywa baza, N=10 k=5): kryterium 1 (brak degradacji) PASS,
kryterium 2 (hit@3 kaskada=5/5 vs płaski=5/5) PASS, kryterium 3 (kontrole
negatywne 0.644/0.553 > 0.55 w obu torach) PASS. Sweep N∈{1,2,3,5,10,20}:
N=5 to zmierzony próg bezpieczny (N<5 degraduje zapytania 3-4), N=10 ma
2x margines — potwierdza domyślną wartość z planu zamiast przyjmować ją
z założenia. Kaskada nie poprawia jakości na 186-dok. korpusie (dystanse
identyczne z płaskim przy N≥5) — zgodnie z przewidywaniem planu: to test
architektury pod skalę mailową, nie optymalizacja pilota.
Decyzja: kaskada (N=10, k=5, claude-haiku-4-5) = domyślna ścieżka retrievalu.
Plan-doc §6.3 zaktualizowany wynikiem; §2 D3 zamknięte rozstrzygnięciem
Oskara (tor kompilacyjny=claude-haiku-4-5, gemma3:12b w odwodzie, decyzja
mailowa odłożona do reconu z flagą prywatności/kosztu).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Migracja 004: document_summary (envelope_id, summary, tags JSONB, model, embedding
VECTOR(1024) + HNSW cosine, embedding_model, UNIQUE(envelope_id, model) od razu — wzorzec
002/003). Zastosowana na żywej bazie kb-postgres@PIHA.
Job documents-ingest-summarize: wejście = document_chunk.text WHERE excluded_reason IS NULL
per koperta source='paperless' (duplikaty przez entities[duplicate_of] pomijane w całości),
wymuszony JSON {summary, tags} przez --backend ollama|anthropic, słownik tagów kontrolowany
(tags-vocab.yaml) z max 3 free-form, map-reduce dla dokumentów >200k znaków (grupy ~20
chunków), bilans + idempotencja + izolacja błędów per wiersz wg wzorców rodziny jobów.
Osobny --embed-summaries (bge-m3, reużywa chunk_embed.embed_chunk).
Bug znaleziony i naprawiony w trakcie pilota: brak options.num_ctx w wywołaniach Ollamy
powodował, że gemma3:12b używał domyślnego runtime kontekstu (~2048 tok), nie
zadeklarowanego 128k — dla 71/157 dokumentów (45%, >8k znaków) treść była cicho ucinana
(zweryfikowane: prompt_eval_count=2051 dla dokumentu 93k znaków). Naprawa: compute_num_ctx()
liczy num_ctx z długości promptu (~3 znaki/token, cap 131072). Cały tor lokalny przeliczony
od zera po naprawie.
Wynik pilota (186 dok. paperless, 3 duplikaty, 26 bez aktywnych chunków → 157 oczekiwanych
na tor): tor lokalny (gemma3:12b) 155/157 (2 izolowane błędy JSON po retry: paperless:24,
paperless:61), tor referencyjny (claude-haiku-4-5) 157/157, 0 błędów JSON, 0 tagów
ucinanych — słownik przestrzegany w 100% przypadków. Oba komplety zembedowane (bge-m3).
Znaleziony przy okazji: prompt do tagów wymagał dopracowania — pierwsza wersja pozwalała
modelowi zwracać tagi po angielsku spoza słownika; wzmocniona instrukcja (słownik w
pierwszej kolejności, "nigdy po angielsku") poprawiła zgodność w 2/3 przypadków testowych.
Porównanie A/B (~15 dok.) i weryfikacja końcowa (bilans, sanity SQL, retrieval po summary)
odłożone do następnej sesji.
Testy: 157 (152 nowe/summarize.py + istniejące), mocki API/Ollama/DB, bilans, idempotencja,
regresja num_ctx.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (sposób podania
klucza API, wybór modelu lokalnego gemma3:12b, naprawa+przeliczenie całego toru lokalnego po
odkryciu buga num_ctx) — udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- node-agent: service_healthy emitowany tylko przy przejsciu w stan zdrowy
(per-service in-memory state), nie co cykl dla kazdego zdrowego serwisu.
To samo dla control-plane HTTP probe. healthcheck_failed/containers_not_running/
incydenty pozostaja emitowane bez zmian (realne sygnaly).
- node-agent: naprawiono _cleanup_control_plane_fs — istniejaca retencja
eventow byla martwa od migracji observer_checkpoint.json na epoch-int
(str(path) <= int rzucal TypeError, lapane cicho przez broad except).
Teraz porownuje epoch-do-epoch i czysci wylacznie service_healthy/node_health
starsze niz checkpoint + 3-dniowy bufor; healthcheck_failed/incydenty/ha_*
zachowane bezterminowo.
- scripts/maintenance/cleanup_event_backlog.py: jednorazowy skrypt do
bezpiecznego czyszczenia backlogu na VPS (dry-run domyslnie, --apply
do usuniecia). Ten sam warunek: typ szumu + starsze niz checkpoint + 1h bufor.
Migracja 003 (UNIQUE+model, excluded_reason) zastosowana na żywej bazie kb-postgres@PIHA
(2683 chunki, bez DELETE). Kalibracja heurystyki ocr_junk (3 sygnały z planu §3.1) ujawniła
realną sprzeczność z planem: sygnał 1 (dowolny znak kontrolny) fałszywie łapał paperless:119
(wymagany aktywny) i legalny angielski tekst — próg doprecyzowany do >=5 wystąpień na
podstawie rozkładu na korpusie. 8 chunków oflagowanych ocr_junk po odjęciu potwierdzonych
fałszywych alarmów (kalendarz, mikro-fragmenty referencyjne).
Dedup: SQL z planu (exact content hash) znalazł 2 pary, ale nie wykrył znanego z pilota
duplikatu paperless:14≡74 (różne OCR, 99.2% chunków identycznych treściowo) — dodany fuzzy
check na potwierdzenie. Odrzucono 3 kandydatury o wysokim nakładaniu jako różne
wersje/typy dokumentów dzielące boilerplate PZU, nie duplikaty. 130 chunków oflagowanych
duplicate + entities[duplicate_of] na 3 kopertach.
chunk_embed.py: heurystyka is_ocr_junk() przed embedem (junk -> insert bez wywołania Ollamy,
embedding=NULL), ON CONFLICT rozszerzony o model, nowy licznik chunks_junk_flagged w bilansie,
testy (kody kreskowe, mojibake nie-junk, dot-leader, idempotencja, model w kluczu konfliktu).
Weryfikacja: 7 zapytań eval-setu z WHERE excluded_reason IS NULL — żadne trafienie nie
degraduje, kontrole negatywne bez zmian (>0.55), śmieć zniknął z top-5 zapytania 2,
paperless:119 pozostał aktywnym trafieniem. Bilans: 2545 aktywne / 130 duplicate / 8 ocr_junk.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (kalibracja
progu sygnału 1, dołączenie 14≡74 mimo braku exact-hash matcha, odrzucenie 3 fałszywych
kandydatur dedup) — plan przewidywał, że heurystyka będzie się mylić; wszystkie decyzje
udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plan-doc (zero kodu): porządki po pilocie retrieval (filtr OCR-śmieci przez
excluded_reason, dedup paperless:14≡74, UNIQUE+model — migracja 003),
document_summary (004, UNIQUE(envelope_id,model) od razu), pilot streszczeń+tagów
dwutorowo API+lokalny GPU z rubryką jakości jako wejście do decyzji mailowej,
kaskada summary→chunk z bramką nie-gorszości na utrwalonym eval-secie, cykliczny
ingest jako systemd-timer@PIHA z metrykami textfile→fleet-prometheus→brain-watchdog,
wiki-kompilat wg szkicu operatora (Karpathy llm-wiki) rozwinięty wykonawczo
(osobne repo kb-wiki, frontmatter z sources, strony jako koperty source='wiki').
7 decyzji z rekomendacjami w §2; granice fazy w §9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Driver fixed 2026-07-16 (nvidia-driver-595-open from distro repo, old
graphics-drivers PPA for jammy disabled) — RTX 4070 Ti SUPER 16GB, CUDA
13.2, nvidia-container-toolkit already present from the 07-15 cutover
prerequisite install. Uncomments deploy.resources.reservations (nvidia
gpu) in services/ollama/docker-compose.yml, restoring clean formatting.
Docs close out the loose ends from the 07-15 declarative cutover:
- ollama-solaria-cutover runbook gets a "Wykonanie" section documenting
what actually happened (mv instead of rsync for the model store, the
missing nvidia-container-toolkit prerequisite, the driver-missing
discovery, CPU-only cutover, then the 07-16 GPU fix) plus a note on
the container disappearing after the 07-15 evening reboot (one-off,
boots fine now, root cause not established).
- hosts/solaria/services.yaml: ollama role comment now reflects actual
GPU-backed state instead of the previously-aspirational wording.
- hosts/solaria/README.md: drop stale Open WebUI mention (not in repo).
- docs/backlog.md: close the NVIDIA driver item; leave two follow-ups
(Ollama call batching before the mail phase, UNIQUE(envelope_id,
chunk_index) schema change for multi-model embeddings at phase 3).
- jobs/documents-ingest/README.md: timing section gets a GPU placeholder
line next to the existing 0.79s/chunk CPU baseline, to be filled in
after the live GPU benchmark.
Live recreate + GPU-vs-CPU embedding benchmark deliberately left for the
operator to run from the main checkout after merge, per worktree-aware
discipline — this worktree only owns the declarative fix and the docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-embed: chunks source='paperless' envelope content
(paragraph-preferring, ~600 tok/chunk, ~150 tok overlap, hard char-fallback
for oversized paragraphs per plan §2 decision 3), embeds each chunk via
Ollama (bge-m3, dim validated against document_chunk's VECTOR(1024) on
every response) and inserts into document_chunk. Lives in documents-ingest
per the plan's own recommendation (§6 step 6) rather than a new package —
reuses the job family's existing idempotency/stats-balance/dry-run
conventions (paperless_adapter.py, gmail-header-backfill).
A 5-angle multi-agent code review of the initial implementation surfaced
three real bugs, fixed here: hard_split() could infinite-loop if
--chunk-overlap >= --chunk-size (now guarded in both hard_split() and
main()); insert_chunk() wasn't error-isolated like embed_chunk(), so a DB
write failure would crash the whole run instead of being counted and
skipped; and ON CONFLICT DO NOTHING's outcome was discarded, so a silently
skipped row (the known gap where document_chunk's UNIQUE constraint
doesn't include `model`) would have been miscounted as a successful insert
- now tracked separately as chunks_conflict_skipped and treated as a
run failure.
Smoke-tested and run to completion live on SOLARIA against the real Ollama
instance and kb-postgres@PIHA: dry-run matched the known phase-2-step-5
figures exactly (186 fetched, 26 empty_content, 2684 chunks planned), a
--limit 10 apply + idempotent re-run + DB/distance sanity checks all
passed, and the full 186-document run inserted 2683/2684 chunks (1 isolated
error - Ollama's runtime context window rejected one pathological
dot-leader table-of-contents chunk that tokenized far more densely than
estimated; documented as a known limitation, not fixed here given it's a
single-chunk edge case). Timing: ~0.79s/chunk average on CPU (SOLARIA's
Ollama runs GPU-less per the recent GPU-reservation-disabled fix), ~35 min
wall-clock for the full pilot - the real input for scoping the later
mail-corpus embedding phase (plan §7's GPU-based estimate doesn't hold
here).
pytest: 101 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
