homelab-codex-ws/services/kb-query/README.md

kb-query

FastAPI search API in front of the module-5 KB retrieval engine (packages/kb-retrieval/). Runs on PIHA, bound to PIHA's LAN IP only (exposure: private, same class as paperless/nextcloud — no public ingress yet). This is a search API, not chat: no answer synthesis over results, that's phase 5.

Endpoints

Endpoint	Method	Purpose
/	GET	Search UI (Jinja2 shell + /static/app.js, no login yet — plan §8 OIDC is a later step)
/static/*	GET	UI assets (app.js, style.css)
/healthz	GET	{"status": "ok", "sol_status": "up"|"down", "fallback_status": "up"|"down"|"unconfigured"} — sol_status comes from the embed router's ~30 s-cached SOLARIA probe (same world view /search routes by), fallback_status is a live cheap probe of ollama-piha; no auth required (monitoring must reach it)
/search?q=<text>&mode=cascade|flat	GET	query_text -> embed -> cascade_query/flat_query -> results, mode defaults to cascade

/search response shape (module 5 phase 4 plan §4, summary/summary_tags added in Krok 4 for the UI's per-envelope result header — additive, does not change any field the plan §4 shape already defined):

{
  "query": "...", "mode": "cascade", "sol_status": "up", "embed_backend": "solaria",
  "results": [
    {"envelope_id": "paperless:119", "source": "paperless", "dist": 0.34,
     "chunk_index": 2, "text": "...", "link": "https://paper.kapala.org/documents/119/details",
     "summary": "...", "summary_tags": ["..."]},
    {"envelope_id": "<Message-ID>", "source": "gmail", "dist": 0.44,
     "chunk_index": 0, "text": "...", "subject": "...", "from": "...", "date": "...",
     "link": null, "mail_ui_url": null, "summary": null, "summary_tags": []}
  ]
}

dist is never filtered server-side — the 0.45/0.55 colour thresholds (below) are a frontend concern, not an API contract. summary/summary_tags come from document_summary for SUMMARY_MODEL; null/[] when the envelope has no summary yet.

Frontend (Krok 4, plan §7)

One page, served from this same FastAPI process — no separate frontend container, no node build step (plan §2 decision 4): app/templates/index.html (Jinja2 shell) + app/static/app.js (vanilla JS, fetch() to /search) + app/static/style.css. Wszystko po polsku.

• Pole zapytania + submit (Enter lub przycisk), przełącznik trybu kaskada/flat (domyślnie kaskada — checkbox "tryb flat (debug)").
• Wyniki grupowane po envelope_id (dokument): nagłówek trafienia to streszczenie dokumentu (summary, tor haiku) gdy dostępne, w przeciwnym razie envelope_id; chunki są rozwijanymi fragmentami (<details>) pod nagłówkiem, posortowane po dist.
• Kolorowanie progów (fazy 3, zweryfikowane bramką): dist < 0.45 zielony, 0.45–0.55 żółty (nadal renderowany, z wizualnym ostrzeżeniem), > 0.55 nigdy nie renderowany jako pojedynczy wynik. Jeśli po tym filtrze żadna grupa nie zostaje nic do pokazania (wszystkie trafienia > 0.55, albo brak trafień w ogóle), całość zastępuje komunikat "Brak odpowiedzi w KB dla tego zapytania" z najlepszym (najniższym) zaobserwowanym dist w nawiasie.
• Źródło: Paperless → link "Otwórz w Paperless" (link); Gmail → metadane (subject/from/date) + przycisk "Kopiuj Message-ID" (envelope_id jest Message-ID, plan §2 decyzja 3) — nie ma dokąd linkować, więc kopiowalny identyfikator zamiast martwego linku.
• Stopka pokazuje sol_status dyskretnie (odświeżane z /healthz przy starcie strony i po każdym wyszukiwaniu).

Embed path — active SOLARIA→PIHA fallback (Krok 2, plan §2 D2/§5)

Query embeddings go through app/embed_router.py (EmbedRouter, one instance per process):

1. SOLARIA's health verdict (GET /api/tags, 1.5 s timeout) is cached for 30 s — no per-request probing.
2. Verdict up → embed on EMBED_PRIMARY_URL (SOLARIA GPU, ~207 ms) under a hard 3 s timeout.
3. A failure during a real embed flips the verdict to down for one TTL window and the same request is served from EMBED_FALLBACK_URL (ollama-piha, CPU, OLLAMA_KEEP_ALIVE=0 — slower, single seconds, but alive) — the user never sees a SOLARIA error while a fallback exists.
4. Verdict down → straight to the fallback until the TTL expires; when SOLARIA wakes up, traffic returns to the GPU within ≤30 s.

Every /search response and log line says which backend embedded the query (embed_backend: "solaria"|"piha", log backend=… in kb-query.embed) — needed to debug result quality per backend. sol_status in the response is the router's world view; the UI footer renders down as „SOLARIA: offline (fallback embed)".

Model invariant, both halves: at startup kb-query pins EMBED_MODEL (bge-m3) to document_chunk.model/document_summary.embedding_model (below); additionally each backend is verified once, at its first use, that its /api/tags actually lists EMBED_MODEL. A backend serving the wrong model is a loud ERROR + 500 — never a silent distance computation across two different embedding spaces. Verification is lazy because SOLARIA may be asleep at boot and must not block startup.

/search returns 503 only when BOTH backends are unreachable (or EMBED_FALLBACK_URL is unset — then the pre-fallback behaviour applies); /healthz still answers, same tolerance pattern as llm-gateway.

Startup invariant (hard-fail)

At startup, kb-query queries document_chunk.model and document_summary.embedding_model for the set of models behind active embeddings, and refuses to start (crash-loop, visible via container restarts) if the configured EMBED_MODEL (default bge-m3) isn't in both sets. This guards against querying with an embedding space that doesn't match what's actually indexed — see app/startup.py for why the check reads document_summary.embedding_model and not .model (the latter is the LLM that wrote the summary, e.g. claude-haiku-4-5, not the embedder).

Configuration

.env — gitignored, copy from env.example. Required: LAN_BIND_IP, KB_DSN. On PIHA also set EMBED_FALLBACK_URL=http://192.168.31.5:11434 (ollama-piha). Optional (defaults in parentheses): EMBED_PRIMARY_URL (http://solaria:11434), EMBED_PRIMARY_NAME/EMBED_FALLBACK_NAME (solaria/piha), EMBED_HEALTH_TTL_S (30), EMBED_HEALTH_TIMEOUT_S (1.5), EMBED_PRIMARY_TIMEOUT_S (3), EMBED_MODEL (bge-m3), SUMMARY_MODEL (claude-haiku-4-5). OLLAMA_URL was renamed to EMBED_PRIMARY_URL in Krok 2 — if an old .env sets OLLAMA_URL, it is ignored.

Deploy (PIHA)

0. Prerequisite: ollama-piha deployed and bge-m3 pulled — see services/ollama-piha/README.md (the pull is a manual deploy step).
1. git pull on PIHA (~/homelab-codex-ws).
2. cp services/kb-query/env.example services/kb-query/.env and fill in the real KB_DSN password (the template already sets EMBED_FALLBACK_URL). On an existing install: add EMBED_FALLBACK_URL=http://192.168.31.5:11434 to the existing .env.

3. docker compose -f services/kb-query/docker-compose.yml \
     -f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build
   

4. Verify: services/kb-query/healthcheck.sh, then from PIHA: curl "http://192.168.31.5:8230/search?q=test" and open http://192.168.31.5:8230/ in a browser.

Fallback verification (execution: operator, after deploy)

• Test A — SOLARIA online: query via UI/curl; response has "embed_backend": "solaria", docker logs kb-query shows backend=solaria, latency ~sub-second.
• Test B — SOLARIA offline: either wait for the nightly power-off, or simulate: set EMBED_PRIMARY_URL=http://192.0.2.1:11434 (TEST-NET, always unreachable) in .env and docker compose … up -d again. Query still works; response has "embed_backend": "piha", log shows backend=piha plus a circuit open for 30s warning on the first hit; latency visibly higher (CPU + cold model load each time, OLLAMA_KEEP_ALIVE=0). Revert .env afterwards if simulated.
• Test C — SOLARIA returns: after it is back up, within ≤30 s (one health-cache TTL) responses show "embed_backend": "solaria" again, no restart needed.

Tests

pip install -e packages/kb-retrieval/
cd services/kb-query && pip install -r requirements.txt pytest pytest-asyncio && pytest

Unit tests mock the DB connection and Ollama HTTP session (no live DB/Ollama required) — same style as packages/kb-retrieval/tests/. tests/test_frontend.py drives GET ///static/* through FastAPI's TestClient without entering it as a context manager, so the DB-requiring lifespan never runs.

Frontend JS has its own pure-function tests (query-URL encoding, threshold colouring, envelope grouping), run without a browser via Node's built-in test runner: node --test services/kb-query/tests/frontend/.

Ingress (kb.kapala.org, plan §8)

Wired up 2026-07-23 (docs/sessions/2026-07-23-kb-f4-ingress.md), no code change in this service — pure infra step:

• npm@PIHA proxy host #35: kb.kapala.org → http://192.168.31.5:8230, cert #49 (*.kapala.org wildcard, DNS-01 via Cloudflare, expires 2026-09-28) — same pattern as paper./vikunja./ha.kapala.org.
• Pi-hole Local DNS (/etc/pihole/custom.list on PIHA, runtime, not in Git): kb.kapala.org → 192.168.31.5. This is the first kapala.org entry in that file — every other kapala.org vhost (paper/ha/immich/vikunja/forgejo) has no LAN override and resolves via the public Cloudflare record (Tailscale IP) even from LAN, a hairpin the plan assumed was already avoided for those too. Not fixed here (out of this task's scope — no other vhosts touched); worth a follow-up if it matters for those services.
• Cloudflare A record kb.kapala.org → 100.108.208.3 (Tailscale PIHA, DNS only): added manually by the operator (no CF API token available in the environment that did the rest of this step), verified against Cloudflare's own authoritative NS and 8.8.8.8/1.1.1.1 — resolves everywhere now, both LAN (via Pi-hole override) and Tailscale/public (via this record).

No auth. OIDC (plan §8: authlib, /login, /auth/callback) is explicitly not implemented — confirmed no forward-auth/reverse-proxy-level auth pattern exists anywhere in this repo (NPM community edition doesn't support it either); the three precedents (paperless/nextcloud/vikunja) all do OIDC inside the app. Building that is real service code (authlib dependency, session middleware, Forgejo OAuth2 app registration) — deliberately deferred to a separate session, decision confirmed with the operator 2026-07-23. Until then kb.kapala.org is reachable by anyone on the LAN/tailnet with no login, same as before this vhost existed.

Out of scope for this step

• OIDC login (see above) — separate session, needs authlib + Forgejo OAuth2 app.
• Calibration verdict for the fallback (plan §5 steps 4–5: live PIHA measurements → keep/tune/degrade decision) — the mechanism is built and default-on when EMBED_FALLBACK_URL is set; the measurements are the operator's post-deploy step (see services/ollama-piha/README.md).