From e7625cd322cbe66e87a6c6023743e32cbb49cb2f Mon Sep 17 00:00:00 2001 From: oskar Date: Wed, 29 Jul 2026 19:01:29 +0200 Subject: [PATCH] =?UTF-8?q?feat(kb):=20aktywny=20fallback=20embedding?= =?UTF-8?q?=C3=B3w=20SOLARIA=E2=86=92PIHA=20dla=20kb-query=20(faza=204=20K?= =?UTF-8?q?rok=202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ostatni krok fazy 4 KB (plan §2 Decyzja 2, §5): kb-query przestaje być martwe przez ~16 h/dobę, gdy SOLARIA (GPU) śpi — zapytania embeduje wtedy lokalna Ollama CPU na PIHA (wolniej: ~790 ms+ vs ~207 ms na GPU, ale działa). Nowy serwis services/ollama-piha (GitOps, owner_node: piha): - ollama/ollama:latest (arm64 natywnie), OLLAMA_KEEP_ALIVE=0 — model zwalnia RAM natychmiast po każdym wywołaniu (spike, nie rezydent; PIHA dzieli 8 GB z HA) - bind wyłącznie 127.0.0.1 + LAN_BIND_IP (192.168.31.5), nigdy 0.0.0.0/Tailscale - named volume ollama_piha_models (NVMe data-root) zamiast bind-mounta — obraz biega jako root w kontenerze i bind łamałby wzorzec uid PIHA (oskar=1004, kontenery uid 1000, setgid pi) - override hosts/piha/runtime/ollama-piha: mem_limit 2560m (wartość startowa z planu, do potwierdzenia kalibracją na żywo), świadomie bez mem_reservation - pull bge-m3 to jawny, ręczny krok deployu (README) — obraz nie ma modeli kb-query — maszyna stanów fallbacku (app/embed_router.py): - health-check SOLARII (GET /api/tags, timeout 1.5 s) z cache 30 s — zero sondowania per request; po powrocie SOLARII ruch wraca na GPU w ≤30 s - primary up → embed na SOLARII z twardym timeoutem 3 s; błąd W TRAKCIE zapytania = jednorazowe przełączenie (krok 3b planu): status down na 30 s i TO SAMO zapytanie leci na fallback — user nie widzi błędu SOLARII - primary down → embed prosto na ollama-piha (bez twardego timeoutu: CPU + zimny load modelu to legalnie pojedyncze sekundy) - 503 tylko gdy oba backendy padłe (lub fallback nieskonfigurowany) - inwariant modelu, druga połowa: każdy backend weryfikowany raz, leniwie przy pierwszym użyciu, że /api/tags zawiera EMBED_MODEL (bge-m3 — ta sama wartość co startowy check przeciw document_chunk.model/document_summary.embedding_model); niezgodność = ERROR log + 500, nigdy ciche liczenie dystansów między różnymi przestrzeniami embeddingów; leniwie, bo śpiąca SOLARIA nie może blokować startu serwisu - odpowiedź /search: nowe pole embed_backend ("solaria"|"piha") + sol_status wg realnego świata routera (UI już renderuje down jako "offline (fallback embed)"); log INFO backend=... elapsed_ms=... per zapytanie - /healthz: sol_status przez cache routera (spójny widok z routingiem) + fallback_status (żywa, tania sonda /api/tags) Konfiguracja spójnie przez env (compose + env.example + service.yaml + README): EMBED_PRIMARY_URL (zastępuje OLLAMA_URL), EMBED_FALLBACK_URL (pusty = brak fallbacku, zachowanie sprzed kroku 2), EMBED_{PRIMARY,FALLBACK}_NAME, EMBED_HEALTH_TTL_S/EMBED_HEALTH_TIMEOUT_S/EMBED_PRIMARY_TIMEOUT_S. Testy: 39 pass (14 nowych w test_embed_router.py: cache TTL, failover w trakcie zapytania, powrót po TTL, oba padłe, mismatch modelu na primary i fallbacku, tag "bge-m3:latest" vs "bge-m3"); docker build + smoke (importy + uvicorn do guardu KB_DSN) OK; compose config OK dla obu stacków. Deploy (Oskar, na PIHA z mastera po merge): cd ~/homelab-codex-ws && git pull # 1. ollama-piha cp services/ollama-piha/env.example services/ollama-piha/.env docker compose -f services/ollama-piha/docker-compose.yml \ -f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \ --env-file services/ollama-piha/.env up -d docker exec ollama-piha ollama pull bge-m3 # ręczny krok, obowiązkowy services/ollama-piha/healthcheck.sh # 2. kb-query (dopisać fallback do istniejącego .env) echo 'EMBED_FALLBACK_URL=http://192.168.31.5:11434' >> services/kb-query/.env docker compose -f services/kb-query/docker-compose.yml \ -f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build services/kb-query/healthcheck.sh # (deploy-node.sh też podniesie oba serwisy z hosts/piha/services.yaml, # ale pull bge-m3 i .env pozostają ręczne) Weryfikacja: testy A/B/C w services/kb-query/README.md (backend=solaria przy SOLARII online; backend=piha przy symulacji offline; powrót na GPU w ≤30 s). Co-Authored-By: Claude Fable 5 --- .../ollama-piha/docker-compose.override.yml | 18 ++ hosts/piha/services.yaml | 27 ++- services/kb-query/README.md | 77 +++++- services/kb-query/app/embed_router.py | 185 +++++++++++++++ services/kb-query/app/main.py | 70 ++++-- services/kb-query/app/search.py | 50 ++-- services/kb-query/docker-compose.yml | 12 +- services/kb-query/env.example | 22 +- services/kb-query/service.yaml | 20 +- services/kb-query/tests/test_embed_router.py | 224 ++++++++++++++++++ services/kb-query/tests/test_search.py | 72 +++--- services/ollama-piha/README.md | 76 ++++++ services/ollama-piha/docker-compose.yml | 33 +++ services/ollama-piha/env.example | 7 + services/ollama-piha/healthcheck.sh | 25 ++ services/ollama-piha/service.yaml | 26 ++ 16 files changed, 854 insertions(+), 90 deletions(-) create mode 100644 hosts/piha/runtime/ollama-piha/docker-compose.override.yml create mode 100644 services/kb-query/app/embed_router.py create mode 100644 services/kb-query/tests/test_embed_router.py create mode 100644 services/ollama-piha/README.md create mode 100644 services/ollama-piha/docker-compose.yml create mode 100644 services/ollama-piha/env.example create mode 100755 services/ollama-piha/healthcheck.sh create mode 100644 services/ollama-piha/service.yaml diff --git a/hosts/piha/runtime/ollama-piha/docker-compose.override.yml b/hosts/piha/runtime/ollama-piha/docker-compose.override.yml new file mode 100644 index 0000000..99d6b97 --- /dev/null +++ b/hosts/piha/runtime/ollama-piha/docker-compose.override.yml @@ -0,0 +1,18 @@ +# PIHA-specific overrides for ollama-piha (KB module 5, phase 4 — embed fallback). +# +# RESOURCE CONTEXT: PIHA is the RAM-bound 8 GB box shared with Home Assistant. +# bge-m3 is ~1.2 GB on disk, estimated 1.5–2 GB resident during an embed burst +# (plan §2 decision 2). OLLAMA_KEEP_ALIVE=0 (base compose) makes that a short +# per-call spike, not a resident cost — but the spike lands exactly when other +# SOLARIA-fallback consumers (e.g. paperless fallback OCR) may also be awake, +# so it gets a hard ceiling. +services: + ollama-piha: + # Plan §2 D2 starting value — the cgroup OOM killer restarts this container + # on breach instead of the host OOM killer picking a victim (which could be + # Home Assistant). Confirm or trim after live calibration (plan §5 step 4: + # a few embeds + docker stats at a normal-load hour). + mem_limit: 2560m + # Deliberately no mem_reservation: the working set is a transient spike and + # the idle daemon is ~100 MB — soft-reserving gigabytes would permanently + # take them from HA for a backend that mostly sleeps. diff --git a/hosts/piha/services.yaml b/hosts/piha/services.yaml index 825a589..b5851f6 100644 --- a/hosts/piha/services.yaml +++ b/hosts/piha/services.yaml @@ -105,14 +105,37 @@ services: # data is in Docker named volume kb_postgres_data — must land on the NVMe # (Docker data-root on /home); see hosts/piha/runtime/kb-postgres override. + ollama-piha: + role: embed-fallback # module 5 phase 4 (plan §2 D2/§5): local CPU bge-m3 for kb-query while SOLARIA sleeps + deployment_model: docker-compose + exposure: private # 127.0.0.1 + LAN bind (LAN_BIND_IP) only; no public/Tailscale exposure + offline_required: false + depends_on: + local: [] + external: [] + ports: + - name: http + container_port: 11434 + host_port: 11434 + protocol: tcp + runtime: + # .env (LAN_BIND_IP) lives alongside the compose file; models live in the + # Docker named volume ollama_piha_models (NVMe data-root, like kb-postgres). + # Deploy note: `docker exec ollama-piha ollama pull bge-m3` is a manual + # post-deploy step — the image ships no models. + config_path: services/ollama-piha + kb-query: role: kb-search-api # module 5 phase 4: FastAPI wrapper over kb_retrieval cascade/flat query deployment_model: docker-compose exposure: private # LAN bind (LAN_BIND_IP); npm@PIHA vhost + OIDC is a later step offline_required: false depends_on: - local: [kb-postgres] - external: [ollama] # SOLARIA may be offline -> /search 502/503, /healthz stays ok + # ollama-piha is a soft local dependency: the embed fallback while SOLARIA + # sleeps — kb-query starts and serves /healthz without it, /search degrades + # to 503 only when BOTH embed backends are unreachable. + local: [kb-postgres, ollama-piha] + external: [ollama] # primary embed backend @ SOLARIA; may be offline -> fallback to ollama-piha ports: - name: http container_port: 8080 diff --git a/services/kb-query/README.md b/services/kb-query/README.md index 0982597..9e6d02e 100644 --- a/services/kb-query/README.md +++ b/services/kb-query/README.md @@ -12,7 +12,7 @@ that's phase 5. |---|---|---| | `/` | GET | Search UI (Jinja2 shell + `/static/app.js`, no login yet — plan §8 OIDC is a later step) | | `/static/*` | GET | UI assets (`app.js`, `style.css`) | -| `/healthz` | GET | `{"status": "ok", "sol_status": "up"\|"down"}` — `sol_status` is a live probe of Ollama@SOLARIA, no auth required (monitoring must reach it) | +| `/healthz` | GET | `{"status": "ok", "sol_status": "up"\|"down", "fallback_status": "up"\|"down"\|"unconfigured"}` — `sol_status` comes from the embed router's ~30 s-cached SOLARIA probe (same world view `/search` routes by), `fallback_status` is a live cheap probe of ollama-piha; no auth required (monitoring must reach it) | | `/search?q=&mode=cascade\|flat` | GET | `query_text -> embed -> cascade_query/flat_query -> results`, `mode` defaults to `cascade` | `/search` response shape (module 5 phase 4 plan §4, `summary`/`summary_tags` @@ -21,7 +21,7 @@ change any field the plan §4 shape already defined): ```json { - "query": "...", "mode": "cascade", "sol_status": "up", + "query": "...", "mode": "cascade", "sol_status": "up", "embed_backend": "solaria", "results": [ {"envelope_id": "paperless:119", "source": "paperless", "dist": 0.34, "chunk_index": 2, "text": "...", "link": "https://paper.kapala.org/documents/119/details", @@ -64,13 +64,39 @@ container, no node build step (plan §2 decision 4): `app/templates/index.html` - Stopka pokazuje `sol_status` dyskretnie (odświeżane z `/healthz` przy starcie strony i po każdym wyszukiwaniu). -## Embed path (current step) +## Embed path — active SOLARIA→PIHA fallback (Krok 2, plan §2 D2/§5) -Calls Ollama on SOLARIA directly per request — no cache, no circuit breaker, -no local-PIHA fallback yet (that state machine, plan §2 decision 2/§5, is a -separate later step). If SOLARIA is unreachable, `/search` returns **503**; -`/healthz` still answers (`sol_status: "down"`), same tolerance pattern as -`llm-gateway`. +Query embeddings go through `app/embed_router.py` (`EmbedRouter`, one instance +per process): + +1. SOLARIA's health verdict (`GET /api/tags`, 1.5 s timeout) is **cached for + 30 s** — no per-request probing. +2. Verdict `up` → embed on `EMBED_PRIMARY_URL` (SOLARIA GPU, ~207 ms) under a + hard 3 s timeout. +3. A failure **during a real embed** flips the verdict to `down` for one TTL + window and the **same request** is served from `EMBED_FALLBACK_URL` + (`ollama-piha`, CPU, `OLLAMA_KEEP_ALIVE=0` — slower, single seconds, but + alive) — the user never sees a SOLARIA error while a fallback exists. +4. Verdict `down` → straight to the fallback until the TTL expires; when + SOLARIA wakes up, traffic returns to the GPU within ≤30 s. + +Every `/search` response and log line says which backend embedded the query +(`embed_backend: "solaria"|"piha"`, log `backend=…` in `kb-query.embed`) — +needed to debug result quality per backend. `sol_status` in the response is +the router's world view; the UI footer renders `down` as +„SOLARIA: offline (fallback embed)". + +**Model invariant, both halves:** at startup kb-query pins `EMBED_MODEL` +(bge-m3) to `document_chunk.model`/`document_summary.embedding_model` (below); +additionally each backend is verified once, at its first use, that its +`/api/tags` actually lists `EMBED_MODEL`. A backend serving the wrong model is +a **loud ERROR + 500** — never a silent distance computation across two +different embedding spaces. Verification is lazy because SOLARIA may be asleep +at boot and must not block startup. + +`/search` returns **503** only when BOTH backends are unreachable (or +`EMBED_FALLBACK_URL` is unset — then the pre-fallback behaviour applies); +`/healthz` still answers, same tolerance pattern as `llm-gateway`. ## Startup invariant (hard-fail) @@ -86,13 +112,23 @@ that *wrote* the summary, e.g. `claude-haiku-4-5`, not the embedder). ## Configuration `.env` — **gitignored**, copy from `env.example`. Required: `LAN_BIND_IP`, -`KB_DSN`. Optional: `OLLAMA_URL`, `EMBED_MODEL`, `SUMMARY_MODEL`. +`KB_DSN`. On PIHA also set `EMBED_FALLBACK_URL=http://192.168.31.5:11434` +(ollama-piha). Optional (defaults in parentheses): `EMBED_PRIMARY_URL` +(`http://solaria:11434`), `EMBED_PRIMARY_NAME`/`EMBED_FALLBACK_NAME` +(`solaria`/`piha`), `EMBED_HEALTH_TTL_S` (30), `EMBED_HEALTH_TIMEOUT_S` (1.5), +`EMBED_PRIMARY_TIMEOUT_S` (3), `EMBED_MODEL` (`bge-m3`), `SUMMARY_MODEL` +(`claude-haiku-4-5`). `OLLAMA_URL` was **renamed** to `EMBED_PRIMARY_URL` in +Krok 2 — if an old `.env` sets `OLLAMA_URL`, it is ignored. ## Deploy (PIHA) +0. Prerequisite: `ollama-piha` deployed and `bge-m3` pulled — see + `services/ollama-piha/README.md` (the pull is a **manual** deploy step). 1. `git pull` on PIHA (`~/homelab-codex-ws`). 2. `cp services/kb-query/env.example services/kb-query/.env` and fill in the - real `KB_DSN` password. + real `KB_DSN` password (the template already sets `EMBED_FALLBACK_URL`). + On an existing install: add `EMBED_FALLBACK_URL=http://192.168.31.5:11434` + to the existing `.env`. 3. ``` docker compose -f services/kb-query/docker-compose.yml \ -f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build @@ -101,6 +137,22 @@ that *wrote* the summary, e.g. `claude-haiku-4-5`, not the embedder). `curl "http://192.168.31.5:8230/search?q=test"` and open `http://192.168.31.5:8230/` in a browser. +## Fallback verification (execution: operator, after deploy) + +- **Test A — SOLARIA online**: query via UI/`curl`; response has + `"embed_backend": "solaria"`, `docker logs kb-query` shows + `backend=solaria`, latency ~sub-second. +- **Test B — SOLARIA offline**: either wait for the nightly power-off, or + simulate: set `EMBED_PRIMARY_URL=http://192.0.2.1:11434` (TEST-NET, always + unreachable) in `.env` and `docker compose … up -d` again. Query still + works; response has `"embed_backend": "piha"`, log shows `backend=piha` + plus a `circuit open for 30s` warning on the first hit; latency visibly + higher (CPU + cold model load each time, `OLLAMA_KEEP_ALIVE=0`). Revert + `.env` afterwards if simulated. +- **Test C — SOLARIA returns**: after it is back up, within ≤30 s (one + health-cache TTL) responses show `"embed_backend": "solaria"` again, no + restart needed. + ## Tests ``` @@ -150,5 +202,8 @@ existed. ## Out of scope for this step -- Local-PIHA embed fallback / circuit breaker (plan §2 decision 2, §5). - OIDC login (see above) — separate session, needs `authlib` + Forgejo OAuth2 app. +- Calibration verdict for the fallback (plan §5 steps 4–5: live PIHA + measurements → keep/tune/degrade decision) — the mechanism is built and + default-on when `EMBED_FALLBACK_URL` is set; the measurements are the + operator's post-deploy step (see `services/ollama-piha/README.md`). diff --git a/services/kb-query/app/embed_router.py b/services/kb-query/app/embed_router.py new file mode 100644 index 0000000..bbf7cf8 --- /dev/null +++ b/services/kb-query/app/embed_router.py @@ -0,0 +1,185 @@ +"""Embed-backend router -- module 5 phase 4, Krok 2 (docs/kb/modules/05-faza4-plan.md §2 +Decyzja 2, §5): the active SOLARIA->PIHA fallback state machine `/search` embeds through. +One instance per process (`app.state.embed_router`), holding the single global `sol_status` +cache the plan describes: + +| state | behaviour | +|-----------------|----------------------------------------------------------------------| +| cache valid | use remembered up/down, no probe | +| cache expired | GET {primary}/api/tags, short timeout -> up/down, cached for the TTL | +| up -> embed | POST {primary}/api/embeddings under a hard timeout | +| embed fails | one-shot flip (plan step 3b): mark down for one TTL window and serve | +| | THE SAME query from the fallback -- the user never sees a SOLARIA | +| | error unless there is no fallback configured | +| down -> embed | straight to the fallback, no primary attempt until the TTL expires | + +Both legs call `kb_retrieval.embed.embed_chunk(..., model=embed_model)` with the same +configured model (the plan's "twardy inwariant"). On top of the DB-side startup check +(app/startup.py pins EMBED_MODEL to `document_chunk.model`/`document_summary.embedding_model`), +each backend is additionally verified ONCE per process lifetime, lazily at its first use: +its `/api/tags` must list `embed_model`. A backend that answers but lacks bge-m3 raises +`ModelMismatchError` (ERROR log, 500 upstream) -- never a silent embed in a vector space +that doesn't match the index. Verification is lazy rather than at startup because either +backend may legitimately be offline when kb-query boots (SOLARIA sleeps ~16 h/day, plan +§1) and must not block the service. + +Timeouts: probe 1.5 s (task spec "krótki, np. 1-2 s"; the plan's 500 ms proposal was for +LAN-adjacent Tailscale -- 1.5 s absorbs LTE-grade jitter without hurting the 30 s cache +economics), primary embed 3 s hard (plan step 3). The fallback embed gets NO hard timeout: +Pi-5 CPU embed plus a cold model load (OLLAMA_KEEP_ALIVE=0 on ollama-piha) is legitimately +slow (single seconds to low tens) and it is the last resort -- cutting it off would turn +"slow but alive" back into "dead". +""" +from __future__ import annotations + +import asyncio +import logging +import time +from dataclasses import dataclass, field + +import aiohttp + +from kb_retrieval.embed import check_ollama_health, embed_chunk + +logger = logging.getLogger("kb-query.embed") + + +class EmbedBackendError(RuntimeError): + """No configured backend could embed the query (primary down and the fallback is + unconfigured or unreachable). Maps to 503 in the HTTP layer.""" + + +class ModelMismatchError(RuntimeError): + """A backend is reachable but its /api/tags does not list the configured embed model. + Config/ops error (wrong URL, model not pulled) -- deliberately loud, maps to 500.""" + + +@dataclass +class _Backend: + name: str + url: str + model_verified: bool = field(default=False) + + +class EmbedRouter: + def __init__( + self, + primary_url: str, + fallback_url: str | None = None, + *, + embed_model: str, + primary_name: str = "solaria", + fallback_name: str = "piha", + health_ttl_s: float = 30.0, + health_timeout_s: float = 1.5, + primary_embed_timeout_s: float = 3.0, + clock=time.monotonic, + ): + self.primary = _Backend(primary_name, primary_url.rstrip("/")) + self.fallback = _Backend(fallback_name, fallback_url.rstrip("/")) if fallback_url else None + self.embed_model = embed_model + self.health_ttl_s = health_ttl_s + self.health_timeout_s = health_timeout_s + self.primary_embed_timeout_s = primary_embed_timeout_s + self._clock = clock + self._sol_status: str | None = None # "up" | "down" | None (never probed yet) + self._checked_at: float = 0.0 + + def _cache_valid(self) -> bool: + return self._sol_status is not None and (self._clock() - self._checked_at) < self.health_ttl_s + + def _set_status(self, status: str) -> None: + self._sol_status = status + self._checked_at = self._clock() + + async def primary_status(self, session: aiohttp.ClientSession) -> str: + """Cached primary liveness ("up"/"down"), probing /api/tags only when the TTL expired. + Shared by /search routing and /healthz so both report the same world view.""" + if not self._cache_valid(): + up = await check_ollama_health(session, self.primary.url, self.health_timeout_s) + self._set_status("up" if up else "down") + return self._sol_status + + async def fallback_status(self, session: aiohttp.ClientSession) -> str: + """Live fallback liveness for /healthz: "up"/"down"/"unconfigured". Uncached -- the + probe is a cheap /api/tags (does not load the model on ollama-piha) and monitoring + calls this every 30 s anyway.""" + if self.fallback is None: + return "unconfigured" + up = await check_ollama_health(session, self.fallback.url, self.health_timeout_s) + return "up" if up else "down" + + async def _verify_model(self, session: aiohttp.ClientSession, backend: _Backend) -> None: + """First-use guard: `/api/tags` of `backend` must list `embed_model` (exact name or + same base, so a pulled "bge-m3:latest" satisfies EMBED_MODEL=bge-m3). Raises + ModelMismatchError when reachable-but-missing; connection errors propagate to the + caller's failover handling. Passed once -> never re-checked for this process.""" + if backend.model_verified: + return + async with session.get( + f"{backend.url}/api/tags", + timeout=aiohttp.ClientTimeout(total=self.health_timeout_s), + ) as resp: + resp.raise_for_status() + data = await resp.json() + names = {m.get("name", "") for m in data.get("models", [])} + base = self.embed_model.split(":")[0] + if not any(n == self.embed_model or n.split(":")[0] == base for n in names): + raise ModelMismatchError( + f"embed backend {backend.name} ({backend.url}) does not serve " + f"model {self.embed_model!r} (available: {sorted(names) or 'none'}) -- " + f"refusing to embed in a vector space that doesn't match document_chunk" + ) + backend.model_verified = True + + async def embed(self, session: aiohttp.ClientSession, text: str) -> tuple[list[float], str]: + """Embed one query text, returning (embedding, backend_name). Implements the state + machine from the module docstring; raises EmbedBackendError (503) when no backend + can serve, ModelMismatchError (500) when the fallback serves the wrong model.""" + if await self.primary_status(session) == "up": + try: + await self._verify_model(session, self.primary) + embedding, elapsed = await asyncio.wait_for( + embed_chunk(session, self.primary.url, self.embed_model, text), + timeout=self.primary_embed_timeout_s, + ) + logger.info( + "query embedded backend=%s url=%s elapsed_ms=%.0f", + self.primary.name, self.primary.url, elapsed * 1000, + ) + return embedding, self.primary.name + except ModelMismatchError as exc: + # Reachable but serving the wrong model set -- loud, then try the fallback + # (which is verified independently below, so this cannot cascade silently). + logger.error("primary embed backend rejected: %s", exc) + self._set_status("down") + except (aiohttp.ClientError, TimeoutError, ValueError) as exc: + # Plan step 3b: one-shot flip -- the SAME query continues to the fallback. + logger.warning( + "primary embed failed backend=%s url=%s (%s: %s); circuit open for %.0fs", + self.primary.name, self.primary.url, type(exc).__name__, exc, self.health_ttl_s, + ) + self._set_status("down") + + if self.fallback is None: + raise EmbedBackendError( + f"primary embed backend {self.primary.name} ({self.primary.url}) is down " + f"and no EMBED_FALLBACK_URL is configured" + ) + + try: + await self._verify_model(session, self.fallback) + except ModelMismatchError: + raise # loud config error -- never silently embed in a mismatched space + except (aiohttp.ClientError, TimeoutError) as exc: + raise EmbedBackendError( + f"primary embed backend {self.primary.name} is down and fallback " + f"{self.fallback.name} ({self.fallback.url}) is unreachable: {exc}" + ) from exc + + embedding, elapsed = await embed_chunk(session, self.fallback.url, self.embed_model, text) + logger.info( + "query embedded backend=%s url=%s elapsed_ms=%.0f (fallback; primary=%s down)", + self.fallback.name, self.fallback.url, elapsed * 1000, self.primary.name, + ) + return embedding, self.fallback.name diff --git a/services/kb-query/app/main.py b/services/kb-query/app/main.py index 1bf7d3d..812ec94 100644 --- a/services/kb-query/app/main.py +++ b/services/kb-query/app/main.py @@ -1,12 +1,15 @@ """kb-query -- module 5 phase 4 (docs/kb/modules/05-faza4-plan.md §4): first user-facing HTTP -entry point to the KB. Wraps `kb_retrieval.cascade_query`/`flat_query` (module 5 phase 3, -already gated PASS -- docs/sessions/2026-07-21.md) in FastAPI. This is a search API, not chat: -no answer synthesis, no LLM call over the results (that is phase 5, out of scope here). +entry point to the KB. Wraps `kb_retrieval` retrieval (module 5 phase 3, gated PASS -- +docs/sessions/2026-07-21.md) in FastAPI. This is a search API, not chat: no answer synthesis, +no LLM call over the results (that is phase 5, out of scope here). -Embed path is deliberately simple for this step: calls Ollama on SOLARIA directly, no -cache/circuit-breaker/local-PIHA-fallback (plan §2 decision 2, §5) -- that state machine is a -later, separate step. A failed embed call (SOLARIA unreachable) surfaces as 503 to the caller -rather than a bare 500. +Embed path (Krok 2, plan §2 decision 2 / §5 -- the active fallback): queries embed through +`app.embed_router.EmbedRouter` -- SOLARIA's GPU Ollama (EMBED_PRIMARY_URL) when its cached +~30 s health-check says up, the local CPU `ollama-piha` (EMBED_FALLBACK_URL) when SOLARIA +sleeps. A mid-query primary failure fails over the SAME request instead of surfacing an +error. Only when BOTH backends are unavailable does `/search` return 503; a backend serving +the wrong model (vs the bge-m3 space `document_chunk` is indexed in) is a loud 500, never a +silent cross-space distance computation. `GET /` (Krok 4, plan §7) serves the search UI from this same FastAPI process -- one image, one container (plan §2 decision 4): a Jinja2 shell + a static vanilla-JS file, no node build step. @@ -18,6 +21,7 @@ available explicitly starting here, but the default stays `cascade` until the qu """ from __future__ import annotations +import logging import os import pathlib from contextlib import asynccontextmanager @@ -28,18 +32,30 @@ from fastapi.responses import HTMLResponse from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates -from kb_retrieval.embed import check_ollama_health - from app.db import create_pool +from app.embed_router import EmbedBackendError, EmbedRouter, ModelMismatchError from app.search import run_search from app.startup import validate_embed_model +# uvicorn only configures its own loggers; without this the router's +# backend=solaria/piha lines (kb-query.embed) would never reach docker logs. +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s") + BASE_DIR = pathlib.Path(__file__).resolve().parent KB_DSN = os.environ.get("KB_DSN") -OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://solaria:11434") +# EMBED_PRIMARY_URL replaced OLLAMA_URL in Krok 2 (aktywny fallback) -- one naming scheme +# for both legs. Same default upstream as before: Ollama @ SOLARIA over Tailscale. +EMBED_PRIMARY_URL = os.environ.get("EMBED_PRIMARY_URL", "http://solaria:11434") +# Unset/empty -> no fallback leg: /search 503s when SOLARIA is down (pre-Krok-2 behaviour). +EMBED_FALLBACK_URL = os.environ.get("EMBED_FALLBACK_URL") or None +EMBED_PRIMARY_NAME = os.environ.get("EMBED_PRIMARY_NAME", "solaria") +EMBED_FALLBACK_NAME = os.environ.get("EMBED_FALLBACK_NAME", "piha") EMBED_MODEL = os.environ.get("EMBED_MODEL", "bge-m3") SUMMARY_MODEL = os.environ.get("SUMMARY_MODEL", "claude-haiku-4-5") -OLLAMA_HEALTH_TIMEOUT_S = 3.0 +# State-machine parameters (plan §2 D2 table; timeouts rationale in app/embed_router.py). +EMBED_HEALTH_TTL_S = float(os.environ.get("EMBED_HEALTH_TTL_S", "30")) +EMBED_HEALTH_TIMEOUT_S = float(os.environ.get("EMBED_HEALTH_TIMEOUT_S", "1.5")) +EMBED_PRIMARY_TIMEOUT_S = float(os.environ.get("EMBED_PRIMARY_TIMEOUT_S", "3")) @asynccontextmanager @@ -50,11 +66,23 @@ async def lifespan(app: FastAPI): pool = await create_pool(KB_DSN) async with pool.acquire() as conn: # Hard invariant (plan §2 decision 2): refuse to start rather than silently serve - # queries against a mismatched embedding space. + # queries against a mismatched embedding space. The per-backend half of the same + # invariant (does the Ollama actually serve EMBED_MODEL?) lives in EmbedRouter and + # runs lazily at each backend's first use -- a sleeping SOLARIA must not block boot. await validate_embed_model(conn, EMBED_MODEL) app.state.pool = pool app.state.http = aiohttp.ClientSession() + app.state.embed_router = EmbedRouter( + EMBED_PRIMARY_URL, + EMBED_FALLBACK_URL, + embed_model=EMBED_MODEL, + primary_name=EMBED_PRIMARY_NAME, + fallback_name=EMBED_FALLBACK_NAME, + health_ttl_s=EMBED_HEALTH_TTL_S, + health_timeout_s=EMBED_HEALTH_TIMEOUT_S, + primary_embed_timeout_s=EMBED_PRIMARY_TIMEOUT_S, + ) try: yield finally: @@ -74,8 +102,15 @@ async def index(request: Request): @app.get("/healthz") async def healthz() -> dict: - sol_up = await check_ollama_health(app.state.http, OLLAMA_URL, OLLAMA_HEALTH_TIMEOUT_S) - return {"status": "ok", "sol_status": "up" if sol_up else "down"} + # sol_status goes through the router's 30 s cache -- /healthz and /search routing + # deliberately share one world view (and monitoring no longer re-probes a sleeping + # SOLARIA on every scrape). fallback_status is a live cheap /api/tags probe. + router = app.state.embed_router + return { + "status": "ok", + "sol_status": await router.primary_status(app.state.http), + "fallback_status": await router.fallback_status(app.state.http), + } @app.get("/search") @@ -86,7 +121,10 @@ async def search( try: async with app.state.pool.acquire() as conn: return await run_search( - conn, app.state.http, OLLAMA_URL, q, mode, EMBED_MODEL, SUMMARY_MODEL + conn, app.state.http, app.state.embed_router, q, mode, SUMMARY_MODEL ) - except aiohttp.ClientError as exc: + except ModelMismatchError as exc: + # Config/ops error (backend up but wrong model set) -- 500, deliberately loud. + raise HTTPException(status_code=500, detail=str(exc)) from exc + except (EmbedBackendError, aiohttp.ClientError) as exc: raise HTTPException(status_code=503, detail=f"embed backend unavailable: {exc}") from exc diff --git a/services/kb-query/app/search.py b/services/kb-query/app/search.py index be1fee6..3ef8748 100644 --- a/services/kb-query/app/search.py +++ b/services/kb-query/app/search.py @@ -1,6 +1,6 @@ """`/search` core -- module 5 phase 4 (docs/kb/modules/05-faza4-plan.md §4). Kept decoupled -from FastAPI so it can be unit-tested with fake `conn`/`session` objects, the same style as -`kb_retrieval`'s own tests, instead of needing a live DB/Ollama behind a TestClient. +from FastAPI so it can be unit-tested with fake `conn`/`session`/`router` objects, the same +style as `kb_retrieval`'s own tests, instead of needing a live DB/Ollama behind a TestClient. Response shape (plan §4 exactly): `{"query", "mode", "sol_status", "results": [...]}`, each result carrying `dist` un-filtered -- the 0.45/0.55 colour thresholds (plan §7) are a frontend @@ -8,43 +8,48 @@ concern (Krok 4, out of this step's scope), never applied server-side. `summary`/`summary_tags` (document_summary, haiku track) are an additive field added in Krok 4 for the frontend's per-envelope result header (plan §7) -- `None`/`[]` when the envelope has no -summary for `summary_model` yet. Purely additive: does not change any field already covered by -the phase-4 gate's HTTP-equivalence check (plan §9). +summary for `summary_model` yet. + +Krok 2 (aktywny fallback, plan §5): the query embedding no longer happens inside +`kb_retrieval.cascade_query`/`flat_query`/`hybrid_query` -- it goes through +`app.embed_router.EmbedRouter` (SOLARIA primary -> PIHA fallback state machine) and the +resulting vector feeds the same `*_retrieve` functions those wrappers call. Two additive +consequences for the response: `sol_status` now reports the router's real world view ("down" +while serving from the fallback -- the frontend already renders that as "offline (fallback +embed)"), and `embed_backend` names which backend actually embedded THIS query (task spec: +needed to debug result quality per backend). """ from __future__ import annotations import aiohttp import asyncpg -from kb_retrieval.retrieval import cascade_query, flat_query, hybrid_query +from kb_retrieval.embed import _vector_literal # same private-import convention as kb_retrieval.retrieval +from kb_retrieval.retrieval import cascade_retrieve, flat_retrieve, hybrid_retrieve from app.db import fetch_envelopes, fetch_summaries +from app.embed_router import EmbedRouter from app.links import build_result async def run_search( conn: asyncpg.Connection, session: aiohttp.ClientSession, - ollama_url: str, + router: EmbedRouter, query_text: str, mode: str, - embed_model: str, summary_model: str, ) -> dict: - if mode == "flat": - retrieval = await flat_query(conn, session, ollama_url, query_text, embed_model=embed_model) - elif mode == "hybrid": - retrieval = await hybrid_query( - conn, session, ollama_url, query_text, - summary_model=summary_model, embed_model=embed_model, - ) - else: - retrieval = await cascade_query( - conn, session, ollama_url, query_text, - summary_model=summary_model, embed_model=embed_model, - ) + embedding, backend = await router.embed(session, query_text) + query_vector = _vector_literal(embedding) + + if mode == "flat": + chunks = await flat_retrieve(conn, query_vector) + elif mode == "hybrid": + chunks = (await hybrid_retrieve(conn, query_vector, summary_model))["chunks"] + else: + chunks = (await cascade_retrieve(conn, query_vector, summary_model))["chunks"] - chunks = retrieval["chunks"] envelope_ids = sorted({c["envelope_id"] for c in chunks}) envelopes = await fetch_envelopes(conn, envelope_ids) summaries = await fetch_summaries(conn, envelope_ids, summary_model) @@ -60,6 +65,9 @@ async def run_search( return { "query": query_text, "mode": mode, - "sol_status": "up", # reaching this point means the embed call above succeeded + # "up" iff the primary (SOLARIA) embedded this very query -- the fallback path + # implies the router just observed the primary down (probe or mid-embed failure). + "sol_status": "up" if backend == router.primary.name else "down", + "embed_backend": backend, "results": results, } diff --git a/services/kb-query/docker-compose.yml b/services/kb-query/docker-compose.yml index dc524dd..f1fdcb4 100644 --- a/services/kb-query/docker-compose.yml +++ b/services/kb-query/docker-compose.yml @@ -17,7 +17,17 @@ services: - "${LAN_BIND_IP}:8230:8080" environment: - KB_DSN=${KB_DSN} - - OLLAMA_URL=${OLLAMA_URL:-http://solaria:11434} + # Embed backends (Krok 2, aktywny fallback — app/embed_router.py): primary is + # Ollama @ SOLARIA (GPU, duty-cycled), fallback is ollama-piha on this host + # (CPU, OLLAMA_KEEP_ALIVE=0). Empty EMBED_FALLBACK_URL disables the fallback + # leg (pre-Krok-2 behaviour: /search 503s while SOLARIA sleeps). + - EMBED_PRIMARY_URL=${EMBED_PRIMARY_URL:-http://solaria:11434} + - EMBED_FALLBACK_URL=${EMBED_FALLBACK_URL:-} + - EMBED_PRIMARY_NAME=${EMBED_PRIMARY_NAME:-solaria} + - EMBED_FALLBACK_NAME=${EMBED_FALLBACK_NAME:-piha} + - EMBED_HEALTH_TTL_S=${EMBED_HEALTH_TTL_S:-30} + - EMBED_HEALTH_TIMEOUT_S=${EMBED_HEALTH_TIMEOUT_S:-1.5} + - EMBED_PRIMARY_TIMEOUT_S=${EMBED_PRIMARY_TIMEOUT_S:-3} - EMBED_MODEL=${EMBED_MODEL:-bge-m3} - SUMMARY_MODEL=${SUMMARY_MODEL:-claude-haiku-4-5} # python:3.12-slim has no curl/wget; same in-container check pattern as llm-gateway. diff --git a/services/kb-query/env.example b/services/kb-query/env.example index 4b23f5d..4ae069b 100644 --- a/services/kb-query/env.example +++ b/services/kb-query/env.example @@ -11,9 +11,25 @@ LAN_BIND_IP=192.168.31.5 # reasoning as paperless-worker@SOLARIA reaching paperless@PIHA over LAN. KB_DSN=postgresql://kb:CHANGE-ME@192.168.31.5:5433/kb -# Ollama upstream (SOLARIA, over Tailscale MagicDNS — same trick as -# llm-gateway's OLLAMA_URL). Optional: defaults to this value if unset. -# OLLAMA_URL=http://solaria:11434 +# Primary embed backend: Ollama @ SOLARIA (GPU, over Tailscale MagicDNS — +# same trick as llm-gateway's OLLAMA_URL). Optional: defaults to this value. +# EMBED_PRIMARY_URL=http://solaria:11434 + +# Fallback embed backend: ollama-piha on this host (services/ollama-piha/), +# reached over PIHA's LAN interface (same pattern as KB_DSN above). SOLARIA is +# powered off ~16 h/day — without this, /search is dead the whole time. Leave +# unset/empty to disable the fallback leg entirely. +EMBED_FALLBACK_URL=http://192.168.31.5:11434 + +# Backend labels used in logs and the /search "embed_backend" response field. +# Optional: defaults solaria / piha. +# EMBED_PRIMARY_NAME=solaria +# EMBED_FALLBACK_NAME=piha + +# Fallback state machine tuning (plan §2 D2). Optional; defaults shown. +# EMBED_HEALTH_TTL_S=30 # how long an up/down verdict on SOLARIA is cached +# EMBED_HEALTH_TIMEOUT_S=1.5 # /api/tags probe timeout +# EMBED_PRIMARY_TIMEOUT_S=3 # hard timeout for a primary embed call # Embedding model kb-query enforces as a startup invariant (plan §2 decision # 2) against document_chunk.model / document_summary.embedding_model. diff --git a/services/kb-query/service.yaml b/services/kb-query/service.yaml index 37c7bb4..701b66e 100644 --- a/services/kb-query/service.yaml +++ b/services/kb-query/service.yaml @@ -5,10 +5,12 @@ service: exposure: private # LAN/Tailscale only, npm@PIHA vhost (kb.kapala.org) is a later step dependencies: - kb-postgres + - ollama-piha # soft: local CPU embed fallback (Krok 2) -- kb-query starts without it - forgejo # OIDC identity provider, wired in a later step (plan §8); not yet enforced - # ollama@SOLARIA is an external, optional runtime dependency (embed calls), not a hard - # dependency here -- SOLARIA may be offline; /search then fails per-request with 503, - # /healthz still answers (same tolerance pattern as llm-gateway). + # ollama@SOLARIA is an external, optional runtime dependency (primary embed backend), not + # a hard dependency here -- SOLARIA sleeps ~16 h/day; the embed router (app/embed_router.py) + # then falls back to ollama-piha on this host. /search 503s only when BOTH legs are down; + # /healthz always answers (same tolerance pattern as llm-gateway). ports: - container: 8080 host: 8230 # LAN_BIND_IP only, never 0.0.0.0 @@ -28,6 +30,12 @@ service: env_vars: - LAN_BIND_IP # required — compose port-bind interpolation - KB_DSN # required — asyncpg DSN for kb-postgres@PIHA - - OLLAMA_URL # optional — defaults to http://solaria:11434 - - EMBED_MODEL # optional — defaults to bge-m3; startup invariant vs document_chunk/document_summary - - SUMMARY_MODEL # optional — defaults to claude-haiku-4-5; cascade_query's stage-1 model + - EMBED_PRIMARY_URL # optional — primary embed backend, defaults to http://solaria:11434 + - EMBED_FALLBACK_URL # optional — local fallback (ollama-piha); empty disables the fallback leg + - EMBED_PRIMARY_NAME # optional — log/response label, defaults to solaria + - EMBED_FALLBACK_NAME # optional — log/response label, defaults to piha + - EMBED_HEALTH_TTL_S # optional — primary health verdict cache, defaults to 30 + - EMBED_HEALTH_TIMEOUT_S # optional — /api/tags probe timeout, defaults to 1.5 + - EMBED_PRIMARY_TIMEOUT_S # optional — hard timeout for a primary embed call, defaults to 3 + - EMBED_MODEL # optional — defaults to bge-m3; startup invariant vs document_chunk/document_summary + per-backend /api/tags check + - SUMMARY_MODEL # optional — defaults to claude-haiku-4-5; cascade stage-1 model diff --git a/services/kb-query/tests/test_embed_router.py b/services/kb-query/tests/test_embed_router.py new file mode 100644 index 0000000..5430913 --- /dev/null +++ b/services/kb-query/tests/test_embed_router.py @@ -0,0 +1,224 @@ +"""Unit tests for the SOLARIA->PIHA embed fallback state machine (app.embed_router) -- no +real Ollama on either side. Fake backends are keyed by URL so one fake session can serve +both legs; the clock is injected so the 30 s TTL is tested without sleeping.""" +from __future__ import annotations + +import pathlib +import sys + +import aiohttp +import pytest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) + +from app.embed_router import EmbedBackendError, EmbedRouter, ModelMismatchError # noqa: E402 + +PRIMARY = "http://solaria:11434" +FALLBACK = "http://192.168.31.5:11434" + + +class _FakeResponse: + def __init__(self, payload=None, exc=None, status=200): + self._payload = payload + self._exc = exc + self.status = status # check_ollama_health reads resp.status directly + + async def __aenter__(self): + if self._exc is not None: + raise self._exc + return self + + async def __aexit__(self, *exc): + return False + + def raise_for_status(self): + pass + + async def json(self): + return self._payload + + +class _FakeBackend: + """One fake Ollama: `models` drives /api/tags, `down=True` refuses every connection, + `embed_exc` makes /api/embeddings fail while /api/tags still answers (the plan's step-3b + scenario: probe says up, the real embed then dies mid-query).""" + + def __init__(self, models=("bge-m3:latest",), down=False, embed_exc=None, vector_value=0.01): + self.models = list(models) + self.down = down + self.embed_exc = embed_exc + self.vector_value = vector_value + self.tags_calls = 0 + self.embed_calls = 0 + + +class _FakeSession: + def __init__(self, backends: dict): + self._backends = backends + + def _backend(self, url): + for base, backend in self._backends.items(): + if url.startswith(base): + return backend + raise AssertionError(f"unexpected url: {url}") + + def get(self, url, timeout=None): + backend = self._backend(url) + backend.tags_calls += 1 + if backend.down: + return _FakeResponse(exc=aiohttp.ClientConnectionError("connection refused")) + return _FakeResponse({"models": [{"name": n} for n in backend.models]}) + + def post(self, url, json): + backend = self._backend(url) + backend.embed_calls += 1 + if backend.down: + return _FakeResponse(exc=aiohttp.ClientConnectionError("connection refused")) + if backend.embed_exc is not None: + return _FakeResponse(exc=backend.embed_exc) + return _FakeResponse({"embedding": [backend.vector_value] * 1024}) + + +class _FakeClock: + def __init__(self): + self.now = 1000.0 + + def __call__(self): + return self.now + + +def _router(fallback_url=FALLBACK, clock=None, **kwargs): + return EmbedRouter( + PRIMARY, + fallback_url, + embed_model="bge-m3", + clock=clock or _FakeClock(), + **kwargs, + ) + + +class TestPrimaryHealthy: + async def test_embeds_on_primary_and_reports_its_name(self): + solaria = _FakeBackend() + piha = _FakeBackend() + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + embedding, backend = await _router().embed(session, "q") + assert backend == "solaria" + assert len(embedding) == 1024 + assert solaria.embed_calls == 1 + assert piha.embed_calls == 0 + + async def test_health_verdict_is_cached_within_ttl(self): + solaria = _FakeBackend() + clock = _FakeClock() + router = _router(clock=clock) + session = _FakeSession({PRIMARY: solaria, FALLBACK: _FakeBackend()}) + await router.embed(session, "q1") + probes_after_first = solaria.tags_calls # probe + first-use model verification + clock.now += 10 # inside the 30 s TTL + await router.embed(session, "q2") + assert solaria.tags_calls == probes_after_first # no new probe, model check done once + + async def test_model_tag_with_latest_suffix_satisfies_bare_model_name(self): + solaria = _FakeBackend(models=["bge-m3:latest"]) + session = _FakeSession({PRIMARY: solaria, FALLBACK: _FakeBackend()}) + _, backend = await _router().embed(session, "q") + assert backend == "solaria" + + +class TestFailover: + async def test_primary_down_at_probe_falls_back_to_piha(self): + solaria = _FakeBackend(down=True) + piha = _FakeBackend(vector_value=0.02) + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + embedding, backend = await _router().embed(session, "q") + assert backend == "piha" + assert embedding[0] == 0.02 + assert solaria.embed_calls == 0 + + async def test_mid_embed_failure_serves_same_query_from_fallback(self): + # Plan §2 D2 step 3b: probe said up, the real embed then dies -- the SAME request + # must come back from the fallback, never as an error to the user. + solaria = _FakeBackend(embed_exc=aiohttp.ClientConnectionError("died mid-embed")) + piha = _FakeBackend(vector_value=0.02) + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + router = _router() + embedding, backend = await router.embed(session, "q") + assert backend == "piha" + assert embedding[0] == 0.02 + assert solaria.embed_calls == 1 + + async def test_down_verdict_is_cached_and_skips_primary_until_ttl_expires(self): + solaria = _FakeBackend(down=True) + piha = _FakeBackend() + clock = _FakeClock() + router = _router(clock=clock) + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + await router.embed(session, "q1") + probes = solaria.tags_calls + clock.now += 10 # still inside TTL -> no re-probe, straight to fallback + _, backend = await router.embed(session, "q2") + assert backend == "piha" + assert solaria.tags_calls == probes + + async def test_traffic_returns_to_primary_after_ttl_expiry(self): + # Test C from the task spec: SOLARIA comes back -> within one TTL window the + # router re-probes and routes to the GPU again. + solaria = _FakeBackend(down=True) + piha = _FakeBackend() + clock = _FakeClock() + router = _router(clock=clock) + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + _, backend = await router.embed(session, "q1") + assert backend == "piha" + solaria.down = False # SOLARIA wakes up + clock.now += 31 # TTL (30 s) expired + _, backend = await router.embed(session, "q2") + assert backend == "solaria" + + +class TestNoUsableBackend: + async def test_primary_down_without_fallback_raises_backend_error(self): + solaria = _FakeBackend(down=True) + session = _FakeSession({PRIMARY: solaria}) + with pytest.raises(EmbedBackendError, match="no EMBED_FALLBACK_URL"): + await _router(fallback_url=None).embed(session, "q") + + async def test_both_backends_down_raises_backend_error(self): + session = _FakeSession({PRIMARY: _FakeBackend(down=True), FALLBACK: _FakeBackend(down=True)}) + with pytest.raises(EmbedBackendError, match="unreachable"): + await _router().embed(session, "q") + + +class TestModelInvariant: + async def test_fallback_without_bge_m3_is_a_loud_error_not_a_silent_embed(self): + solaria = _FakeBackend(down=True) + piha = _FakeBackend(models=["llama3:8b"]) + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + with pytest.raises(ModelMismatchError, match="bge-m3"): + await _router().embed(session, "q") + assert piha.embed_calls == 0 # never embedded in the wrong vector space + + async def test_primary_without_bge_m3_fails_over_to_verified_fallback(self): + solaria = _FakeBackend(models=["llama3:8b"]) + piha = _FakeBackend() + session = _FakeSession({PRIMARY: solaria, FALLBACK: piha}) + _, backend = await _router().embed(session, "q") + assert backend == "piha" + assert solaria.embed_calls == 0 + + +class TestStatusReporting: + async def test_primary_status_up_and_fallback_status_up(self): + session = _FakeSession({PRIMARY: _FakeBackend(), FALLBACK: _FakeBackend()}) + router = _router() + assert await router.primary_status(session) == "up" + assert await router.fallback_status(session) == "up" + + async def test_fallback_status_unconfigured_without_fallback_url(self): + session = _FakeSession({PRIMARY: _FakeBackend()}) + assert await _router(fallback_url=None).fallback_status(session) == "unconfigured" + + async def test_primary_status_down_when_probe_fails(self): + session = _FakeSession({PRIMARY: _FakeBackend(down=True), FALLBACK: _FakeBackend()}) + assert await _router().primary_status(session) == "down" diff --git a/services/kb-query/tests/test_search.py b/services/kb-query/tests/test_search.py index cdcb97a..8a48ce0 100644 --- a/services/kb-query/tests/test_search.py +++ b/services/kb-query/tests/test_search.py @@ -75,30 +75,23 @@ class _FakeConn: raise AssertionError(f"unexpected query: {query}") -class _FakeEmbedResponse: - def __init__(self, payload): - self._payload = payload - - async def __aenter__(self): - return self - - async def __aexit__(self, *exc): - return False - - def raise_for_status(self): - pass - - async def json(self): - return self._payload - - class _FakeSession: - def __init__(self): - self.post_calls: list[dict] = [] + """run_search no longer talks HTTP itself -- embedding goes through the router + (below), so the session is just passed through untouched.""" - def post(self, url, json): - self.post_calls.append({"url": url, "json": json}) - return _FakeEmbedResponse({"embedding": [0.01] * 1024}) + +class _FakeRouter: + """Stands in for app.embed_router.EmbedRouter: returns a fixed embedding and the name + of the backend that 'served' it, mirroring EmbedRouter.embed's contract.""" + + def __init__(self, backend="solaria", primary_name="solaria"): + self._backend = backend + self.primary = type("B", (), {"name": primary_name})() + self.embed_calls: list[str] = [] + + async def embed(self, session, text): + self.embed_calls.append(text) + return [0.01] * 1024, self._backend class TestRunSearchHappyPath: @@ -110,11 +103,12 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "polisa PZU", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "polisa PZU", "cascade", "claude-haiku-4-5" ) assert result["query"] == "polisa PZU" assert result["mode"] == "cascade" assert result["sol_status"] == "up" + assert result["embed_backend"] == "solaria" assert len(result["results"]) == 1 hit = result["results"][0] assert hit["envelope_id"] == "paperless:119" @@ -131,7 +125,7 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "q", "flat", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "q", "flat", "claude-haiku-4-5" ) assert result["mode"] == "flat" assert len(result["results"]) == 1 @@ -145,7 +139,7 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "polisa PZU", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "polisa PZU", "cascade", "claude-haiku-4-5" ) hit = result["results"][0] assert hit["summary"] == "Polisa OC 2024" @@ -159,7 +153,7 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "polisa PZU", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "polisa PZU", "cascade", "claude-haiku-4-5" ) hit = result["results"][0] assert hit["summary"] is None @@ -180,7 +174,7 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "q", "hybrid", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "q", "hybrid", "claude-haiku-4-5" ) assert result["mode"] == "hybrid" envelope_ids = [r["envelope_id"] for r in result["results"]] @@ -201,7 +195,7 @@ class TestRunSearchHappyPath: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "q", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "q", "cascade", "claude-haiku-4-5" ) hit = result["results"][0] assert hit["source"] == "gmail" @@ -210,6 +204,24 @@ class TestRunSearchHappyPath: assert hit["mail_ui_url"] is None +class TestRunSearchEmbedBackendMarking: + async def test_fallback_embed_marks_backend_piha_and_sol_status_down(self): + # Task spec: the response must say WHICH backend embedded the query (quality + # debugging), and sol_status must reflect the router's world view -- the frontend + # renders "down" as "offline (fallback embed)". + conn = _FakeConn( + summaries=[("paperless:119", 0.1)], + chunks_by_envelope={"paperless:119": [(2, "hit text", 0.34)]}, + envelopes={"paperless:119": {"source": "paperless", "entities": []}}, + ) + result = await run_search( + conn, _FakeSession(), _FakeRouter(backend="piha"), "q", "cascade", "claude-haiku-4-5" + ) + assert result["embed_backend"] == "piha" + assert result["sol_status"] == "down" + assert len(result["results"]) == 1 + + class TestRunSearchNoGoodResults: async def test_results_above_no_answer_threshold_are_still_returned_unfiltered(self): # Plan §7: the 0.55 "no answer" colour threshold is a frontend concern -- the API @@ -222,7 +234,7 @@ class TestRunSearchNoGoodResults: ) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "unrelated query", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "unrelated query", "cascade", "claude-haiku-4-5" ) assert len(result["results"]) == 1 assert result["results"][0]["dist"] == 0.62 @@ -231,6 +243,6 @@ class TestRunSearchNoGoodResults: conn = _FakeConn(summaries=[], chunks_by_envelope={}, envelopes={}) session = _FakeSession() result = await run_search( - conn, session, "http://fake-ollama", "nothing matches", "cascade", "bge-m3", "claude-haiku-4-5" + conn, session, _FakeRouter(), "nothing matches", "cascade", "claude-haiku-4-5" ) assert result["results"] == [] diff --git a/services/ollama-piha/README.md b/services/ollama-piha/README.md new file mode 100644 index 0000000..6336d93 --- /dev/null +++ b/services/ollama-piha/README.md @@ -0,0 +1,76 @@ +# ollama-piha + +Local CPU Ollama on **PIHA**, serving exactly one purpose: the **fallback embed +backend** for `kb-query` while SOLARIA (the GPU node, ~16 h/day powered off) +sleeps. Model: `bge-m3` — the **same** model as SOLARIA's Ollama, because query +embeddings must live in the same vector space as the pgvector index +(`document_chunk.embedding VECTOR(1024)`); a different/smaller model is not an +option (module 5 phase 4 plan §2 decision 2). + +Expected latency: bge-m3 embeds in ~207 ms on SOLARIA's GPU vs ~790 ms on x86 +CPU; on the Pi 5 expect single seconds per query (plus model load, since the +model is never resident — see below). Slower but alive beats fast but dead. + +## Design constraints + +- **`OLLAMA_KEEP_ALIVE=0`** (pinned in compose): PIHA is the RAM-bound 8 GB box + shared with Home Assistant. The model is unloaded immediately after every + call — a transient ~1.5–2 GB spike per embed, ~100 MB idle daemon, never a + resident cost. +- **`mem_limit: 2560m`** (host override, `hosts/piha/runtime/ollama-piha/`): + hard cgroup ceiling, plan §2 D2 starting value. The cgroup OOM killer + restarts this container instead of the host OOM killer picking a victim + (which could be Home Assistant). Confirm/trim after live calibration. +- **Bind**: `127.0.0.1` + `LAN_BIND_IP` (192.168.31.5) only — kb-query calls it + over the host LAN interface (same pattern as kb-query → kb-postgres:5433). + Never `0.0.0.0`, never a Tailscale bind, no public ingress. +- **Storage**: Docker named volume `ollama_piha_models` (NVMe data-root), not a + bind mount — the ollama image runs as in-container root and would break + PIHA's uid pattern (host oskar=1004, containers uid 1000, setgid group pi) + if it wrote to a shared bind directory. + +## Deploy (PIHA, master, after merge) + +```bash +cd ~/homelab-codex-ws && git pull +cp services/ollama-piha/env.example services/ollama-piha/.env # LAN_BIND_IP +docker compose -f services/ollama-piha/docker-compose.yml \ + -f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \ + --env-file services/ollama-piha/.env up -d +``` + +**Then pull the model — this does NOT happen automatically:** + +```bash +docker exec ollama-piha ollama pull bge-m3 +``` + +Verify: + +```bash +services/ollama-piha/healthcheck.sh # checks container + API + bge-m3 present +time curl -s http://127.0.0.1:11434/api/embeddings \ + -d '{"model":"bge-m3","prompt":"test kalibracyjny"}' | head -c 80 +``` + +(`deploy-node.sh` on PIHA also picks this service up from +`hosts/piha/services.yaml` once `.env` exists — the `ollama pull bge-m3` step +stays manual either way.) + +## Calibration (plan §5 step 4 — gate, not formality) + +Before trusting the fallback under load, on live PIHA at a normal (not +night-quiet) hour: run a few embeds as above while watching +`docker stats ollama-piha`, note peak RAM and wall time. Verdict per plan §5 +step 5: keep as default fallback / tune `mem_limit` / fall back to explicit +503 degradation. + +## Relation to kb-query + +kb-query's router (`services/kb-query/app/embed_router.py`) health-checks +SOLARIA with a ~30 s cache and only sends embeds here while SOLARIA is down. +kb-query verifies at first use that this backend actually serves `bge-m3` +(`/api/tags`) and refuses to embed against a mismatched model. Configuration: +`EMBED_FALLBACK_URL=http://192.168.31.5:11434` in `services/kb-query/.env`. +See `services/kb-query/README.md` for the fallback verification plan (tests +A/B/C). diff --git a/services/ollama-piha/docker-compose.yml b/services/ollama-piha/docker-compose.yml new file mode 100644 index 0000000..8a39598 --- /dev/null +++ b/services/ollama-piha/docker-compose.yml @@ -0,0 +1,33 @@ +services: + ollama-piha: + # Multi-arch image with a published linux/arm64 manifest — runs natively on + # the Pi 5, no emulation (same mechanism as pgvector/pgvector:pg16 on PIHA). + image: ollama/ollama:latest + container_name: ollama-piha + restart: unless-stopped + environment: + # PIHA is the RAM-bound 8 GB box shared with Home Assistant. bge-m3 must be + # unloaded from RAM immediately after every call — a short spike per embed + # (idle daemon ~100 MB), never a resident ~2 GB cost. Fallback traffic is + # rare by design: kb-query only embeds here while SOLARIA sleeps. + - OLLAMA_KEEP_ALIVE=0 + ports: + # Loopback: healthcheck.sh + manual curl on the node itself. + # LAN IP: kb-query runs in a separate compose project/network and reaches + # this backend over the host's LAN interface — same reasoning as + # kb-query -> kb-postgres (:5433). No 0.0.0.0, no Tailscale bind: nothing + # off-node ever embeds here (SOLARIA-side callers use SOLARIA's own Ollama). + # Requires .env (from env.example) next to this file at deploy. + - "127.0.0.1:11434:11434" + - "${LAN_BIND_IP}:11434:11434" + volumes: + # Named volume, NOT a bind mount: the ollama image runs as in-container + # root and would fill a bind dir with root-owned files, breaking PIHA's + # uid pattern (host oskar=1004, containers uid 1000 + setgid group pi). + # Docker-managed volumes live under the data-root on the NVMe — same + # convention as kb_postgres_data. + - ollama_piha_models:/root/.ollama + +volumes: + ollama_piha_models: + name: ollama_piha_models diff --git a/services/ollama-piha/env.example b/services/ollama-piha/env.example new file mode 100644 index 0000000..b495c7e --- /dev/null +++ b/services/ollama-piha/env.example @@ -0,0 +1,7 @@ +# ollama-piha — copy to .env (gitignored) next to docker-compose.yml; docker +# compose picks it up automatically. Same convention as services/kb-query. + +# LAN IP of PIHA. The published port (11434) binds ONLY to 127.0.0.1 and this +# interface — never 0.0.0.0. kb-query's EMBED_FALLBACK_URL points at +# http://:11434. Verify after host rebuilds: ip -4 addr. +LAN_BIND_IP=192.168.31.5 diff --git a/services/ollama-piha/healthcheck.sh b/services/ollama-piha/healthcheck.sh new file mode 100755 index 0000000..23fb4c3 --- /dev/null +++ b/services/ollama-piha/healthcheck.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# Healthcheck for ollama-piha (local CPU embed fallback for kb-query) + +# Container must be running +if ! docker ps --filter "name=ollama-piha" --filter "status=running" | grep -qw "ollama-piha"; then + echo "[FAIL] ollama-piha container is not running" + exit 1 +fi + +# API must answer on loopback +TAGS=$(curl -sf http://127.0.0.1:11434/api/tags) +if [ $? -ne 0 ]; then + echo "[FAIL] ollama-piha API is not responding on 127.0.0.1:11434" + exit 1 +fi + +# bge-m3 must be pulled — without it the API answers /api/tags but every kb-query +# fallback embed fails the model check (deploy step 3 in README.md). +if ! echo "$TAGS" | grep -q '"bge-m3'; then + echo "[FAIL] model bge-m3 missing (run: docker exec ollama-piha ollama pull bge-m3)" + exit 1 +fi + +echo "[OK] ollama-piha is healthy (bge-m3 present)" +exit 0 diff --git a/services/ollama-piha/service.yaml b/services/ollama-piha/service.yaml new file mode 100644 index 0000000..fe198d8 --- /dev/null +++ b/services/ollama-piha/service.yaml @@ -0,0 +1,26 @@ +service: + name: ollama-piha + owner_node: piha + role: embed-fallback # module 5 phase 4 (plan §2 D2/§5): local CPU bge-m3 embed backend for kb-query while SOLARIA sleeps + exposure: private # 127.0.0.1 + LAN bind only — never 0.0.0.0, never public ingress, no Tailscale bind + dependencies: [] + ports: + - container: 11434 + host: 11434 # bound to 127.0.0.1 and LAN_BIND_IP only + protocol: tcp + healthcheck: + type: http + endpoint: http://127.0.0.1:11434/api/tags + interval: 1m + timeout: 10s + retries: 3 + restart_policy: unless-stopped + persistence: + paths: [] # models live in the Docker named volume ollama_piha_models (NVMe data-root) + runtime: + config_files: + - .env # LAN_BIND_IP (gitignored, from env.example) + env_vars: + - LAN_BIND_IP # required — compose port-bind interpolation + # OLLAMA_KEEP_ALIVE is pinned to 0 in docker-compose.yml (not operator-tunable): + # the model must never stay resident on the RAM-bound, HA-sharing Pi 5.