74 lines
3.9 KiB
Markdown
74 lines
3.9 KiB
Markdown
|
# ollama-piha
|
||
|
|
|
||
|
|
Local, CPU-only Ollama on **PIHA** (arm64, 4 cores, no acceleration —
|
||
|
|
`hosts/piha/capabilities.yaml`), serving exactly one purpose: `kb-query`'s
|
||
|
|
embed fallback (`services/kb-query/app/fallback.py`, module 5 phase 4 plan
|
||
|
|
§2 decision 2 / §5) when Ollama@SOLARIA is unreachable or times out.
|
||
|
|
|
||
|
|
**This is not a general-purpose Ollama instance** — no other service should
|
||
|
|
point at it. It runs `bge-m3` only, the same embedding model kb-query's
|
||
|
|
startup invariant enforces against `document_chunk`/`document_summary`
|
||
|
|
(`services/kb-query/app/startup.py`). Do not pull additional models onto it.
|
||
|
|
|
||
|
|
## Why `OLLAMA_KEEP_ALIVE=0`
|
||
|
|
|
||
|
|
The model is loaded into RAM only for the duration of a request and released
|
||
|
|
immediately after, rather than staying resident. On a memory-constrained
|
||
|
|
RPi5 already running kb-postgres, paperless, Home Assistant, Immich,
|
||
|
|
Forgejo, and more (see `docs/infra/piha-slim-audit-2026-07-02.md`), a second
|
||
|
|
permanently-resident ~1.5-2 GB model is a worse trade than a short RAM spike
|
||
|
|
that only happens when this fallback is actually exercised (SOLARIA down —
|
||
|
|
rare, plan §1.2 `availability_target: medium`).
|
||
|
|
|
||
|
|
## Deploy
|
||
|
|
|
||
|
|
1. `git pull` on PIHA.
|
||
|
|
2. `cp services/ollama-piha/env.example services/ollama-piha/.env` (fill in
|
||
|
|
`LAN_BIND_IP` if it differs from the default).
|
||
|
|
3. `docker compose -f services/ollama-piha/docker-compose.yml -f hosts/piha/runtime/ollama-piha/docker-compose.override.yml up -d`
|
||
|
|
4. `docker exec ollama-piha ollama pull bge-m3` — not baked into the image;
|
||
|
|
the model must be pulled once after first start (persists in the
|
||
|
|
`/opt/homelab/data/ollama-piha` named volume across restarts).
|
||
|
|
5. Verify: `services/ollama-piha/healthcheck.sh`, then
|
||
|
|
`curl http://192.168.31.5:11434/api/tags` should list `bge-m3`.
|
||
|
|
|
||
|
|
Point `kb-query`'s `OLLAMA_PIHA_URL` at `http://192.168.31.5:11434` once this
|
||
|
|
is live (see `services/kb-query/env.example`).
|
||
|
|
|
||
|
|
## Calibration status (plan §5) — GO, measured live 2026-07-27
|
||
|
|
|
||
|
|
Measured on live PIHA under normal load (kb-postgres, paperless, Immich,
|
||
|
|
Home Assistant, Forgejo, etc. all running, not a quiet-night window),
|
||
|
|
3 sequential embed calls through `docker exec ollama-piha ollama pull bge-m3`
|
||
|
|
+ `/api/embeddings`:
|
||
|
|
|
||
|
|
- **Latency**: 5.25s (first call), 4.41s, 4.16s — consistently single-digit
|
||
|
|
seconds, never tens of seconds. Latency doesn't drop on later calls because
|
||
|
|
`OLLAMA_KEEP_ALIVE=0` reloads the model every time by design (`ollama ps`
|
||
|
|
shows zero resident models between calls) — this is the expected trade-off
|
||
|
|
documented above, not a bug.
|
||
|
|
- **RAM**: peak ~983 MiB during a burst (`docker stats`, baseline idle ~66
|
||
|
|
MiB), comfortably inside the `mem_limit: 2560m` ceiling
|
||
|
|
(`hosts/piha/runtime/ollama-piha/docker-compose.override.yml`). System-wide
|
||
|
|
`available` memory never dropped below ~1.3 GiB during the burst and
|
||
|
|
settled back to ~4.2 GiB after — well above the "≥500 MB spare" bar from
|
||
|
|
the plan.
|
||
|
|
|
||
|
|
**Verdict: GO — enabled as the default fallback**, no
|
||
|
|
`KB_QUERY_LOCAL_FALLBACK_ENABLED`-style flag needed. `kb-query`'s
|
||
|
|
`OLLAMA_PIHA_URL` points at this container's real LAN address
|
||
|
|
(`http://192.168.31.5:11434`) in the live PIHA deployment. Verified live: a
|
||
|
|
sol-down simulation (`OLLAMA_URL` on kb-query pointed at an unreachable
|
||
|
|
SOLARIA address) produced `sol_status: "down"` and correct `/search` results
|
||
|
|
from this container, with `dist` within ~3e-4 of the SOLARIA-GPU baseline
|
||
|
|
(same top-k order, same hit@3 gate outcome) — see
|
||
|
|
`docs/sessions/2026-07-27-kb-f4-fallback.md` for the full numbers.
|
||
|
|
|
||
|
|
**One-time finding from this calibration**: PIHA also had a leftover, fully
|
||
|
|
undocumented *native* (non-Docker) `ollama.service` (systemd, v0.6.1, running
|
||
|
|
since 2026-06-22, zero models ever pulled) that conflicted with this
|
||
|
|
container's port binding. Confirmed dead (only this session's own probe
|
||
|
|
requests in its journal) and disabled (`systemctl disable --now`, not
|
||
|
|
uninstalled — reversible) before deploying this container. See the session
|
||
|
|
doc for the backlog note to fully remove it if nothing breaks.
|