From 3292ab54e252415169079cf4d7d343d866b2918b Mon Sep 17 00:00:00 2001 From: oskar Date: Tue, 4 Aug 2026 15:04:29 +0200 Subject: [PATCH] =?UTF-8?q?feat(kb):=20SPLIT=20service+runbook=20=E2=80=94?= =?UTF-8?q?=2010=20serwisow=20->=2020=20dokumentow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wzorzec mechaniczny: sekcje deploy/verify/install/testy wycinane do kb/runbooks/-*.md, reszta zostaje dokumentem type: service. Wzajemne `links` w obie strony. Tresc sekcji nietknieta — przenoszone doslownie, dodany wylacznie naglowek H1 nowego runbooka. kb-query, paperless-worker, planner-agent, ha-diag-agent, ollama-piha, narty27, home-assistant, ha-mcp, job-gmail-header-backfill, job-mail-body-ingest. Weryfikacja: dla kazdego pliku multizbior niepustych linii (main + runbook) == oryginal z HEAD. Zero zgubionych, zero dodanych. Recon szacowal 13 splitow service+runbook; faktycznie 2-typowych jest 10, pozostale 5 (paperless, nextcloud, gokapi, fleet-prometheus, deploy-runner) sa 3-typowe i ida osobno jako splity wielotypowe. Co-Authored-By: Claude Opus 5 (1M context) --- kb/runbooks/gmail-header-backfill-run.md | 49 ++++++++ kb/runbooks/ha-diag-agent-runbook.md | 64 +++++++++++ kb/runbooks/ha-mcp-install.md | 97 ++++++++++++++++ .../runbooks/home-assistant-deploy.md | 35 ++---- kb/runbooks/kb-query-deploy.md | 61 ++++++++++ kb/runbooks/mail-body-ingest-run.md | 44 ++++++++ .../runbooks/narty27-deploy.md | 33 ++---- kb/runbooks/ollama-piha-deploy.md | 63 +++++++++++ kb/runbooks/paperless-worker-deploy.md | 67 +++++++++++ kb/runbooks/planner-agent-deploy.md | 105 ++++++++++++++++++ .../README.md => kb/services/ha-diag-agent.md | 62 ++--------- .../ha-mcp/README.md => kb/services/ha-mcp.md | 95 ++-------------- kb/services/home-assistant.md | 37 ++++++ .../services/job-gmail-header-backfill.md | 47 ++------ .../services/job-mail-body-ingest.md | 42 ++----- .../README.md => kb/services/kb-query.md | 59 ++-------- kb/services/narty27.md | 35 ++++++ .../README.md => kb/services/ollama-piha.md | 61 ++-------- .../services/paperless-worker.md | 65 ++--------- .../README.md => kb/services/planner-agent.md | 103 ++--------------- 20 files changed, 722 insertions(+), 502 deletions(-) create mode 100644 kb/runbooks/gmail-header-backfill-run.md create mode 100644 kb/runbooks/ha-diag-agent-runbook.md create mode 100644 kb/runbooks/ha-mcp-install.md rename services/home-assistant/README.md => kb/runbooks/home-assistant-deploy.md (80%) create mode 100644 kb/runbooks/kb-query-deploy.md create mode 100644 kb/runbooks/mail-body-ingest-run.md rename services/narty27/README.md => kb/runbooks/narty27-deploy.md (60%) create mode 100644 kb/runbooks/ollama-piha-deploy.md create mode 100644 kb/runbooks/paperless-worker-deploy.md create mode 100644 kb/runbooks/planner-agent-deploy.md rename services/ha-diag-agent/README.md => kb/services/ha-diag-agent.md (68%) rename services/ha-mcp/README.md => kb/services/ha-mcp.md (70%) create mode 100644 kb/services/home-assistant.md rename jobs/gmail-header-backfill/README.md => kb/services/job-gmail-header-backfill.md (83%) rename jobs/mail-body-ingest/README.md => kb/services/job-mail-body-ingest.md (80%) rename services/kb-query/README.md => kb/services/kb-query.md (79%) create mode 100644 kb/services/narty27.md rename services/ollama-piha/README.md => kb/services/ollama-piha.md (51%) rename services/paperless-worker/README.md => kb/services/paperless-worker.md (73%) rename services/planner-agent/README.md => kb/services/planner-agent.md (67%) diff --git a/kb/runbooks/gmail-header-backfill-run.md b/kb/runbooks/gmail-header-backfill-run.md new file mode 100644 index 0000000..4a24998 --- /dev/null +++ b/kb/runbooks/gmail-header-backfill-run.md @@ -0,0 +1,49 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-14 +links: + - ../services/job-gmail-header-backfill.md +--- + +# gmail-header-backfill — uruchomienie i testy + +## Usage + +```bash +# Dry run (default) — parse and count only, no DB writes: +gmail-header-backfill --dsn postgresql://kb:@localhost:5433/kb --limit 100 + +# Real run — apply the UPDATE for this slice: +gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply + +# Next slice — offset is stable/deterministic (ORDER BY id), independent of +# how many rows in earlier slices were already backfilled: +gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply +``` + +DSN can also come from the `KB_DSN` env var instead of `--dsn`. + +`--limit`/`--offset` exist so the full 225 030-row backfill can be run in +verifiable partitions instead of one long unattended run (plan §5.3) — start +small (`--limit 100`), check the result in the DB, then widen. + +## Tests + +```bash +pip install -e jobs/gmail-header-backfill/ +cd jobs/gmail-header-backfill && pytest +``` + +Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml` +bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an +in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`, +quoted display names with commas, multiple `Delivered-To` occurrences, RFC +2047 encoded-words including Polish diacritics, malformed encoded-words that +must not raise, missing/multiple `From`, `date_raw` preserving literal text +vs. `Date` header reformatting), idempotency (rows already carrying a +`headers` entity are skipped and never re-appended), batch flushing, and +`--limit`/`--offset` query shape. + diff --git a/kb/runbooks/ha-diag-agent-runbook.md b/kb/runbooks/ha-diag-agent-runbook.md new file mode 100644 index 0000000..607d703 --- /dev/null +++ b/kb/runbooks/ha-diag-agent-runbook.md @@ -0,0 +1,64 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-06-11 +links: + - ../services/ha-diag-agent.md +--- + +# ha-diag-agent — deployment i testy + +## First-time deployment + +See **[DEPLOY.md](DEPLOY.md)** for the full procedure: HA token creation, +per-host `.env` config, deploy commands, verification steps, 48h shadow-mode +observation, and rollback. + +**Shadow mode** (`HA_DIAG_SHADOW_MODE`, default `true` on the control-plane): +`ha_websocket_dead` events are downgraded to `alert_only` with a `[SHADOW MODE]` +note instead of queuing an automatic `container_restart`. Set to `false` in +`/opt/homelab/config/control-plane/.env` on the VPS when ready for live actions. + +## Deployment + +```bash +# 1. Create config on target node +ssh oskar@ +mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent +cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF' +HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra +HA_TOKEN= +NODE_NAME=piha # or chelsty-infra +LOCATION_TAG=ken # or chelsty +CHECK_INTERVAL=60 +EOF + +# 2. Deploy +scripts/deploy/deploy.sh --service ha-diag-agent + +# 3. Verify +docker ps --filter name=ha-diag-agent +docker exec ha-diag-agent python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8087/health', timeout=5).read().decode())" +``` + +### chelsty-infra note + +`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated): +```bash +docker-compose -f docker-compose.yml up -d --build +``` + +### HA long-lived token + +In HA UI: Profile → Long-Lived Access Tokens → Create token. + +## Running Tests + +```bash +cd services/ha-diag-agent +pip install -e ".[dev]" +pytest tests/ -v +``` + diff --git a/kb/runbooks/ha-mcp-install.md b/kb/runbooks/ha-mcp-install.md new file mode 100644 index 0000000..76c9f6f --- /dev/null +++ b/kb/runbooks/ha-mcp-install.md @@ -0,0 +1,97 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-30 +links: + - ../services/ha-mcp.md +--- + +# ha-mcp — instalacja i rejestracja + +## Install + +The `mcp` SDK (and its pydantic/anyio/httpx dependency tree) is not packaged +for Debian and is not needed by anything else in this repo, so it goes into a +venv rather than into the system Python: + +```bash +python3 -m venv --system-site-packages services/ha-mcp/.venv +services/ha-mcp/.venv/bin/pip install mcp pytest +``` + +`--system-site-packages` is deliberate: `requests`, `PyYAML` and +`websocket-client` are already installed system-wide and used by +`scripts/ha/lib/*`; the venv reuses those exact versions instead of shadowing +them with a second copy. `.venv/` is already covered by the repo `.gitignore`. + +`pip install --break-system-packages mcp` also works and is one line shorter, +but it writes into the system interpreter that runs every deploy script on +this workstation — a venv keeps a 40-package dependency tree out of that blast +radius for a tool only Claude Code uses. Use the venv. + +`run.sh` prefers `services/ha-mcp/.venv/bin/python` and falls back to the +system `python3`; if neither can import `mcp` it exits with one actionable +line on stderr rather than a traceback. + +## Registration in Claude Code + +`.mcp.json` in the repo root (project scope — shared with anyone who checks +out this repo): + +```json +{ + "mcpServers": { + "ha": { + "command": "./services/ha-mcp/run.sh", + "args": [], + "env": {} + } + } +} +``` + +The command path is relative, so it resolves in any checkout (main or task +worktree) as long as Claude Code is started from the repo root. Start CC +there; on first run it asks whether to trust the project's MCP servers. Check +with `/mcp` — the server appears as `ha`, its tools as `mcp__ha__`. +Which repo the server reads is derived from its own location; override with +the `HA_MCP_REPO` environment variable if you ever need to point one checkout +at another's config. + +## Standalone + +```bash +./services/ha-mcp/run.sh # stdio server — speaks JSON-RPC on stdout +services/ha-mcp/tests/run.sh # offline test suite (no HA, no token) +services/ha-mcp/.venv/bin/python services/ha-mcp/tests/smoke_live.py [instance] [entity_id] +``` + +`smoke_live.py` is the only thing here that touches the network (GET only). A +one-off tool call without a client is easiest through the package: + +```bash +PYTHONPATH=services/ha-mcp/src services/ha-mcp/.venv/bin/python -c " +from ha_mcp import tools; from ha_mcp.backend import LiveBackend +from ha_mcp.config import get_instance, repo_root +n, c = get_instance('ken', repo_root()); print(tools.instance_status(LiveBackend(n, c, repo_root())))" +``` + +## Tests + +```bash +services/ha-mcp/tests/run.sh # 42 tests, offline +``` + +Offline in the same sense as `scripts/ha/tests/*`: no network, no HA +instance, no token. Small hand-made fixtures under `tests/fixtures/` (shaped +like real `/api/states` and WebSocket registry payloads) cover the tool logic; +the repo's own `config/ken/automations/` and newest `fixtures/ken-states-*.yaml` +cover the repo-backed and at-scale paths (ranking over ~1650 real entities +behaves differently from ranking over ten). + +Live smoke (read-only, run against `ken` on 2026-07-30): +`instance_status` → HA 2026.7.2, 1647 entities, 377 unavailable, 115 +automations, 13 areas; `get_state("sensor.thsalon_temperature")` → `24.5 °C`. + diff --git a/services/home-assistant/README.md b/kb/runbooks/home-assistant-deploy.md similarity index 80% rename from services/home-assistant/README.md rename to kb/runbooks/home-assistant-deploy.md index 040fbd0..c44a7d5 100644 --- a/services/home-assistant/README.md +++ b/kb/runbooks/home-assistant-deploy.md @@ -1,29 +1,14 @@ -# home-assistant (configs-as-code) +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-22 +links: + - ../services/home-assistant.md +--- -**Status: phase 1 (partial).** Read-only import tooling, plus a deploy -(repo -> instance) write path for the `api` adapter's automations/scripts/ -scenes scope only (`scripts/ha/deploy.sh`) — see "Deploy" below. Dashboards, -helpers, and the `docker-exec` adapter have no write path yet. See -`DESIGN.md` for the full phasing, adapter, sync, and validation model, and -for the open questions still blocking phase 2/3. - -## Layout - -``` -services/home-assistant/ -├── DESIGN.md # decision registry — read this first -├── instances.yaml # per-instance adapter/host/token config -├── .gitignore # excludes secrets/db/log/token paths from every import -├── config// # canonical, normalized /config mirror per instance -├── storage-export// # curated .storage/* export (registries, dashboards) -└── fixtures/ # dated /api/states snapshots -``` - -Instances: `ken` (RPi4/HAOS, LAN 192.168.31.7, api adapter — canonical home -instance since the 2026-07-22 cutover), `ken-legacy` (PIHA, container -`homeassistant5`, docker-exec adapter, archived — pre-migration instance, -import only, never deploy), `chelsty-ha` (Tailscale, api adapter — see -`instances.yaml` and `DESIGN.md` "Incident log"). +# home-assistant — import, deploy, testy ## Import diff --git a/kb/runbooks/kb-query-deploy.md b/kb/runbooks/kb-query-deploy.md new file mode 100644 index 0000000..e4fd5f4 --- /dev/null +++ b/kb/runbooks/kb-query-deploy.md @@ -0,0 +1,61 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-29 +links: + - ../services/kb-query.md +--- + +# kb-query — deploy i weryfikacja (PIHA) + +## Deploy (PIHA) + +0. Prerequisite: `ollama-piha` deployed and `bge-m3` pulled — see + `services/ollama-piha/README.md` (the pull is a **manual** deploy step). +1. `git pull` on PIHA (`~/homelab-codex-ws`). +2. `cp services/kb-query/env.example services/kb-query/.env` and fill in the + real `KB_DSN` password (the template already sets `EMBED_FALLBACK_URL`). + On an existing install: add `EMBED_FALLBACK_URL=http://192.168.31.5:11434` + to the existing `.env`. +3. ``` + docker compose -f services/kb-query/docker-compose.yml \ + -f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build + ``` +4. Verify: `services/kb-query/healthcheck.sh`, then from PIHA: + `curl "http://192.168.31.5:8230/search?q=test"` and open + `http://192.168.31.5:8230/` in a browser. + +## Fallback verification (execution: operator, after deploy) + +- **Test A — SOLARIA online**: query via UI/`curl`; response has + `"embed_backend": "solaria"`, `docker logs kb-query` shows + `backend=solaria`, latency ~sub-second. +- **Test B — SOLARIA offline**: either wait for the nightly power-off, or + simulate: set `EMBED_PRIMARY_URL=http://192.0.2.1:11434` (TEST-NET, always + unreachable) in `.env` and `docker compose … up -d` again. Query still + works; response has `"embed_backend": "piha"`, log shows `backend=piha` + plus a `circuit open for 30s` warning on the first hit; latency visibly + higher (CPU + cold model load each time, `OLLAMA_KEEP_ALIVE=0`). Revert + `.env` afterwards if simulated. +- **Test C — SOLARIA returns**: after it is back up, within ≤30 s (one + health-cache TTL) responses show `"embed_backend": "solaria"` again, no + restart needed. + +## Tests + +``` +pip install -e packages/kb-retrieval/ +cd services/kb-query && pip install -r requirements.txt pytest pytest-asyncio && pytest +``` + +Unit tests mock the DB connection and Ollama HTTP session (no live DB/Ollama +required) — same style as `packages/kb-retrieval/tests/`. `tests/test_frontend.py` +drives `GET /`/`/static/*` through FastAPI's `TestClient` without entering it +as a context manager, so the DB-requiring `lifespan` never runs. + +Frontend JS has its own pure-function tests (query-URL encoding, threshold +colouring, envelope grouping), run without a browser via Node's built-in +test runner: `node --test services/kb-query/tests/frontend/`. + diff --git a/kb/runbooks/mail-body-ingest-run.md b/kb/runbooks/mail-body-ingest-run.md new file mode 100644 index 0000000..c406594 --- /dev/null +++ b/kb/runbooks/mail-body-ingest-run.md @@ -0,0 +1,44 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-22 +links: + - ../services/job-mail-body-ingest.md +--- + +# mail-body-ingest — uruchomienie i testy + +## Usage + +```bash +# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero +# DB writes (including the threading UPDATE): +mail-body-ingest --dsn postgresql://kb:@piha:5433/kb --archive-root /home/oskar/kb/mail/archive + +# Etap A pilot — last 12 months only (plan Decyzja 9): +mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1 + +# Smoke-test slice: +mail-body-ingest --dsn ... --apply --limit 10 +``` + +DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default +`http://localhost:11434` — this job is meant to run where Ollama lives). + +## Tests + +```bash +pip install -e "jobs/mail-body-ingest[dev]" +cd jobs/mail-body-ingest && pytest +``` + +Pure unit tests (48), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect` +and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers: +quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/ +gmail_quote skipping), newsletter classification, threading extraction, prefix building, +body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse +fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks +never reaching Ollama, Ollama-offline batch isolation, and dimension-mismatch abort. + diff --git a/services/narty27/README.md b/kb/runbooks/narty27-deploy.md similarity index 60% rename from services/narty27/README.md rename to kb/runbooks/narty27-deploy.md index 2767e2f..8403267 100644 --- a/services/narty27/README.md +++ b/kb/runbooks/narty27-deploy.md @@ -1,27 +1,14 @@ -# narty27 +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-31 +links: + - ../services/narty27.md +--- -Static hosting for a single self-contained `viz.html` (narty 2027 / Saalbach KB -export) on PIHA. Plain `nginx:alpine` serving one Docker named volume — no build, -no database, no dependencies. - -- URL: `http://192.168.31.5:8240/viz.html` (and `/` — same file, see below) -- Exposure: private (LAN/Tailscale only; no npm vhost, no public ingress) -- Volume: `narty27_narty27_content` → `/usr/share/nginx/html:ro` - -## Content is personal and lives outside the repo - -The visualisation is personal content. It is **never committed** — not to this -repo, not to any other. It exists in exactly two places: - -1. the source on SOLARIA (`~/narty-2027/saalbach-kb/viz.html`), and -2. the `narty27_narty27_content` Docker volume on PIHA. - -There is no backup job and no bind mount under `/opt/homelab/data/`. If the -volume is lost, re-run the update procedure below from SOLARIA. - -Two copies of the same file are stored in the volume: `viz.html` (canonical name) -and `index.html` (so the bare root `http://192.168.31.5:8240/` works without a -path). Both must be refreshed together on every update. +# narty27 — uruchomienie i operacje ## Updating the content (from SOLARIA) diff --git a/kb/runbooks/ollama-piha-deploy.md b/kb/runbooks/ollama-piha-deploy.md new file mode 100644 index 0000000..dc44fb9 --- /dev/null +++ b/kb/runbooks/ollama-piha-deploy.md @@ -0,0 +1,63 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-30 +links: + - ../services/ollama-piha.md +--- + +# ollama-piha — deploy i kalibracja + +## Deploy (PIHA, master, after merge) + +```bash +cd ~/homelab-codex-ws && git pull +cp services/ollama-piha/env.example services/ollama-piha/.env # LAN_BIND_IP +docker compose -f services/ollama-piha/docker-compose.yml \ + -f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \ + --env-file services/ollama-piha/.env up -d +``` + +**Then pull the model — this does NOT happen automatically:** + +```bash +docker exec ollama-piha ollama pull bge-m3 +``` + +Verify: + +```bash +services/ollama-piha/healthcheck.sh # checks container + API + bge-m3 present +time curl -s http://127.0.0.1:11434/api/embeddings \ + -d '{"model":"bge-m3","prompt":"test kalibracyjny"}' | head -c 80 +``` + +(`deploy-node.sh` on PIHA also picks this service up from +`hosts/piha/services.yaml` once `.env` exists — the `ollama pull bge-m3` step +stays manual either way.) + +## Calibration (plan §5 step 4 — gate, not formality) + +Before trusting the fallback under load, on live PIHA at a normal (not +night-quiet) hour: run a few embeds as above while watching +`docker stats ollama-piha`, note peak RAM and wall time. Verdict per plan §5 +step 5: keep as default fallback / tune `mem_limit` / fall back to explicit +503 degradation. + +**Calibration status: measured 2026-07-27 — verdict GO** (live PIHA under +normal load, 3 consecutive `/api/embeddings` calls after `ollama pull bge-m3`; +full protocol in `docs/sessions/2026-07-27-kb-f4-fallback.md` §4): + +- Latency: 5.25 s (cold start) / 4.41 s / 4.16 s — single seconds as expected, + no warm-up between calls by design (`OLLAMA_KEEP_ALIVE=0` releases the model + after every request, `ollama ps` shows nothing resident in between). +- RAM: idle ~66 MiB, burst peak ~983 MiB (`docker stats` sampled at 0.3 s) — + well inside the 2560m ceiling; host `available` never dropped below ~1.3 GiB. +- Kept as **default fallback** (no feature flag). The measurement was taken + against the same container configuration this repo deploys (image, + `OLLAMA_KEEP_ALIVE=0`, `mem_limit: 2560m`), so it carries over; only the + model storage differed (bind mount then, named volume now), which does not + affect RAM/latency. + diff --git a/kb/runbooks/paperless-worker-deploy.md b/kb/runbooks/paperless-worker-deploy.md new file mode 100644 index 0000000..7c517cb --- /dev/null +++ b/kb/runbooks/paperless-worker-deploy.md @@ -0,0 +1,67 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-07-12 +links: + - ../services/paperless-worker.md +--- + +# Paperless OCR worker — NFS i cutover + +## NFS: export na PIHA, mount na SOLARIA + +Transfer idzie po **LAN** (PIHA `192.168.31.5` ↔ SOLARIA `192.168.31.70`, +1 Gb/s, ten sam switch) — NIE po Tailscale. Przepustowość nie jest wąskim +gardłem OCR. + +### Host-side na PIHA (NIE w compose — krok przy deployu modułu 3) + +``` +# /etc/exports na PIHA — export TYLKO dla SOLARII: +/opt/homelab/data/paperless 192.168.31.70(rw,sync,no_subtree_check,no_root_squash) +``` + +```bash +sudo apt install nfs-kernel-server # jesli brak +sudo exportfs -ra +``` + +`no_root_squash` jest potrzebne, bo entrypoint obrazu (root) robi `chown` na +katalogach przy starcie kontenera na SOLARII; export jest ograniczony do +jednego IP w zaufanym LAN. + +### Strona SOLARII + +Mounty definiuje compose jako named volumes z driverem NFS — **zero wpisów +w /etc/fstab**; jedyny host-side wymóg to pakiet klienta: + +```bash +sudo apt install nfs-common +``` + +### UID mapping (krytyczne) + +Pliki na exporcie mają numerycznego właściciela — NFS nie tłumaczy nazw. +Dlatego `USERMAP_UID/GID=1000` jest ustawione **w obu** compose (PIHA +i SOLARIA); zmiana po jednej stronie = worker traci dostęp do plików. +Weryfikacja po deployu: `./healthcheck.sh` robi test zapisu na mount. + +## Cutover checklist (przy deployu modułu 3 — po działającym module 2) + +1. ✅ `services/paperless/` działa na PIHA (healthcheck zielony). +2. ✅ Export NFS na PIHA (wyżej) + `showmount -e 192.168.31.5` z SOLARII. +3. ✅ `nfs-common` na SOLARII. +4. ✅ `.env` z `env.example` — sekrety SKOPIOWANE z PIHA, nie nowe. +5. ✅ `docker compose up -d` + `./healthcheck.sh`. +6. ✅ Test (2026-07-12): PDF-y wrzucone do consume na PIHA, część odebrana i + dokończona przez worker@SOLARIA (dowód w logach, zero File not found). +7. ⬜ Test fallbacku: stop workera → zadanie czeka/mieli PIHA → start → drenaż + (jeszcze niewykonany formalnie, ale mechanizm nie zmienił się tym fixem — + fallback na PIHA działał już wcześniej, patrz sekcja "Fallback" wyżej). +8. ⬜ **OTWARTE**: wpis `paperless-worker` w `hosts/solaria/services.yaml` + + `inventory/topology.yaml` (obecnie SOLARIA ma tam tylko `node-agent`) — + bez tego supervisor/observer nie widzą tego serwisu w desired-state, więc + drift między `hosts/solaria/services.yaml` a rzeczywistością nie jest + wykrywany. Patrz `docs/backlog.md`. diff --git a/kb/runbooks/planner-agent-deploy.md b/kb/runbooks/planner-agent-deploy.md new file mode 100644 index 0000000..10966bf --- /dev/null +++ b/kb/runbooks/planner-agent-deploy.md @@ -0,0 +1,105 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-05-27 +links: + - ../services/planner-agent.md +--- + +# planner-agent — deploy, uruchamianie, testy + +## Deployment na SOLARIA + +```bash +# 1. Przygotuj .env na solaria +ssh oskar@100.100.231.104 +mkdir -p /opt/homelab/config/planner-agent +cat > /opt/homelab/config/planner-agent/.env << 'EOF' +OLLAMA_HOST=http://host-gateway:11434 +OLLAMA_MODEL=qwen2.5-coder:14b +REDIS_URL=redis://100.108.208.3:6379 +NODE_NAME=solaria +COOLDOWN_SECONDS=300 +RUNTIME_PATH=/opt/homelab +EOF + +# 2. Deploy przez standardowy skrypt (z SATURN) +scripts/deploy/deploy.sh --service planner-agent + +# 3. Weryfikacja +docker ps --filter name=planner-agent +docker logs planner-agent --tail 30 +ls -la /opt/homelab/state/planner-agent.heartbeat +``` + +> **Uwaga:** stage `verify` w deploy.sh może failować bezpośrednio po starcie +> (race condition — heartbeat jest pisany co ~5 s). Kontener jest zdrowy gdy +> `docker ps` pokazuje `(healthy)`. + +--- + +## Uruchamianie lokalne (bez Dockera) + +```bash +cd services/planner-agent + +# Zainstaluj zależności +pip install -r requirements.txt + +# Wyeksportuj zmienne (lub stwórz .env i użyj `export $(cat .env | xargs)`) +export REDIS_URL=redis://localhost:6379 +export OLLAMA_HOST=http://localhost:11434 +export OLLAMA_MODEL=qwen2.5:7b +export NODE_NAME=dev-local +export RUNTIME_PATH=/tmp/homelab-dev +export COOLDOWN_SECONDS=10 + +# Uruchom +python src/planner.py +``` + +--- + +## Testowanie + +### Testy jednostkowe + +```bash +cd services/planner-agent +pip install -r requirements.txt pytest pytest-asyncio +pytest tests/ -v +# 49 testów planner + 34 testy llm_router +``` + +### Ręczny end-to-end test przez Redis + +```bash +# Opublikuj sztuczne zdarzenie unhealthy na kanale health_events +redis-cli -h 100.108.208.3 PUBLISH health_events '{ + "type": "service_unhealthy", + "node": "piha", + "service": "mosquitto", + "severity": "error", + "payload": {"exit_code": 1, "reason": "OOMKilled"}, + "timestamp": "2026-05-27T20:00:00Z" +}' + +# Obserwuj logi planera +docker logs planner-agent -f + +# Sprawdź czy propozycja trafiła do pending +ls /opt/homelab/actions/pending/ +cat /opt/homelab/actions/pending/plan-piha-mosquitto-*.json +``` + +### Śledzenie metryk LLM + +```bash +# Subskrybuj metryki routera w czasie rzeczywistym +redis-cli -h 100.108.208.3 SUBSCRIBE llm_router_metrics +``` + +--- + diff --git a/services/ha-diag-agent/README.md b/kb/services/ha-diag-agent.md similarity index 68% rename from services/ha-diag-agent/README.md rename to kb/services/ha-diag-agent.md index 9c61877..e0d24e1 100644 --- a/services/ha-diag-agent/README.md +++ b/kb/services/ha-diag-agent.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-06-11 +links: + - ../runbooks/ha-diag-agent-runbook.md +--- + # ha-diag-agent Per-host Home Assistant diagnostic agent. Polls HA REST API on a schedule, @@ -52,17 +62,6 @@ checks are APScheduler intervals (stateless REST polls). Event routing in supervisor (Phase 5) maps these to `notify` actions. `ha_websocket_recovered` should be routed to clear any active `ha_websocket_dead` incident. -## First-time deployment - -See **[DEPLOY.md](DEPLOY.md)** for the full procedure: HA token creation, -per-host `.env` config, deploy commands, verification steps, 48h shadow-mode -observation, and rollback. - -**Shadow mode** (`HA_DIAG_SHADOW_MODE`, default `true` on the control-plane): -`ha_websocket_dead` events are downgraded to `alert_only` with a `[SHADOW MODE]` -note instead of queuing an automatic `container_restart`. Set to `false` in -`/opt/homelab/config/control-plane/.env` on the VPS when ready for live actions. - ## Deployment model The agent is deployed **per-host** but targets a potentially remote HA instance: @@ -78,47 +77,6 @@ OS VM. `chelsty-infra` is the hypervisor but does not run HA itself. The agent o gets a new Tailscale IP, update `HA_URL` in `/opt/homelab/config/ha-diag-agent/.env` on `chelsty-infra`. -## Deployment - -```bash -# 1. Create config on target node -ssh oskar@ -mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent -cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF' -HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra -HA_TOKEN= -NODE_NAME=piha # or chelsty-infra -LOCATION_TAG=ken # or chelsty -CHECK_INTERVAL=60 -EOF - -# 2. Deploy -scripts/deploy/deploy.sh --service ha-diag-agent - -# 3. Verify -docker ps --filter name=ha-diag-agent -docker exec ha-diag-agent python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8087/health', timeout=5).read().decode())" -``` - -### chelsty-infra note - -`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated): -```bash -docker-compose -f docker-compose.yml up -d --build -``` - -### HA long-lived token - -In HA UI: Profile → Long-Lived Access Tokens → Create token. - -## Running Tests - -```bash -cd services/ha-diag-agent -pip install -e ".[dev]" -pytest tests/ -v -``` - ## Optional YAML config Place `/opt/homelab/config/ha-diag-agent/ha-diag-agent.yaml` on the node. diff --git a/services/ha-mcp/README.md b/kb/services/ha-mcp.md similarity index 70% rename from services/ha-mcp/README.md rename to kb/services/ha-mcp.md index 4ee8282..8f0dab0 100644 --- a/services/ha-mcp/README.md +++ b/kb/services/ha-mcp.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-30 +links: + - ../runbooks/ha-mcp-install.md +--- + # ha-mcp — read-only MCP server for Home Assistant **Status: phase 2a** of `services/home-assistant/DESIGN.md` — own minimal MCP @@ -32,74 +42,6 @@ verify). See `services/home-assistant/DESIGN.md`, "Sync model" and "Validation gate". Nothing in this server bypasses that, and nothing in this server should ever learn to. -## Install - -The `mcp` SDK (and its pydantic/anyio/httpx dependency tree) is not packaged -for Debian and is not needed by anything else in this repo, so it goes into a -venv rather than into the system Python: - -```bash -python3 -m venv --system-site-packages services/ha-mcp/.venv -services/ha-mcp/.venv/bin/pip install mcp pytest -``` - -`--system-site-packages` is deliberate: `requests`, `PyYAML` and -`websocket-client` are already installed system-wide and used by -`scripts/ha/lib/*`; the venv reuses those exact versions instead of shadowing -them with a second copy. `.venv/` is already covered by the repo `.gitignore`. - -`pip install --break-system-packages mcp` also works and is one line shorter, -but it writes into the system interpreter that runs every deploy script on -this workstation — a venv keeps a 40-package dependency tree out of that blast -radius for a tool only Claude Code uses. Use the venv. - -`run.sh` prefers `services/ha-mcp/.venv/bin/python` and falls back to the -system `python3`; if neither can import `mcp` it exits with one actionable -line on stderr rather than a traceback. - -## Registration in Claude Code - -`.mcp.json` in the repo root (project scope — shared with anyone who checks -out this repo): - -```json -{ - "mcpServers": { - "ha": { - "command": "./services/ha-mcp/run.sh", - "args": [], - "env": {} - } - } -} -``` - -The command path is relative, so it resolves in any checkout (main or task -worktree) as long as Claude Code is started from the repo root. Start CC -there; on first run it asks whether to trust the project's MCP servers. Check -with `/mcp` — the server appears as `ha`, its tools as `mcp__ha__`. -Which repo the server reads is derived from its own location; override with -the `HA_MCP_REPO` environment variable if you ever need to point one checkout -at another's config. - -## Standalone - -```bash -./services/ha-mcp/run.sh # stdio server — speaks JSON-RPC on stdout -services/ha-mcp/tests/run.sh # offline test suite (no HA, no token) -services/ha-mcp/.venv/bin/python services/ha-mcp/tests/smoke_live.py [instance] [entity_id] -``` - -`smoke_live.py` is the only thing here that touches the network (GET only). A -one-off tool call without a client is easiest through the package: - -```bash -PYTHONPATH=services/ha-mcp/src services/ha-mcp/.venv/bin/python -c " -from ha_mcp import tools; from ha_mcp.backend import LiveBackend -from ha_mcp.config import get_instance, repo_root -n, c = get_instance('ken', repo_root()); print(tools.instance_status(LiveBackend(n, c, repo_root())))" -``` - ## Tools All seven are read-only, take an optional `instance` (default `ken` — the @@ -231,23 +173,6 @@ a log line, in a tool result, or in this repo. A missing token is a reported tool error naming the path it looked at — not a crash, and not a silent empty result. -## Tests - -```bash -services/ha-mcp/tests/run.sh # 42 tests, offline -``` - -Offline in the same sense as `scripts/ha/tests/*`: no network, no HA -instance, no token. Small hand-made fixtures under `tests/fixtures/` (shaped -like real `/api/states` and WebSocket registry payloads) cover the tool logic; -the repo's own `config/ken/automations/` and newest `fixtures/ken-states-*.yaml` -cover the repo-backed and at-scale paths (ranking over ~1650 real entities -behaves differently from ranking over ten). - -Live smoke (read-only, run against `ken` on 2026-07-30): -`instance_status` → HA 2026.7.2, 1647 entities, 377 unavailable, 115 -automations, 13 areas; `get_state("sensor.thsalon_temperature")` → `24.5 °C`. - ## Not a deployed service No `docker-compose.yml`, no `service.yaml`, no `healthcheck.sh`: this is diff --git a/kb/services/home-assistant.md b/kb/services/home-assistant.md new file mode 100644 index 0000000..56f3d92 --- /dev/null +++ b/kb/services/home-assistant.md @@ -0,0 +1,37 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-22 +links: + - ../runbooks/home-assistant-deploy.md +--- + +# home-assistant (configs-as-code) + +**Status: phase 1 (partial).** Read-only import tooling, plus a deploy +(repo -> instance) write path for the `api` adapter's automations/scripts/ +scenes scope only (`scripts/ha/deploy.sh`) — see "Deploy" below. Dashboards, +helpers, and the `docker-exec` adapter have no write path yet. See +`DESIGN.md` for the full phasing, adapter, sync, and validation model, and +for the open questions still blocking phase 2/3. + +## Layout + +``` +services/home-assistant/ +├── DESIGN.md # decision registry — read this first +├── instances.yaml # per-instance adapter/host/token config +├── .gitignore # excludes secrets/db/log/token paths from every import +├── config// # canonical, normalized /config mirror per instance +├── storage-export// # curated .storage/* export (registries, dashboards) +└── fixtures/ # dated /api/states snapshots +``` + +Instances: `ken` (RPi4/HAOS, LAN 192.168.31.7, api adapter — canonical home +instance since the 2026-07-22 cutover), `ken-legacy` (PIHA, container +`homeassistant5`, docker-exec adapter, archived — pre-migration instance, +import only, never deploy), `chelsty-ha` (Tailscale, api adapter — see +`instances.yaml` and `DESIGN.md` "Incident log"). + diff --git a/jobs/gmail-header-backfill/README.md b/kb/services/job-gmail-header-backfill.md similarity index 83% rename from jobs/gmail-header-backfill/README.md rename to kb/services/job-gmail-header-backfill.md index 04301d5..9870d82 100644 --- a/jobs/gmail-header-backfill/README.md +++ b/kb/services/job-gmail-header-backfill.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-14 +links: + - ../runbooks/gmail-header-backfill-run.md +--- + # gmail-header-backfill One-shot job, module 5 phase 2 backfill (`docs/kb/modules/05-faza2-plan.md`, §5). @@ -27,26 +37,6 @@ Install (from repo root, on PIHA): pip install -e jobs/gmail-header-backfill/ ``` -## Usage - -```bash -# Dry run (default) — parse and count only, no DB writes: -gmail-header-backfill --dsn postgresql://kb:@localhost:5433/kb --limit 100 - -# Real run — apply the UPDATE for this slice: -gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply - -# Next slice — offset is stable/deterministic (ORDER BY id), independent of -# how many rows in earlier slices were already backfilled: -gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply -``` - -DSN can also come from the `KB_DSN` env var instead of `--dsn`. - -`--limit`/`--offset` exist so the full 225 030-row backfill can be run in -verifiable partitions instead of one long unattended run (plan §5.3) — start -small (`--limit 100`), check the result in the DB, then widen. - ## Source of headers: the archived `.eml`, not the original mbox Same access pattern as `documents-ingest`: `archive_root / row["raw_ref"]` -> @@ -171,23 +161,6 @@ of streaming one mbox — the plan's estimate is "same order of magnitude, likely tens of minutes." Not measured directly in this change — see the DoD note below. -## Tests - -```bash -pip install -e jobs/gmail-header-backfill/ -cd jobs/gmail-header-backfill && pytest -``` - -Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml` -bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an -in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`, -quoted display names with commas, multiple `Delivered-To` occurrences, RFC -2047 encoded-words including Polish diacritics, malformed encoded-words that -must not raise, missing/multiple `From`, `date_raw` preserving literal text -vs. `Date` header reformatting), idempotency (rows already carrying a -`headers` entity are skipped and never re-appended), batch flushing, and -`--limit`/`--offset` query shape. - ## Definition of Done Per `CLAUDE.md`: this job's smoke run is `gmail-header-backfill --dsn ... diff --git a/jobs/mail-body-ingest/README.md b/kb/services/job-mail-body-ingest.md similarity index 80% rename from jobs/mail-body-ingest/README.md rename to kb/services/job-mail-body-ingest.md index 99fc1ce..757499e 100644 --- a/jobs/mail-body-ingest/README.md +++ b/kb/services/job-mail-body-ingest.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-22 +links: + - ../runbooks/mail-body-ingest-run.md +--- + # mail-body-ingest Module 5, faza mailowa (`docs/kb/modules/05-faza-mailowa-plan.md`, §5, Krok 2). Second full @@ -28,23 +38,6 @@ pip install -e packages/kb-retrieval/ pip install -e jobs/mail-body-ingest/ ``` -## Usage - -```bash -# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero -# DB writes (including the threading UPDATE): -mail-body-ingest --dsn postgresql://kb:@piha:5433/kb --archive-root /home/oskar/kb/mail/archive - -# Etap A pilot — last 12 months only (plan Decyzja 9): -mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1 - -# Smoke-test slice: -mail-body-ingest --dsn ... --apply --limit 10 -``` - -DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default -`http://localhost:11434` — this job is meant to run where Ollama lives). - ## Pipeline (per envelope) 1. **Read** `archive_root / raw_ref` — `missing_file`/`read_error` counted like @@ -107,21 +100,6 @@ fetch is already scoped to it. Built this way from the start per the plan's note models (not an active bug there today, since one run always uses one model, but worth not repeating the ambiguity here). -## Tests - -```bash -pip install -e "jobs/mail-body-ingest[dev]" -cd jobs/mail-body-ingest && pytest -``` - -Pure unit tests (48), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect` -and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers: -quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/ -gmail_quote skipping), newsletter classification, threading extraction, prefix building, -body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse -fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks -never reaching Ollama, Ollama-offline batch isolation, and dimension-mismatch abort. - ## Definition of Done Per `CLAUDE.md`: `pytest` passes (48/48) + a `--limit 5` dry-run smoke against live diff --git a/services/kb-query/README.md b/kb/services/kb-query.md similarity index 79% rename from services/kb-query/README.md rename to kb/services/kb-query.md index 9e6d02e..0df1fee 100644 --- a/services/kb-query/README.md +++ b/kb/services/kb-query.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-29 +links: + - ../runbooks/kb-query-deploy.md +--- + # kb-query FastAPI search API in front of the module-5 KB retrieval engine @@ -120,55 +130,6 @@ that *wrote* the summary, e.g. `claude-haiku-4-5`, not the embedder). (`claude-haiku-4-5`). `OLLAMA_URL` was **renamed** to `EMBED_PRIMARY_URL` in Krok 2 — if an old `.env` sets `OLLAMA_URL`, it is ignored. -## Deploy (PIHA) - -0. Prerequisite: `ollama-piha` deployed and `bge-m3` pulled — see - `services/ollama-piha/README.md` (the pull is a **manual** deploy step). -1. `git pull` on PIHA (`~/homelab-codex-ws`). -2. `cp services/kb-query/env.example services/kb-query/.env` and fill in the - real `KB_DSN` password (the template already sets `EMBED_FALLBACK_URL`). - On an existing install: add `EMBED_FALLBACK_URL=http://192.168.31.5:11434` - to the existing `.env`. -3. ``` - docker compose -f services/kb-query/docker-compose.yml \ - -f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build - ``` -4. Verify: `services/kb-query/healthcheck.sh`, then from PIHA: - `curl "http://192.168.31.5:8230/search?q=test"` and open - `http://192.168.31.5:8230/` in a browser. - -## Fallback verification (execution: operator, after deploy) - -- **Test A — SOLARIA online**: query via UI/`curl`; response has - `"embed_backend": "solaria"`, `docker logs kb-query` shows - `backend=solaria`, latency ~sub-second. -- **Test B — SOLARIA offline**: either wait for the nightly power-off, or - simulate: set `EMBED_PRIMARY_URL=http://192.0.2.1:11434` (TEST-NET, always - unreachable) in `.env` and `docker compose … up -d` again. Query still - works; response has `"embed_backend": "piha"`, log shows `backend=piha` - plus a `circuit open for 30s` warning on the first hit; latency visibly - higher (CPU + cold model load each time, `OLLAMA_KEEP_ALIVE=0`). Revert - `.env` afterwards if simulated. -- **Test C — SOLARIA returns**: after it is back up, within ≤30 s (one - health-cache TTL) responses show `"embed_backend": "solaria"` again, no - restart needed. - -## Tests - -``` -pip install -e packages/kb-retrieval/ -cd services/kb-query && pip install -r requirements.txt pytest pytest-asyncio && pytest -``` - -Unit tests mock the DB connection and Ollama HTTP session (no live DB/Ollama -required) — same style as `packages/kb-retrieval/tests/`. `tests/test_frontend.py` -drives `GET /`/`/static/*` through FastAPI's `TestClient` without entering it -as a context manager, so the DB-requiring `lifespan` never runs. - -Frontend JS has its own pure-function tests (query-URL encoding, threshold -colouring, envelope grouping), run without a browser via Node's built-in -test runner: `node --test services/kb-query/tests/frontend/`. - ## Ingress (`kb.kapala.org`, plan §8) Wired up 2026-07-23 (`docs/sessions/2026-07-23-kb-f4-ingress.md`), **no code diff --git a/kb/services/narty27.md b/kb/services/narty27.md new file mode 100644 index 0000000..9e8565c --- /dev/null +++ b/kb/services/narty27.md @@ -0,0 +1,35 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-31 +links: + - ../runbooks/narty27-deploy.md +--- + +# narty27 + +Static hosting for a single self-contained `viz.html` (narty 2027 / Saalbach KB +export) on PIHA. Plain `nginx:alpine` serving one Docker named volume — no build, +no database, no dependencies. + +- URL: `http://192.168.31.5:8240/viz.html` (and `/` — same file, see below) +- Exposure: private (LAN/Tailscale only; no npm vhost, no public ingress) +- Volume: `narty27_narty27_content` → `/usr/share/nginx/html:ro` + +## Content is personal and lives outside the repo + +The visualisation is personal content. It is **never committed** — not to this +repo, not to any other. It exists in exactly two places: + +1. the source on SOLARIA (`~/narty-2027/saalbach-kb/viz.html`), and +2. the `narty27_narty27_content` Docker volume on PIHA. + +There is no backup job and no bind mount under `/opt/homelab/data/`. If the +volume is lost, re-run the update procedure below from SOLARIA. + +Two copies of the same file are stored in the volume: `viz.html` (canonical name) +and `index.html` (so the bare root `http://192.168.31.5:8240/` works without a +path). Both must be refreshed together on every update. + diff --git a/services/ollama-piha/README.md b/kb/services/ollama-piha.md similarity index 51% rename from services/ollama-piha/README.md rename to kb/services/ollama-piha.md index 6931f12..6f8b73e 100644 --- a/services/ollama-piha/README.md +++ b/kb/services/ollama-piha.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-30 +links: + - ../runbooks/ollama-piha-deploy.md +--- + # ollama-piha Local CPU Ollama on **PIHA**, serving exactly one purpose: the **fallback embed @@ -29,57 +39,6 @@ model is never resident — see below). Slower but alive beats fast but dead. PIHA's uid pattern (host oskar=1004, containers uid 1000, setgid group pi) if it wrote to a shared bind directory. -## Deploy (PIHA, master, after merge) - -```bash -cd ~/homelab-codex-ws && git pull -cp services/ollama-piha/env.example services/ollama-piha/.env # LAN_BIND_IP -docker compose -f services/ollama-piha/docker-compose.yml \ - -f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \ - --env-file services/ollama-piha/.env up -d -``` - -**Then pull the model — this does NOT happen automatically:** - -```bash -docker exec ollama-piha ollama pull bge-m3 -``` - -Verify: - -```bash -services/ollama-piha/healthcheck.sh # checks container + API + bge-m3 present -time curl -s http://127.0.0.1:11434/api/embeddings \ - -d '{"model":"bge-m3","prompt":"test kalibracyjny"}' | head -c 80 -``` - -(`deploy-node.sh` on PIHA also picks this service up from -`hosts/piha/services.yaml` once `.env` exists — the `ollama pull bge-m3` step -stays manual either way.) - -## Calibration (plan §5 step 4 — gate, not formality) - -Before trusting the fallback under load, on live PIHA at a normal (not -night-quiet) hour: run a few embeds as above while watching -`docker stats ollama-piha`, note peak RAM and wall time. Verdict per plan §5 -step 5: keep as default fallback / tune `mem_limit` / fall back to explicit -503 degradation. - -**Calibration status: measured 2026-07-27 — verdict GO** (live PIHA under -normal load, 3 consecutive `/api/embeddings` calls after `ollama pull bge-m3`; -full protocol in `docs/sessions/2026-07-27-kb-f4-fallback.md` §4): - -- Latency: 5.25 s (cold start) / 4.41 s / 4.16 s — single seconds as expected, - no warm-up between calls by design (`OLLAMA_KEEP_ALIVE=0` releases the model - after every request, `ollama ps` shows nothing resident in between). -- RAM: idle ~66 MiB, burst peak ~983 MiB (`docker stats` sampled at 0.3 s) — - well inside the 2560m ceiling; host `available` never dropped below ~1.3 GiB. -- Kept as **default fallback** (no feature flag). The measurement was taken - against the same container configuration this repo deploys (image, - `OLLAMA_KEEP_ALIVE=0`, `mem_limit: 2560m`), so it carries over; only the - model storage differed (bind mount then, named volume now), which does not - affect RAM/latency. - ## Relation to kb-query kb-query's router (`services/kb-query/app/embed_router.py`) health-checks diff --git a/services/paperless-worker/README.md b/kb/services/paperless-worker.md similarity index 73% rename from services/paperless-worker/README.md rename to kb/services/paperless-worker.md index f03803b..f0db938 100644 --- a/services/paperless-worker/README.md +++ b/kb/services/paperless-worker.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-07-12 +links: + - ../runbooks/paperless-worker-deploy.md +--- + # Paperless OCR worker (SOLARIA) Ciężki OCR filaru KB #2 (moduł 3, `docs/kb/modules/03-paperless-ocr-worker.md`). @@ -108,58 +118,3 @@ miała własny, lokalny, pusty `/tmp/paperless`. Fix: NFS-owy wolumen katalog `/opt/homelab/data/paperless/scratch` (już istnieje na PIHA, `chown 1000:1000`), montowany na `/tmp/paperless` po obu stronach. -## NFS: export na PIHA, mount na SOLARIA - -Transfer idzie po **LAN** (PIHA `192.168.31.5` ↔ SOLARIA `192.168.31.70`, -1 Gb/s, ten sam switch) — NIE po Tailscale. Przepustowość nie jest wąskim -gardłem OCR. - -### Host-side na PIHA (NIE w compose — krok przy deployu modułu 3) - -``` -# /etc/exports na PIHA — export TYLKO dla SOLARII: -/opt/homelab/data/paperless 192.168.31.70(rw,sync,no_subtree_check,no_root_squash) -``` - -```bash -sudo apt install nfs-kernel-server # jesli brak -sudo exportfs -ra -``` - -`no_root_squash` jest potrzebne, bo entrypoint obrazu (root) robi `chown` na -katalogach przy starcie kontenera na SOLARII; export jest ograniczony do -jednego IP w zaufanym LAN. - -### Strona SOLARII - -Mounty definiuje compose jako named volumes z driverem NFS — **zero wpisów -w /etc/fstab**; jedyny host-side wymóg to pakiet klienta: - -```bash -sudo apt install nfs-common -``` - -### UID mapping (krytyczne) - -Pliki na exporcie mają numerycznego właściciela — NFS nie tłumaczy nazw. -Dlatego `USERMAP_UID/GID=1000` jest ustawione **w obu** compose (PIHA -i SOLARIA); zmiana po jednej stronie = worker traci dostęp do plików. -Weryfikacja po deployu: `./healthcheck.sh` robi test zapisu na mount. - -## Cutover checklist (przy deployu modułu 3 — po działającym module 2) - -1. ✅ `services/paperless/` działa na PIHA (healthcheck zielony). -2. ✅ Export NFS na PIHA (wyżej) + `showmount -e 192.168.31.5` z SOLARII. -3. ✅ `nfs-common` na SOLARII. -4. ✅ `.env` z `env.example` — sekrety SKOPIOWANE z PIHA, nie nowe. -5. ✅ `docker compose up -d` + `./healthcheck.sh`. -6. ✅ Test (2026-07-12): PDF-y wrzucone do consume na PIHA, część odebrana i - dokończona przez worker@SOLARIA (dowód w logach, zero File not found). -7. ⬜ Test fallbacku: stop workera → zadanie czeka/mieli PIHA → start → drenaż - (jeszcze niewykonany formalnie, ale mechanizm nie zmienił się tym fixem — - fallback na PIHA działał już wcześniej, patrz sekcja "Fallback" wyżej). -8. ⬜ **OTWARTE**: wpis `paperless-worker` w `hosts/solaria/services.yaml` + - `inventory/topology.yaml` (obecnie SOLARIA ma tam tylko `node-agent`) — - bez tego supervisor/observer nie widzą tego serwisu w desired-state, więc - drift między `hosts/solaria/services.yaml` a rzeczywistością nie jest - wykrywany. Patrz `docs/backlog.md`. diff --git a/services/planner-agent/README.md b/kb/services/planner-agent.md similarity index 67% rename from services/planner-agent/README.md rename to kb/services/planner-agent.md index ac7f19d..507351e 100644 --- a/services/planner-agent/README.md +++ b/kb/services/planner-agent.md @@ -1,3 +1,13 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-05-27 +links: + - ../runbooks/planner-agent-deploy.md +--- + # planner-agent Asynchroniczny agent diagnozujący zdarzenia zdrowotne w homelabowej infrastrukturze. @@ -110,99 +120,6 @@ w docker-compose.yml (nie jest w env_file — sekret injektowany przez operatora --- -## Deployment na SOLARIA - -```bash -# 1. Przygotuj .env na solaria -ssh oskar@100.100.231.104 -mkdir -p /opt/homelab/config/planner-agent -cat > /opt/homelab/config/planner-agent/.env << 'EOF' -OLLAMA_HOST=http://host-gateway:11434 -OLLAMA_MODEL=qwen2.5-coder:14b -REDIS_URL=redis://100.108.208.3:6379 -NODE_NAME=solaria -COOLDOWN_SECONDS=300 -RUNTIME_PATH=/opt/homelab -EOF - -# 2. Deploy przez standardowy skrypt (z SATURN) -scripts/deploy/deploy.sh --service planner-agent - -# 3. Weryfikacja -docker ps --filter name=planner-agent -docker logs planner-agent --tail 30 -ls -la /opt/homelab/state/planner-agent.heartbeat -``` - -> **Uwaga:** stage `verify` w deploy.sh może failować bezpośrednio po starcie -> (race condition — heartbeat jest pisany co ~5 s). Kontener jest zdrowy gdy -> `docker ps` pokazuje `(healthy)`. - ---- - -## Uruchamianie lokalne (bez Dockera) - -```bash -cd services/planner-agent - -# Zainstaluj zależności -pip install -r requirements.txt - -# Wyeksportuj zmienne (lub stwórz .env i użyj `export $(cat .env | xargs)`) -export REDIS_URL=redis://localhost:6379 -export OLLAMA_HOST=http://localhost:11434 -export OLLAMA_MODEL=qwen2.5:7b -export NODE_NAME=dev-local -export RUNTIME_PATH=/tmp/homelab-dev -export COOLDOWN_SECONDS=10 - -# Uruchom -python src/planner.py -``` - ---- - -## Testowanie - -### Testy jednostkowe - -```bash -cd services/planner-agent -pip install -r requirements.txt pytest pytest-asyncio -pytest tests/ -v -# 49 testów planner + 34 testy llm_router -``` - -### Ręczny end-to-end test przez Redis - -```bash -# Opublikuj sztuczne zdarzenie unhealthy na kanale health_events -redis-cli -h 100.108.208.3 PUBLISH health_events '{ - "type": "service_unhealthy", - "node": "piha", - "service": "mosquitto", - "severity": "error", - "payload": {"exit_code": 1, "reason": "OOMKilled"}, - "timestamp": "2026-05-27T20:00:00Z" -}' - -# Obserwuj logi planera -docker logs planner-agent -f - -# Sprawdź czy propozycja trafiła do pending -ls /opt/homelab/actions/pending/ -cat /opt/homelab/actions/pending/plan-piha-mosquitto-*.json -``` - -### Śledzenie metryk LLM - -```bash -# Subskrybuj metryki routera w czasie rzeczywistym -redis-cli -h 100.108.208.3 SUBSCRIBE llm_router_metrics -``` - ---- - ## Healthcheck Skrypt `healthcheck.sh` sprawdza czy plik heartbeat