feat(kb): SPLIT service+runbook — 10 serwisow -> 20 dokumentow
Wzorzec mechaniczny: sekcje deploy/verify/install/testy wycinane do kb/runbooks/<serwis>-*.md, reszta zostaje dokumentem type: service. Wzajemne `links` w obie strony. Tresc sekcji nietknieta — przenoszone doslownie, dodany wylacznie naglowek H1 nowego runbooka. kb-query, paperless-worker, planner-agent, ha-diag-agent, ollama-piha, narty27, home-assistant, ha-mcp, job-gmail-header-backfill, job-mail-body-ingest. Weryfikacja: dla kazdego pliku multizbior niepustych linii (main + runbook) == oryginal z HEAD. Zero zgubionych, zero dodanych. Recon szacowal 13 splitow service+runbook; faktycznie 2-typowych jest 10, pozostale 5 (paperless, nextcloud, gokapi, fleet-prometheus, deploy-runner) sa 3-typowe i ida osobno jako splity wielotypowe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
oskar
parent
246101e8c5
commit
0b18d2fc3b
49
kb/runbooks/gmail-header-backfill-run.md
Normal file
49
kb/runbooks/gmail-header-backfill-run.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-14
|
||||
links:
|
||||
- ../services/job-gmail-header-backfill.md
|
||||
---
|
||||
|
||||
# gmail-header-backfill — uruchomienie i testy
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Dry run (default) — parse and count only, no DB writes:
|
||||
gmail-header-backfill --dsn postgresql://kb:<pw>@localhost:5433/kb --limit 100
|
||||
|
||||
# Real run — apply the UPDATE for this slice:
|
||||
gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply
|
||||
|
||||
# Next slice — offset is stable/deterministic (ORDER BY id), independent of
|
||||
# how many rows in earlier slices were already backfilled:
|
||||
gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply
|
||||
```
|
||||
|
||||
DSN can also come from the `KB_DSN` env var instead of `--dsn`.
|
||||
|
||||
`--limit`/`--offset` exist so the full 225 030-row backfill can be run in
|
||||
verifiable partitions instead of one long unattended run (plan §5.3) — start
|
||||
small (`--limit 100`), check the result in the DB, then widen.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e jobs/gmail-header-backfill/
|
||||
cd jobs/gmail-header-backfill && pytest
|
||||
```
|
||||
|
||||
Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml`
|
||||
bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an
|
||||
in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`,
|
||||
quoted display names with commas, multiple `Delivered-To` occurrences, RFC
|
||||
2047 encoded-words including Polish diacritics, malformed encoded-words that
|
||||
must not raise, missing/multiple `From`, `date_raw` preserving literal text
|
||||
vs. `Date` header reformatting), idempotency (rows already carrying a
|
||||
`headers` entity are skipped and never re-appended), batch flushing, and
|
||||
`--limit`/`--offset` query shape.
|
||||
|
||||
64
kb/runbooks/ha-diag-agent-runbook.md
Normal file
64
kb/runbooks/ha-diag-agent-runbook.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-06-11
|
||||
links:
|
||||
- ../services/ha-diag-agent.md
|
||||
---
|
||||
|
||||
# ha-diag-agent — deployment i testy
|
||||
|
||||
## First-time deployment
|
||||
|
||||
See **[DEPLOY.md](DEPLOY.md)** for the full procedure: HA token creation,
|
||||
per-host `.env` config, deploy commands, verification steps, 48h shadow-mode
|
||||
observation, and rollback.
|
||||
|
||||
**Shadow mode** (`HA_DIAG_SHADOW_MODE`, default `true` on the control-plane):
|
||||
`ha_websocket_dead` events are downgraded to `alert_only` with a `[SHADOW MODE]`
|
||||
note instead of queuing an automatic `container_restart`. Set to `false` in
|
||||
`/opt/homelab/config/control-plane/.env` on the VPS when ready for live actions.
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
# 1. Create config on target node
|
||||
ssh oskar@<node-ip>
|
||||
mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent
|
||||
cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF'
|
||||
HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra
|
||||
HA_TOKEN=<long-lived-token>
|
||||
NODE_NAME=piha # or chelsty-infra
|
||||
LOCATION_TAG=ken # or chelsty
|
||||
CHECK_INTERVAL=60
|
||||
EOF
|
||||
|
||||
# 2. Deploy
|
||||
scripts/deploy/deploy.sh --service ha-diag-agent
|
||||
|
||||
# 3. Verify
|
||||
docker ps --filter name=ha-diag-agent
|
||||
docker exec ha-diag-agent python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8087/health', timeout=5).read().decode())"
|
||||
```
|
||||
|
||||
### chelsty-infra note
|
||||
|
||||
`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated):
|
||||
```bash
|
||||
docker-compose -f docker-compose.yml up -d --build
|
||||
```
|
||||
|
||||
### HA long-lived token
|
||||
|
||||
In HA UI: Profile → Long-Lived Access Tokens → Create token.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
cd services/ha-diag-agent
|
||||
pip install -e ".[dev]"
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
97
kb/runbooks/ha-mcp-install.md
Normal file
97
kb/runbooks/ha-mcp-install.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-30
|
||||
links:
|
||||
- ../services/ha-mcp.md
|
||||
---
|
||||
|
||||
# ha-mcp — instalacja i rejestracja
|
||||
|
||||
## Install
|
||||
|
||||
The `mcp` SDK (and its pydantic/anyio/httpx dependency tree) is not packaged
|
||||
for Debian and is not needed by anything else in this repo, so it goes into a
|
||||
venv rather than into the system Python:
|
||||
|
||||
```bash
|
||||
python3 -m venv --system-site-packages services/ha-mcp/.venv
|
||||
services/ha-mcp/.venv/bin/pip install mcp pytest
|
||||
```
|
||||
|
||||
`--system-site-packages` is deliberate: `requests`, `PyYAML` and
|
||||
`websocket-client` are already installed system-wide and used by
|
||||
`scripts/ha/lib/*`; the venv reuses those exact versions instead of shadowing
|
||||
them with a second copy. `.venv/` is already covered by the repo `.gitignore`.
|
||||
|
||||
`pip install --break-system-packages mcp` also works and is one line shorter,
|
||||
but it writes into the system interpreter that runs every deploy script on
|
||||
this workstation — a venv keeps a 40-package dependency tree out of that blast
|
||||
radius for a tool only Claude Code uses. Use the venv.
|
||||
|
||||
`run.sh` prefers `services/ha-mcp/.venv/bin/python` and falls back to the
|
||||
system `python3`; if neither can import `mcp` it exits with one actionable
|
||||
line on stderr rather than a traceback.
|
||||
|
||||
## Registration in Claude Code
|
||||
|
||||
`.mcp.json` in the repo root (project scope — shared with anyone who checks
|
||||
out this repo):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"ha": {
|
||||
"command": "./services/ha-mcp/run.sh",
|
||||
"args": [],
|
||||
"env": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The command path is relative, so it resolves in any checkout (main or task
|
||||
worktree) as long as Claude Code is started from the repo root. Start CC
|
||||
there; on first run it asks whether to trust the project's MCP servers. Check
|
||||
with `/mcp` — the server appears as `ha`, its tools as `mcp__ha__<tool>`.
|
||||
Which repo the server reads is derived from its own location; override with
|
||||
the `HA_MCP_REPO` environment variable if you ever need to point one checkout
|
||||
at another's config.
|
||||
|
||||
## Standalone
|
||||
|
||||
```bash
|
||||
./services/ha-mcp/run.sh # stdio server — speaks JSON-RPC on stdout
|
||||
services/ha-mcp/tests/run.sh # offline test suite (no HA, no token)
|
||||
services/ha-mcp/.venv/bin/python services/ha-mcp/tests/smoke_live.py [instance] [entity_id]
|
||||
```
|
||||
|
||||
`smoke_live.py` is the only thing here that touches the network (GET only). A
|
||||
one-off tool call without a client is easiest through the package:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=services/ha-mcp/src services/ha-mcp/.venv/bin/python -c "
|
||||
from ha_mcp import tools; from ha_mcp.backend import LiveBackend
|
||||
from ha_mcp.config import get_instance, repo_root
|
||||
n, c = get_instance('ken', repo_root()); print(tools.instance_status(LiveBackend(n, c, repo_root())))"
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
services/ha-mcp/tests/run.sh # 42 tests, offline
|
||||
```
|
||||
|
||||
Offline in the same sense as `scripts/ha/tests/*`: no network, no HA
|
||||
instance, no token. Small hand-made fixtures under `tests/fixtures/` (shaped
|
||||
like real `/api/states` and WebSocket registry payloads) cover the tool logic;
|
||||
the repo's own `config/ken/automations/` and newest `fixtures/ken-states-*.yaml`
|
||||
cover the repo-backed and at-scale paths (ranking over ~1650 real entities
|
||||
behaves differently from ranking over ten).
|
||||
|
||||
Live smoke (read-only, run against `ken` on 2026-07-30):
|
||||
`instance_status` → HA 2026.7.2, 1647 entities, 377 unavailable, 115
|
||||
automations, 13 areas; `get_state("sensor.thsalon_temperature")` → `24.5 °C`.
|
||||
|
||||
|
|
@ -1,29 +1,14 @@
|
|||
# home-assistant (configs-as-code)
|
||||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-22
|
||||
links:
|
||||
- ../services/home-assistant.md
|
||||
---
|
||||
|
||||
**Status: phase 1 (partial).** Read-only import tooling, plus a deploy
|
||||
(repo -> instance) write path for the `api` adapter's automations/scripts/
|
||||
scenes scope only (`scripts/ha/deploy.sh`) — see "Deploy" below. Dashboards,
|
||||
helpers, and the `docker-exec` adapter have no write path yet. See
|
||||
`DESIGN.md` for the full phasing, adapter, sync, and validation model, and
|
||||
for the open questions still blocking phase 2/3.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
services/home-assistant/
|
||||
├── DESIGN.md # decision registry — read this first
|
||||
├── instances.yaml # per-instance adapter/host/token config
|
||||
├── .gitignore # excludes secrets/db/log/token paths from every import
|
||||
├── config/<instance>/ # canonical, normalized /config mirror per instance
|
||||
├── storage-export/<instance>/ # curated .storage/* export (registries, dashboards)
|
||||
└── fixtures/ # dated /api/states snapshots
|
||||
```
|
||||
|
||||
Instances: `ken` (RPi4/HAOS, LAN 192.168.31.7, api adapter — canonical home
|
||||
instance since the 2026-07-22 cutover), `ken-legacy` (PIHA, container
|
||||
`homeassistant5`, docker-exec adapter, archived — pre-migration instance,
|
||||
import only, never deploy), `chelsty-ha` (Tailscale, api adapter — see
|
||||
`instances.yaml` and `DESIGN.md` "Incident log").
|
||||
# home-assistant — import, deploy, testy
|
||||
|
||||
## Import
|
||||
|
||||
61
kb/runbooks/kb-query-deploy.md
Normal file
61
kb/runbooks/kb-query-deploy.md
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-29
|
||||
links:
|
||||
- ../services/kb-query.md
|
||||
---
|
||||
|
||||
# kb-query — deploy i weryfikacja (PIHA)
|
||||
|
||||
## Deploy (PIHA)
|
||||
|
||||
0. Prerequisite: `ollama-piha` deployed and `bge-m3` pulled — see
|
||||
`services/ollama-piha/README.md` (the pull is a **manual** deploy step).
|
||||
1. `git pull` on PIHA (`~/homelab-codex-ws`).
|
||||
2. `cp services/kb-query/env.example services/kb-query/.env` and fill in the
|
||||
real `KB_DSN` password (the template already sets `EMBED_FALLBACK_URL`).
|
||||
On an existing install: add `EMBED_FALLBACK_URL=http://192.168.31.5:11434`
|
||||
to the existing `.env`.
|
||||
3. ```
|
||||
docker compose -f services/kb-query/docker-compose.yml \
|
||||
-f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build
|
||||
```
|
||||
4. Verify: `services/kb-query/healthcheck.sh`, then from PIHA:
|
||||
`curl "http://192.168.31.5:8230/search?q=test"` and open
|
||||
`http://192.168.31.5:8230/` in a browser.
|
||||
|
||||
## Fallback verification (execution: operator, after deploy)
|
||||
|
||||
- **Test A — SOLARIA online**: query via UI/`curl`; response has
|
||||
`"embed_backend": "solaria"`, `docker logs kb-query` shows
|
||||
`backend=solaria`, latency ~sub-second.
|
||||
- **Test B — SOLARIA offline**: either wait for the nightly power-off, or
|
||||
simulate: set `EMBED_PRIMARY_URL=http://192.0.2.1:11434` (TEST-NET, always
|
||||
unreachable) in `.env` and `docker compose … up -d` again. Query still
|
||||
works; response has `"embed_backend": "piha"`, log shows `backend=piha`
|
||||
plus a `circuit open for 30s` warning on the first hit; latency visibly
|
||||
higher (CPU + cold model load each time, `OLLAMA_KEEP_ALIVE=0`). Revert
|
||||
`.env` afterwards if simulated.
|
||||
- **Test C — SOLARIA returns**: after it is back up, within ≤30 s (one
|
||||
health-cache TTL) responses show `"embed_backend": "solaria"` again, no
|
||||
restart needed.
|
||||
|
||||
## Tests
|
||||
|
||||
```
|
||||
pip install -e packages/kb-retrieval/
|
||||
cd services/kb-query && pip install -r requirements.txt pytest pytest-asyncio && pytest
|
||||
```
|
||||
|
||||
Unit tests mock the DB connection and Ollama HTTP session (no live DB/Ollama
|
||||
required) — same style as `packages/kb-retrieval/tests/`. `tests/test_frontend.py`
|
||||
drives `GET /`/`/static/*` through FastAPI's `TestClient` without entering it
|
||||
as a context manager, so the DB-requiring `lifespan` never runs.
|
||||
|
||||
Frontend JS has its own pure-function tests (query-URL encoding, threshold
|
||||
colouring, envelope grouping), run without a browser via Node's built-in
|
||||
test runner: `node --test services/kb-query/tests/frontend/`.
|
||||
|
||||
44
kb/runbooks/mail-body-ingest-run.md
Normal file
44
kb/runbooks/mail-body-ingest-run.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-22
|
||||
links:
|
||||
- ../services/job-mail-body-ingest.md
|
||||
---
|
||||
|
||||
# mail-body-ingest — uruchomienie i testy
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero
|
||||
# DB writes (including the threading UPDATE):
|
||||
mail-body-ingest --dsn postgresql://kb:<pw>@piha:5433/kb --archive-root /home/oskar/kb/mail/archive
|
||||
|
||||
# Etap A pilot — last 12 months only (plan Decyzja 9):
|
||||
mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1
|
||||
|
||||
# Smoke-test slice:
|
||||
mail-body-ingest --dsn ... --apply --limit 10
|
||||
```
|
||||
|
||||
DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default
|
||||
`http://localhost:11434` — this job is meant to run where Ollama lives).
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e "jobs/mail-body-ingest[dev]"
|
||||
cd jobs/mail-body-ingest && pytest
|
||||
```
|
||||
|
||||
Pure unit tests (48), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect`
|
||||
and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers:
|
||||
quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/
|
||||
gmail_quote skipping), newsletter classification, threading extraction, prefix building,
|
||||
body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse
|
||||
fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks
|
||||
never reaching Ollama, Ollama-offline batch isolation, and dimension-mismatch abort.
|
||||
|
||||
|
|
@ -1,27 +1,14 @@
|
|||
# narty27
|
||||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-31
|
||||
links:
|
||||
- ../services/narty27.md
|
||||
---
|
||||
|
||||
Static hosting for a single self-contained `viz.html` (narty 2027 / Saalbach KB
|
||||
export) on PIHA. Plain `nginx:alpine` serving one Docker named volume — no build,
|
||||
no database, no dependencies.
|
||||
|
||||
- URL: `http://192.168.31.5:8240/viz.html` (and `/` — same file, see below)
|
||||
- Exposure: private (LAN/Tailscale only; no npm vhost, no public ingress)
|
||||
- Volume: `narty27_narty27_content` → `/usr/share/nginx/html:ro`
|
||||
|
||||
## Content is personal and lives outside the repo
|
||||
|
||||
The visualisation is personal content. It is **never committed** — not to this
|
||||
repo, not to any other. It exists in exactly two places:
|
||||
|
||||
1. the source on SOLARIA (`~/narty-2027/saalbach-kb/viz.html`), and
|
||||
2. the `narty27_narty27_content` Docker volume on PIHA.
|
||||
|
||||
There is no backup job and no bind mount under `/opt/homelab/data/`. If the
|
||||
volume is lost, re-run the update procedure below from SOLARIA.
|
||||
|
||||
Two copies of the same file are stored in the volume: `viz.html` (canonical name)
|
||||
and `index.html` (so the bare root `http://192.168.31.5:8240/` works without a
|
||||
path). Both must be refreshed together on every update.
|
||||
# narty27 — uruchomienie i operacje
|
||||
|
||||
## Updating the content (from SOLARIA)
|
||||
|
||||
63
kb/runbooks/ollama-piha-deploy.md
Normal file
63
kb/runbooks/ollama-piha-deploy.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-30
|
||||
links:
|
||||
- ../services/ollama-piha.md
|
||||
---
|
||||
|
||||
# ollama-piha — deploy i kalibracja
|
||||
|
||||
## Deploy (PIHA, master, after merge)
|
||||
|
||||
```bash
|
||||
cd ~/homelab-codex-ws && git pull
|
||||
cp services/ollama-piha/env.example services/ollama-piha/.env # LAN_BIND_IP
|
||||
docker compose -f services/ollama-piha/docker-compose.yml \
|
||||
-f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \
|
||||
--env-file services/ollama-piha/.env up -d
|
||||
```
|
||||
|
||||
**Then pull the model — this does NOT happen automatically:**
|
||||
|
||||
```bash
|
||||
docker exec ollama-piha ollama pull bge-m3
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
services/ollama-piha/healthcheck.sh # checks container + API + bge-m3 present
|
||||
time curl -s http://127.0.0.1:11434/api/embeddings \
|
||||
-d '{"model":"bge-m3","prompt":"test kalibracyjny"}' | head -c 80
|
||||
```
|
||||
|
||||
(`deploy-node.sh` on PIHA also picks this service up from
|
||||
`hosts/piha/services.yaml` once `.env` exists — the `ollama pull bge-m3` step
|
||||
stays manual either way.)
|
||||
|
||||
## Calibration (plan §5 step 4 — gate, not formality)
|
||||
|
||||
Before trusting the fallback under load, on live PIHA at a normal (not
|
||||
night-quiet) hour: run a few embeds as above while watching
|
||||
`docker stats ollama-piha`, note peak RAM and wall time. Verdict per plan §5
|
||||
step 5: keep as default fallback / tune `mem_limit` / fall back to explicit
|
||||
503 degradation.
|
||||
|
||||
**Calibration status: measured 2026-07-27 — verdict GO** (live PIHA under
|
||||
normal load, 3 consecutive `/api/embeddings` calls after `ollama pull bge-m3`;
|
||||
full protocol in `docs/sessions/2026-07-27-kb-f4-fallback.md` §4):
|
||||
|
||||
- Latency: 5.25 s (cold start) / 4.41 s / 4.16 s — single seconds as expected,
|
||||
no warm-up between calls by design (`OLLAMA_KEEP_ALIVE=0` releases the model
|
||||
after every request, `ollama ps` shows nothing resident in between).
|
||||
- RAM: idle ~66 MiB, burst peak ~983 MiB (`docker stats` sampled at 0.3 s) —
|
||||
well inside the 2560m ceiling; host `available` never dropped below ~1.3 GiB.
|
||||
- Kept as **default fallback** (no feature flag). The measurement was taken
|
||||
against the same container configuration this repo deploys (image,
|
||||
`OLLAMA_KEEP_ALIVE=0`, `mem_limit: 2560m`), so it carries over; only the
|
||||
model storage differed (bind mount then, named volume now), which does not
|
||||
affect RAM/latency.
|
||||
|
||||
67
kb/runbooks/paperless-worker-deploy.md
Normal file
67
kb/runbooks/paperless-worker-deploy.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-12
|
||||
links:
|
||||
- ../services/paperless-worker.md
|
||||
---
|
||||
|
||||
# Paperless OCR worker — NFS i cutover
|
||||
|
||||
## NFS: export na PIHA, mount na SOLARIA
|
||||
|
||||
Transfer idzie po **LAN** (PIHA `192.168.31.5` ↔ SOLARIA `192.168.31.70`,
|
||||
1 Gb/s, ten sam switch) — NIE po Tailscale. Przepustowość nie jest wąskim
|
||||
gardłem OCR.
|
||||
|
||||
### Host-side na PIHA (NIE w compose — krok przy deployu modułu 3)
|
||||
|
||||
```
|
||||
# /etc/exports na PIHA — export TYLKO dla SOLARII:
|
||||
/opt/homelab/data/paperless 192.168.31.70(rw,sync,no_subtree_check,no_root_squash)
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo apt install nfs-kernel-server # jesli brak
|
||||
sudo exportfs -ra
|
||||
```
|
||||
|
||||
`no_root_squash` jest potrzebne, bo entrypoint obrazu (root) robi `chown` na
|
||||
katalogach przy starcie kontenera na SOLARII; export jest ograniczony do
|
||||
jednego IP w zaufanym LAN.
|
||||
|
||||
### Strona SOLARII
|
||||
|
||||
Mounty definiuje compose jako named volumes z driverem NFS — **zero wpisów
|
||||
w /etc/fstab**; jedyny host-side wymóg to pakiet klienta:
|
||||
|
||||
```bash
|
||||
sudo apt install nfs-common
|
||||
```
|
||||
|
||||
### UID mapping (krytyczne)
|
||||
|
||||
Pliki na exporcie mają numerycznego właściciela — NFS nie tłumaczy nazw.
|
||||
Dlatego `USERMAP_UID/GID=1000` jest ustawione **w obu** compose (PIHA
|
||||
i SOLARIA); zmiana po jednej stronie = worker traci dostęp do plików.
|
||||
Weryfikacja po deployu: `./healthcheck.sh` robi test zapisu na mount.
|
||||
|
||||
## Cutover checklist (przy deployu modułu 3 — po działającym module 2)
|
||||
|
||||
1. ✅ `services/paperless/` działa na PIHA (healthcheck zielony).
|
||||
2. ✅ Export NFS na PIHA (wyżej) + `showmount -e 192.168.31.5` z SOLARII.
|
||||
3. ✅ `nfs-common` na SOLARII.
|
||||
4. ✅ `.env` z `env.example` — sekrety SKOPIOWANE z PIHA, nie nowe.
|
||||
5. ✅ `docker compose up -d` + `./healthcheck.sh`.
|
||||
6. ✅ Test (2026-07-12): PDF-y wrzucone do consume na PIHA, część odebrana i
|
||||
dokończona przez worker@SOLARIA (dowód w logach, zero File not found).
|
||||
7. ⬜ Test fallbacku: stop workera → zadanie czeka/mieli PIHA → start → drenaż
|
||||
(jeszcze niewykonany formalnie, ale mechanizm nie zmienił się tym fixem —
|
||||
fallback na PIHA działał już wcześniej, patrz sekcja "Fallback" wyżej).
|
||||
8. ⬜ **OTWARTE**: wpis `paperless-worker` w `hosts/solaria/services.yaml` +
|
||||
`inventory/topology.yaml` (obecnie SOLARIA ma tam tylko `node-agent`) —
|
||||
bez tego supervisor/observer nie widzą tego serwisu w desired-state, więc
|
||||
drift między `hosts/solaria/services.yaml` a rzeczywistością nie jest
|
||||
wykrywany. Patrz `docs/backlog.md`.
|
||||
105
kb/runbooks/planner-agent-deploy.md
Normal file
105
kb/runbooks/planner-agent-deploy.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: runbook
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-05-27
|
||||
links:
|
||||
- ../services/planner-agent.md
|
||||
---
|
||||
|
||||
# planner-agent — deploy, uruchamianie, testy
|
||||
|
||||
## Deployment na SOLARIA
|
||||
|
||||
```bash
|
||||
# 1. Przygotuj .env na solaria
|
||||
ssh oskar@100.100.231.104
|
||||
mkdir -p /opt/homelab/config/planner-agent
|
||||
cat > /opt/homelab/config/planner-agent/.env << 'EOF'
|
||||
OLLAMA_HOST=http://host-gateway:11434
|
||||
OLLAMA_MODEL=qwen2.5-coder:14b
|
||||
REDIS_URL=redis://100.108.208.3:6379
|
||||
NODE_NAME=solaria
|
||||
COOLDOWN_SECONDS=300
|
||||
RUNTIME_PATH=/opt/homelab
|
||||
EOF
|
||||
|
||||
# 2. Deploy przez standardowy skrypt (z SATURN)
|
||||
scripts/deploy/deploy.sh --service planner-agent
|
||||
|
||||
# 3. Weryfikacja
|
||||
docker ps --filter name=planner-agent
|
||||
docker logs planner-agent --tail 30
|
||||
ls -la /opt/homelab/state/planner-agent.heartbeat
|
||||
```
|
||||
|
||||
> **Uwaga:** stage `verify` w deploy.sh może failować bezpośrednio po starcie
|
||||
> (race condition — heartbeat jest pisany co ~5 s). Kontener jest zdrowy gdy
|
||||
> `docker ps` pokazuje `(healthy)`.
|
||||
|
||||
---
|
||||
|
||||
## Uruchamianie lokalne (bez Dockera)
|
||||
|
||||
```bash
|
||||
cd services/planner-agent
|
||||
|
||||
# Zainstaluj zależności
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Wyeksportuj zmienne (lub stwórz .env i użyj `export $(cat .env | xargs)`)
|
||||
export REDIS_URL=redis://localhost:6379
|
||||
export OLLAMA_HOST=http://localhost:11434
|
||||
export OLLAMA_MODEL=qwen2.5:7b
|
||||
export NODE_NAME=dev-local
|
||||
export RUNTIME_PATH=/tmp/homelab-dev
|
||||
export COOLDOWN_SECONDS=10
|
||||
|
||||
# Uruchom
|
||||
python src/planner.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testowanie
|
||||
|
||||
### Testy jednostkowe
|
||||
|
||||
```bash
|
||||
cd services/planner-agent
|
||||
pip install -r requirements.txt pytest pytest-asyncio
|
||||
pytest tests/ -v
|
||||
# 49 testów planner + 34 testy llm_router
|
||||
```
|
||||
|
||||
### Ręczny end-to-end test przez Redis
|
||||
|
||||
```bash
|
||||
# Opublikuj sztuczne zdarzenie unhealthy na kanale health_events
|
||||
redis-cli -h 100.108.208.3 PUBLISH health_events '{
|
||||
"type": "service_unhealthy",
|
||||
"node": "piha",
|
||||
"service": "mosquitto",
|
||||
"severity": "error",
|
||||
"payload": {"exit_code": 1, "reason": "OOMKilled"},
|
||||
"timestamp": "2026-05-27T20:00:00Z"
|
||||
}'
|
||||
|
||||
# Obserwuj logi planera
|
||||
docker logs planner-agent -f
|
||||
|
||||
# Sprawdź czy propozycja trafiła do pending
|
||||
ls /opt/homelab/actions/pending/
|
||||
cat /opt/homelab/actions/pending/plan-piha-mosquitto-*.json
|
||||
```
|
||||
|
||||
### Śledzenie metryk LLM
|
||||
|
||||
```bash
|
||||
# Subskrybuj metryki routera w czasie rzeczywistym
|
||||
redis-cli -h 100.108.208.3 SUBSCRIBE llm_router_metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-06-11
|
||||
links:
|
||||
- ../runbooks/ha-diag-agent-runbook.md
|
||||
---
|
||||
|
||||
# ha-diag-agent
|
||||
|
||||
Per-host Home Assistant diagnostic agent. Polls HA REST API on a schedule,
|
||||
|
|
@ -52,17 +62,6 @@ checks are APScheduler intervals (stateless REST polls).
|
|||
Event routing in supervisor (Phase 5) maps these to `notify` actions.
|
||||
`ha_websocket_recovered` should be routed to clear any active `ha_websocket_dead` incident.
|
||||
|
||||
## First-time deployment
|
||||
|
||||
See **[DEPLOY.md](DEPLOY.md)** for the full procedure: HA token creation,
|
||||
per-host `.env` config, deploy commands, verification steps, 48h shadow-mode
|
||||
observation, and rollback.
|
||||
|
||||
**Shadow mode** (`HA_DIAG_SHADOW_MODE`, default `true` on the control-plane):
|
||||
`ha_websocket_dead` events are downgraded to `alert_only` with a `[SHADOW MODE]`
|
||||
note instead of queuing an automatic `container_restart`. Set to `false` in
|
||||
`/opt/homelab/config/control-plane/.env` on the VPS when ready for live actions.
|
||||
|
||||
## Deployment model
|
||||
|
||||
The agent is deployed **per-host** but targets a potentially remote HA instance:
|
||||
|
|
@ -78,47 +77,6 @@ OS VM. `chelsty-infra` is the hypervisor but does not run HA itself. The agent o
|
|||
gets a new Tailscale IP, update `HA_URL` in `/opt/homelab/config/ha-diag-agent/.env` on
|
||||
`chelsty-infra`.
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
# 1. Create config on target node
|
||||
ssh oskar@<node-ip>
|
||||
mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent
|
||||
cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF'
|
||||
HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra
|
||||
HA_TOKEN=<long-lived-token>
|
||||
NODE_NAME=piha # or chelsty-infra
|
||||
LOCATION_TAG=ken # or chelsty
|
||||
CHECK_INTERVAL=60
|
||||
EOF
|
||||
|
||||
# 2. Deploy
|
||||
scripts/deploy/deploy.sh --service ha-diag-agent
|
||||
|
||||
# 3. Verify
|
||||
docker ps --filter name=ha-diag-agent
|
||||
docker exec ha-diag-agent python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8087/health', timeout=5).read().decode())"
|
||||
```
|
||||
|
||||
### chelsty-infra note
|
||||
|
||||
`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated):
|
||||
```bash
|
||||
docker-compose -f docker-compose.yml up -d --build
|
||||
```
|
||||
|
||||
### HA long-lived token
|
||||
|
||||
In HA UI: Profile → Long-Lived Access Tokens → Create token.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
cd services/ha-diag-agent
|
||||
pip install -e ".[dev]"
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
## Optional YAML config
|
||||
|
||||
Place `/opt/homelab/config/ha-diag-agent/ha-diag-agent.yaml` on the node.
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-30
|
||||
links:
|
||||
- ../runbooks/ha-mcp-install.md
|
||||
---
|
||||
|
||||
# ha-mcp — read-only MCP server for Home Assistant
|
||||
|
||||
**Status: phase 2a** of `services/home-assistant/DESIGN.md` — own minimal MCP
|
||||
|
|
@ -32,74 +42,6 @@ verify). See `services/home-assistant/DESIGN.md`, "Sync model" and
|
|||
"Validation gate". Nothing in this server bypasses that, and nothing in this
|
||||
server should ever learn to.
|
||||
|
||||
## Install
|
||||
|
||||
The `mcp` SDK (and its pydantic/anyio/httpx dependency tree) is not packaged
|
||||
for Debian and is not needed by anything else in this repo, so it goes into a
|
||||
venv rather than into the system Python:
|
||||
|
||||
```bash
|
||||
python3 -m venv --system-site-packages services/ha-mcp/.venv
|
||||
services/ha-mcp/.venv/bin/pip install mcp pytest
|
||||
```
|
||||
|
||||
`--system-site-packages` is deliberate: `requests`, `PyYAML` and
|
||||
`websocket-client` are already installed system-wide and used by
|
||||
`scripts/ha/lib/*`; the venv reuses those exact versions instead of shadowing
|
||||
them with a second copy. `.venv/` is already covered by the repo `.gitignore`.
|
||||
|
||||
`pip install --break-system-packages mcp` also works and is one line shorter,
|
||||
but it writes into the system interpreter that runs every deploy script on
|
||||
this workstation — a venv keeps a 40-package dependency tree out of that blast
|
||||
radius for a tool only Claude Code uses. Use the venv.
|
||||
|
||||
`run.sh` prefers `services/ha-mcp/.venv/bin/python` and falls back to the
|
||||
system `python3`; if neither can import `mcp` it exits with one actionable
|
||||
line on stderr rather than a traceback.
|
||||
|
||||
## Registration in Claude Code
|
||||
|
||||
`.mcp.json` in the repo root (project scope — shared with anyone who checks
|
||||
out this repo):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"ha": {
|
||||
"command": "./services/ha-mcp/run.sh",
|
||||
"args": [],
|
||||
"env": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The command path is relative, so it resolves in any checkout (main or task
|
||||
worktree) as long as Claude Code is started from the repo root. Start CC
|
||||
there; on first run it asks whether to trust the project's MCP servers. Check
|
||||
with `/mcp` — the server appears as `ha`, its tools as `mcp__ha__<tool>`.
|
||||
Which repo the server reads is derived from its own location; override with
|
||||
the `HA_MCP_REPO` environment variable if you ever need to point one checkout
|
||||
at another's config.
|
||||
|
||||
## Standalone
|
||||
|
||||
```bash
|
||||
./services/ha-mcp/run.sh # stdio server — speaks JSON-RPC on stdout
|
||||
services/ha-mcp/tests/run.sh # offline test suite (no HA, no token)
|
||||
services/ha-mcp/.venv/bin/python services/ha-mcp/tests/smoke_live.py [instance] [entity_id]
|
||||
```
|
||||
|
||||
`smoke_live.py` is the only thing here that touches the network (GET only). A
|
||||
one-off tool call without a client is easiest through the package:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=services/ha-mcp/src services/ha-mcp/.venv/bin/python -c "
|
||||
from ha_mcp import tools; from ha_mcp.backend import LiveBackend
|
||||
from ha_mcp.config import get_instance, repo_root
|
||||
n, c = get_instance('ken', repo_root()); print(tools.instance_status(LiveBackend(n, c, repo_root())))"
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
All seven are read-only, take an optional `instance` (default `ken` — the
|
||||
|
|
@ -231,23 +173,6 @@ a log line, in a tool result, or in this repo. A missing token is a reported
|
|||
tool error naming the path it looked at — not a crash, and not a silent empty
|
||||
result.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
services/ha-mcp/tests/run.sh # 42 tests, offline
|
||||
```
|
||||
|
||||
Offline in the same sense as `scripts/ha/tests/*`: no network, no HA
|
||||
instance, no token. Small hand-made fixtures under `tests/fixtures/` (shaped
|
||||
like real `/api/states` and WebSocket registry payloads) cover the tool logic;
|
||||
the repo's own `config/ken/automations/` and newest `fixtures/ken-states-*.yaml`
|
||||
cover the repo-backed and at-scale paths (ranking over ~1650 real entities
|
||||
behaves differently from ranking over ten).
|
||||
|
||||
Live smoke (read-only, run against `ken` on 2026-07-30):
|
||||
`instance_status` → HA 2026.7.2, 1647 entities, 377 unavailable, 115
|
||||
automations, 13 areas; `get_state("sensor.thsalon_temperature")` → `24.5 °C`.
|
||||
|
||||
## Not a deployed service
|
||||
|
||||
No `docker-compose.yml`, no `service.yaml`, no `healthcheck.sh`: this is
|
||||
37
kb/services/home-assistant.md
Normal file
37
kb/services/home-assistant.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-22
|
||||
links:
|
||||
- ../runbooks/home-assistant-deploy.md
|
||||
---
|
||||
|
||||
# home-assistant (configs-as-code)
|
||||
|
||||
**Status: phase 1 (partial).** Read-only import tooling, plus a deploy
|
||||
(repo -> instance) write path for the `api` adapter's automations/scripts/
|
||||
scenes scope only (`scripts/ha/deploy.sh`) — see "Deploy" below. Dashboards,
|
||||
helpers, and the `docker-exec` adapter have no write path yet. See
|
||||
`DESIGN.md` for the full phasing, adapter, sync, and validation model, and
|
||||
for the open questions still blocking phase 2/3.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
services/home-assistant/
|
||||
├── DESIGN.md # decision registry — read this first
|
||||
├── instances.yaml # per-instance adapter/host/token config
|
||||
├── .gitignore # excludes secrets/db/log/token paths from every import
|
||||
├── config/<instance>/ # canonical, normalized /config mirror per instance
|
||||
├── storage-export/<instance>/ # curated .storage/* export (registries, dashboards)
|
||||
└── fixtures/ # dated /api/states snapshots
|
||||
```
|
||||
|
||||
Instances: `ken` (RPi4/HAOS, LAN 192.168.31.7, api adapter — canonical home
|
||||
instance since the 2026-07-22 cutover), `ken-legacy` (PIHA, container
|
||||
`homeassistant5`, docker-exec adapter, archived — pre-migration instance,
|
||||
import only, never deploy), `chelsty-ha` (Tailscale, api adapter — see
|
||||
`instances.yaml` and `DESIGN.md` "Incident log").
|
||||
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-14
|
||||
links:
|
||||
- ../runbooks/gmail-header-backfill-run.md
|
||||
---
|
||||
|
||||
# gmail-header-backfill
|
||||
|
||||
One-shot job, module 5 phase 2 backfill (`docs/kb/modules/05-faza2-plan.md`, §5).
|
||||
|
|
@ -27,26 +37,6 @@ Install (from repo root, on PIHA):
|
|||
pip install -e jobs/gmail-header-backfill/
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Dry run (default) — parse and count only, no DB writes:
|
||||
gmail-header-backfill --dsn postgresql://kb:<pw>@localhost:5433/kb --limit 100
|
||||
|
||||
# Real run — apply the UPDATE for this slice:
|
||||
gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply
|
||||
|
||||
# Next slice — offset is stable/deterministic (ORDER BY id), independent of
|
||||
# how many rows in earlier slices were already backfilled:
|
||||
gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply
|
||||
```
|
||||
|
||||
DSN can also come from the `KB_DSN` env var instead of `--dsn`.
|
||||
|
||||
`--limit`/`--offset` exist so the full 225 030-row backfill can be run in
|
||||
verifiable partitions instead of one long unattended run (plan §5.3) — start
|
||||
small (`--limit 100`), check the result in the DB, then widen.
|
||||
|
||||
## Source of headers: the archived `.eml`, not the original mbox
|
||||
|
||||
Same access pattern as `documents-ingest`: `archive_root / row["raw_ref"]` ->
|
||||
|
|
@ -171,23 +161,6 @@ of streaming one mbox — the plan's estimate is "same order of magnitude,
|
|||
likely tens of minutes." Not measured directly in this change — see the
|
||||
DoD note below.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e jobs/gmail-header-backfill/
|
||||
cd jobs/gmail-header-backfill && pytest
|
||||
```
|
||||
|
||||
Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml`
|
||||
bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an
|
||||
in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`,
|
||||
quoted display names with commas, multiple `Delivered-To` occurrences, RFC
|
||||
2047 encoded-words including Polish diacritics, malformed encoded-words that
|
||||
must not raise, missing/multiple `From`, `date_raw` preserving literal text
|
||||
vs. `Date` header reformatting), idempotency (rows already carrying a
|
||||
`headers` entity are skipped and never re-appended), batch flushing, and
|
||||
`--limit`/`--offset` query shape.
|
||||
|
||||
## Definition of Done
|
||||
|
||||
Per `CLAUDE.md`: this job's smoke run is `gmail-header-backfill --dsn ...
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-22
|
||||
links:
|
||||
- ../runbooks/mail-body-ingest-run.md
|
||||
---
|
||||
|
||||
# mail-body-ingest
|
||||
|
||||
Module 5, faza mailowa (`docs/kb/modules/05-faza-mailowa-plan.md`, §5, Krok 2). Second full
|
||||
|
|
@ -28,23 +38,6 @@ pip install -e packages/kb-retrieval/
|
|||
pip install -e jobs/mail-body-ingest/
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero
|
||||
# DB writes (including the threading UPDATE):
|
||||
mail-body-ingest --dsn postgresql://kb:<pw>@piha:5433/kb --archive-root /home/oskar/kb/mail/archive
|
||||
|
||||
# Etap A pilot — last 12 months only (plan Decyzja 9):
|
||||
mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1
|
||||
|
||||
# Smoke-test slice:
|
||||
mail-body-ingest --dsn ... --apply --limit 10
|
||||
```
|
||||
|
||||
DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default
|
||||
`http://localhost:11434` — this job is meant to run where Ollama lives).
|
||||
|
||||
## Pipeline (per envelope)
|
||||
|
||||
1. **Read** `archive_root / raw_ref` — `missing_file`/`read_error` counted like
|
||||
|
|
@ -107,21 +100,6 @@ fetch is already scoped to it. Built this way from the start per the plan's note
|
|||
models (not an active bug there today, since one run always uses one model, but worth not
|
||||
repeating the ambiguity here).
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e "jobs/mail-body-ingest[dev]"
|
||||
cd jobs/mail-body-ingest && pytest
|
||||
```
|
||||
|
||||
Pure unit tests (48), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect`
|
||||
and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers:
|
||||
quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/
|
||||
gmail_quote skipping), newsletter classification, threading extraction, prefix building,
|
||||
body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse
|
||||
fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks
|
||||
never reaching Ollama, Ollama-offline batch isolation, and dimension-mismatch abort.
|
||||
|
||||
## Definition of Done
|
||||
|
||||
Per `CLAUDE.md`: `pytest` passes (48/48) + a `--limit 5` dry-run smoke against live
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-29
|
||||
links:
|
||||
- ../runbooks/kb-query-deploy.md
|
||||
---
|
||||
|
||||
# kb-query
|
||||
|
||||
FastAPI search API in front of the module-5 KB retrieval engine
|
||||
|
|
@ -120,55 +130,6 @@ that *wrote* the summary, e.g. `claude-haiku-4-5`, not the embedder).
|
|||
(`claude-haiku-4-5`). `OLLAMA_URL` was **renamed** to `EMBED_PRIMARY_URL` in
|
||||
Krok 2 — if an old `.env` sets `OLLAMA_URL`, it is ignored.
|
||||
|
||||
## Deploy (PIHA)
|
||||
|
||||
0. Prerequisite: `ollama-piha` deployed and `bge-m3` pulled — see
|
||||
`services/ollama-piha/README.md` (the pull is a **manual** deploy step).
|
||||
1. `git pull` on PIHA (`~/homelab-codex-ws`).
|
||||
2. `cp services/kb-query/env.example services/kb-query/.env` and fill in the
|
||||
real `KB_DSN` password (the template already sets `EMBED_FALLBACK_URL`).
|
||||
On an existing install: add `EMBED_FALLBACK_URL=http://192.168.31.5:11434`
|
||||
to the existing `.env`.
|
||||
3. ```
|
||||
docker compose -f services/kb-query/docker-compose.yml \
|
||||
-f hosts/piha/runtime/kb-query/docker-compose.override.yml up -d --build
|
||||
```
|
||||
4. Verify: `services/kb-query/healthcheck.sh`, then from PIHA:
|
||||
`curl "http://192.168.31.5:8230/search?q=test"` and open
|
||||
`http://192.168.31.5:8230/` in a browser.
|
||||
|
||||
## Fallback verification (execution: operator, after deploy)
|
||||
|
||||
- **Test A — SOLARIA online**: query via UI/`curl`; response has
|
||||
`"embed_backend": "solaria"`, `docker logs kb-query` shows
|
||||
`backend=solaria`, latency ~sub-second.
|
||||
- **Test B — SOLARIA offline**: either wait for the nightly power-off, or
|
||||
simulate: set `EMBED_PRIMARY_URL=http://192.0.2.1:11434` (TEST-NET, always
|
||||
unreachable) in `.env` and `docker compose … up -d` again. Query still
|
||||
works; response has `"embed_backend": "piha"`, log shows `backend=piha`
|
||||
plus a `circuit open for 30s` warning on the first hit; latency visibly
|
||||
higher (CPU + cold model load each time, `OLLAMA_KEEP_ALIVE=0`). Revert
|
||||
`.env` afterwards if simulated.
|
||||
- **Test C — SOLARIA returns**: after it is back up, within ≤30 s (one
|
||||
health-cache TTL) responses show `"embed_backend": "solaria"` again, no
|
||||
restart needed.
|
||||
|
||||
## Tests
|
||||
|
||||
```
|
||||
pip install -e packages/kb-retrieval/
|
||||
cd services/kb-query && pip install -r requirements.txt pytest pytest-asyncio && pytest
|
||||
```
|
||||
|
||||
Unit tests mock the DB connection and Ollama HTTP session (no live DB/Ollama
|
||||
required) — same style as `packages/kb-retrieval/tests/`. `tests/test_frontend.py`
|
||||
drives `GET /`/`/static/*` through FastAPI's `TestClient` without entering it
|
||||
as a context manager, so the DB-requiring `lifespan` never runs.
|
||||
|
||||
Frontend JS has its own pure-function tests (query-URL encoding, threshold
|
||||
colouring, envelope grouping), run without a browser via Node's built-in
|
||||
test runner: `node --test services/kb-query/tests/frontend/`.
|
||||
|
||||
## Ingress (`kb.kapala.org`, plan §8)
|
||||
|
||||
Wired up 2026-07-23 (`docs/sessions/2026-07-23-kb-f4-ingress.md`), **no code
|
||||
35
kb/services/narty27.md
Normal file
35
kb/services/narty27.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-31
|
||||
links:
|
||||
- ../runbooks/narty27-deploy.md
|
||||
---
|
||||
|
||||
# narty27
|
||||
|
||||
Static hosting for a single self-contained `viz.html` (narty 2027 / Saalbach KB
|
||||
export) on PIHA. Plain `nginx:alpine` serving one Docker named volume — no build,
|
||||
no database, no dependencies.
|
||||
|
||||
- URL: `http://192.168.31.5:8240/viz.html` (and `/` — same file, see below)
|
||||
- Exposure: private (LAN/Tailscale only; no npm vhost, no public ingress)
|
||||
- Volume: `narty27_narty27_content` → `/usr/share/nginx/html:ro`
|
||||
|
||||
## Content is personal and lives outside the repo
|
||||
|
||||
The visualisation is personal content. It is **never committed** — not to this
|
||||
repo, not to any other. It exists in exactly two places:
|
||||
|
||||
1. the source on SOLARIA (`~/narty-2027/saalbach-kb/viz.html`), and
|
||||
2. the `narty27_narty27_content` Docker volume on PIHA.
|
||||
|
||||
There is no backup job and no bind mount under `/opt/homelab/data/`. If the
|
||||
volume is lost, re-run the update procedure below from SOLARIA.
|
||||
|
||||
Two copies of the same file are stored in the volume: `viz.html` (canonical name)
|
||||
and `index.html` (so the bare root `http://192.168.31.5:8240/` works without a
|
||||
path). Both must be refreshed together on every update.
|
||||
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-30
|
||||
links:
|
||||
- ../runbooks/ollama-piha-deploy.md
|
||||
---
|
||||
|
||||
# ollama-piha
|
||||
|
||||
Local CPU Ollama on **PIHA**, serving exactly one purpose: the **fallback embed
|
||||
|
|
@ -29,57 +39,6 @@ model is never resident — see below). Slower but alive beats fast but dead.
|
|||
PIHA's uid pattern (host oskar=1004, containers uid 1000, setgid group pi)
|
||||
if it wrote to a shared bind directory.
|
||||
|
||||
## Deploy (PIHA, master, after merge)
|
||||
|
||||
```bash
|
||||
cd ~/homelab-codex-ws && git pull
|
||||
cp services/ollama-piha/env.example services/ollama-piha/.env # LAN_BIND_IP
|
||||
docker compose -f services/ollama-piha/docker-compose.yml \
|
||||
-f hosts/piha/runtime/ollama-piha/docker-compose.override.yml \
|
||||
--env-file services/ollama-piha/.env up -d
|
||||
```
|
||||
|
||||
**Then pull the model — this does NOT happen automatically:**
|
||||
|
||||
```bash
|
||||
docker exec ollama-piha ollama pull bge-m3
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
services/ollama-piha/healthcheck.sh # checks container + API + bge-m3 present
|
||||
time curl -s http://127.0.0.1:11434/api/embeddings \
|
||||
-d '{"model":"bge-m3","prompt":"test kalibracyjny"}' | head -c 80
|
||||
```
|
||||
|
||||
(`deploy-node.sh` on PIHA also picks this service up from
|
||||
`hosts/piha/services.yaml` once `.env` exists — the `ollama pull bge-m3` step
|
||||
stays manual either way.)
|
||||
|
||||
## Calibration (plan §5 step 4 — gate, not formality)
|
||||
|
||||
Before trusting the fallback under load, on live PIHA at a normal (not
|
||||
night-quiet) hour: run a few embeds as above while watching
|
||||
`docker stats ollama-piha`, note peak RAM and wall time. Verdict per plan §5
|
||||
step 5: keep as default fallback / tune `mem_limit` / fall back to explicit
|
||||
503 degradation.
|
||||
|
||||
**Calibration status: measured 2026-07-27 — verdict GO** (live PIHA under
|
||||
normal load, 3 consecutive `/api/embeddings` calls after `ollama pull bge-m3`;
|
||||
full protocol in `docs/sessions/2026-07-27-kb-f4-fallback.md` §4):
|
||||
|
||||
- Latency: 5.25 s (cold start) / 4.41 s / 4.16 s — single seconds as expected,
|
||||
no warm-up between calls by design (`OLLAMA_KEEP_ALIVE=0` releases the model
|
||||
after every request, `ollama ps` shows nothing resident in between).
|
||||
- RAM: idle ~66 MiB, burst peak ~983 MiB (`docker stats` sampled at 0.3 s) —
|
||||
well inside the 2560m ceiling; host `available` never dropped below ~1.3 GiB.
|
||||
- Kept as **default fallback** (no feature flag). The measurement was taken
|
||||
against the same container configuration this repo deploys (image,
|
||||
`OLLAMA_KEEP_ALIVE=0`, `mem_limit: 2560m`), so it carries over; only the
|
||||
model storage differed (bind mount then, named volume now), which does not
|
||||
affect RAM/latency.
|
||||
|
||||
## Relation to kb-query
|
||||
|
||||
kb-query's router (`services/kb-query/app/embed_router.py`) health-checks
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-07-12
|
||||
links:
|
||||
- ../runbooks/paperless-worker-deploy.md
|
||||
---
|
||||
|
||||
# Paperless OCR worker (SOLARIA)
|
||||
|
||||
Ciężki OCR filaru KB #2 (moduł 3, `docs/kb/modules/03-paperless-ocr-worker.md`).
|
||||
|
|
@ -108,58 +118,3 @@ miała własny, lokalny, pusty `/tmp/paperless`. Fix: NFS-owy wolumen
|
|||
katalog `/opt/homelab/data/paperless/scratch` (już istnieje na PIHA,
|
||||
`chown 1000:1000`), montowany na `/tmp/paperless` po obu stronach.
|
||||
|
||||
## NFS: export na PIHA, mount na SOLARIA
|
||||
|
||||
Transfer idzie po **LAN** (PIHA `192.168.31.5` ↔ SOLARIA `192.168.31.70`,
|
||||
1 Gb/s, ten sam switch) — NIE po Tailscale. Przepustowość nie jest wąskim
|
||||
gardłem OCR.
|
||||
|
||||
### Host-side na PIHA (NIE w compose — krok przy deployu modułu 3)
|
||||
|
||||
```
|
||||
# /etc/exports na PIHA — export TYLKO dla SOLARII:
|
||||
/opt/homelab/data/paperless 192.168.31.70(rw,sync,no_subtree_check,no_root_squash)
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo apt install nfs-kernel-server # jesli brak
|
||||
sudo exportfs -ra
|
||||
```
|
||||
|
||||
`no_root_squash` jest potrzebne, bo entrypoint obrazu (root) robi `chown` na
|
||||
katalogach przy starcie kontenera na SOLARII; export jest ograniczony do
|
||||
jednego IP w zaufanym LAN.
|
||||
|
||||
### Strona SOLARII
|
||||
|
||||
Mounty definiuje compose jako named volumes z driverem NFS — **zero wpisów
|
||||
w /etc/fstab**; jedyny host-side wymóg to pakiet klienta:
|
||||
|
||||
```bash
|
||||
sudo apt install nfs-common
|
||||
```
|
||||
|
||||
### UID mapping (krytyczne)
|
||||
|
||||
Pliki na exporcie mają numerycznego właściciela — NFS nie tłumaczy nazw.
|
||||
Dlatego `USERMAP_UID/GID=1000` jest ustawione **w obu** compose (PIHA
|
||||
i SOLARIA); zmiana po jednej stronie = worker traci dostęp do plików.
|
||||
Weryfikacja po deployu: `./healthcheck.sh` robi test zapisu na mount.
|
||||
|
||||
## Cutover checklist (przy deployu modułu 3 — po działającym module 2)
|
||||
|
||||
1. ✅ `services/paperless/` działa na PIHA (healthcheck zielony).
|
||||
2. ✅ Export NFS na PIHA (wyżej) + `showmount -e 192.168.31.5` z SOLARII.
|
||||
3. ✅ `nfs-common` na SOLARII.
|
||||
4. ✅ `.env` z `env.example` — sekrety SKOPIOWANE z PIHA, nie nowe.
|
||||
5. ✅ `docker compose up -d` + `./healthcheck.sh`.
|
||||
6. ✅ Test (2026-07-12): PDF-y wrzucone do consume na PIHA, część odebrana i
|
||||
dokończona przez worker@SOLARIA (dowód w logach, zero File not found).
|
||||
7. ⬜ Test fallbacku: stop workera → zadanie czeka/mieli PIHA → start → drenaż
|
||||
(jeszcze niewykonany formalnie, ale mechanizm nie zmienił się tym fixem —
|
||||
fallback na PIHA działał już wcześniej, patrz sekcja "Fallback" wyżej).
|
||||
8. ⬜ **OTWARTE**: wpis `paperless-worker` w `hosts/solaria/services.yaml` +
|
||||
`inventory/topology.yaml` (obecnie SOLARIA ma tam tylko `node-agent`) —
|
||||
bez tego supervisor/observer nie widzą tego serwisu w desired-state, więc
|
||||
drift między `hosts/solaria/services.yaml` a rzeczywistością nie jest
|
||||
wykrywany. Patrz `docs/backlog.md`.
|
||||
|
|
@ -1,3 +1,13 @@
|
|||
---
|
||||
okf: "0.1"
|
||||
type: service
|
||||
visibility: private
|
||||
status: active
|
||||
updated: 2026-05-27
|
||||
links:
|
||||
- ../runbooks/planner-agent-deploy.md
|
||||
---
|
||||
|
||||
# planner-agent
|
||||
|
||||
Asynchroniczny agent diagnozujący zdarzenia zdrowotne w homelabowej infrastrukturze.
|
||||
|
|
@ -110,99 +120,6 @@ w docker-compose.yml (nie jest w env_file — sekret injektowany przez operatora
|
|||
|
||||
---
|
||||
|
||||
## Deployment na SOLARIA
|
||||
|
||||
```bash
|
||||
# 1. Przygotuj .env na solaria
|
||||
ssh oskar@100.100.231.104
|
||||
mkdir -p /opt/homelab/config/planner-agent
|
||||
cat > /opt/homelab/config/planner-agent/.env << 'EOF'
|
||||
OLLAMA_HOST=http://host-gateway:11434
|
||||
OLLAMA_MODEL=qwen2.5-coder:14b
|
||||
REDIS_URL=redis://100.108.208.3:6379
|
||||
NODE_NAME=solaria
|
||||
COOLDOWN_SECONDS=300
|
||||
RUNTIME_PATH=/opt/homelab
|
||||
EOF
|
||||
|
||||
# 2. Deploy przez standardowy skrypt (z SATURN)
|
||||
scripts/deploy/deploy.sh --service planner-agent
|
||||
|
||||
# 3. Weryfikacja
|
||||
docker ps --filter name=planner-agent
|
||||
docker logs planner-agent --tail 30
|
||||
ls -la /opt/homelab/state/planner-agent.heartbeat
|
||||
```
|
||||
|
||||
> **Uwaga:** stage `verify` w deploy.sh może failować bezpośrednio po starcie
|
||||
> (race condition — heartbeat jest pisany co ~5 s). Kontener jest zdrowy gdy
|
||||
> `docker ps` pokazuje `(healthy)`.
|
||||
|
||||
---
|
||||
|
||||
## Uruchamianie lokalne (bez Dockera)
|
||||
|
||||
```bash
|
||||
cd services/planner-agent
|
||||
|
||||
# Zainstaluj zależności
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Wyeksportuj zmienne (lub stwórz .env i użyj `export $(cat .env | xargs)`)
|
||||
export REDIS_URL=redis://localhost:6379
|
||||
export OLLAMA_HOST=http://localhost:11434
|
||||
export OLLAMA_MODEL=qwen2.5:7b
|
||||
export NODE_NAME=dev-local
|
||||
export RUNTIME_PATH=/tmp/homelab-dev
|
||||
export COOLDOWN_SECONDS=10
|
||||
|
||||
# Uruchom
|
||||
python src/planner.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testowanie
|
||||
|
||||
### Testy jednostkowe
|
||||
|
||||
```bash
|
||||
cd services/planner-agent
|
||||
pip install -r requirements.txt pytest pytest-asyncio
|
||||
pytest tests/ -v
|
||||
# 49 testów planner + 34 testy llm_router
|
||||
```
|
||||
|
||||
### Ręczny end-to-end test przez Redis
|
||||
|
||||
```bash
|
||||
# Opublikuj sztuczne zdarzenie unhealthy na kanale health_events
|
||||
redis-cli -h 100.108.208.3 PUBLISH health_events '{
|
||||
"type": "service_unhealthy",
|
||||
"node": "piha",
|
||||
"service": "mosquitto",
|
||||
"severity": "error",
|
||||
"payload": {"exit_code": 1, "reason": "OOMKilled"},
|
||||
"timestamp": "2026-05-27T20:00:00Z"
|
||||
}'
|
||||
|
||||
# Obserwuj logi planera
|
||||
docker logs planner-agent -f
|
||||
|
||||
# Sprawdź czy propozycja trafiła do pending
|
||||
ls /opt/homelab/actions/pending/
|
||||
cat /opt/homelab/actions/pending/plan-piha-mosquitto-*.json
|
||||
```
|
||||
|
||||
### Śledzenie metryk LLM
|
||||
|
||||
```bash
|
||||
# Subskrybuj metryki routera w czasie rzeczywistym
|
||||
redis-cli -h 100.108.208.3 SUBSCRIBE llm_router_metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Skrypt `healthcheck.sh` sprawdza czy plik heartbeat
|
||||
Loading…
Reference in a new issue