homelab-codex-ws/hosts/piha/services.yaml

144 lines
4.8 KiB
YAML
Raw Normal View History

host: piha
services:
ha-diag-agent:
role: ha-diagnostic-agent
deployment_model: docker-compose
exposure: local-only
offline_required: false
depends_on:
local: []
external: [homeassistant]
config:
target_url: http://localhost:8123
location_tag: "ken"
events_dir: /opt/homelab/events/piha
runtime:
config_path: /opt/homelab/config/ha-diag-agent
data_path: /var/lib/ha-diag-agent
feat(node-agent): implement health monitor and safe cleanup policy scripts/monitor/health-monitor.sh (new): - Standalone bash health monitor: disk/RAM/CPU checks + docker container health - Per-node-type cleanup policy enforced: lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops sd_card (piha, saturn): dangling images + containers, rate-limited once/24h ai_node (solaria): dangling + containers + build cache, NEVER -a standard (vps): dangling + containers + build cache + CP filesystem rotation - VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d, events >3d AND past observer checkpoint - Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu, containers_not_running, healthcheck_failed) services/node-agent/ (new): - Python daemon (node_agent.py): same policy as bash script, Docker SDK for container checks and cleanup, /proc for system metrics - Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var) - Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0 - docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only observer.py: - Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure - Handle disk_pressure: record severity on node, clear when healthy - Handle high_memory / high_cpu: record pressure level for correlation supervisor.py: - Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha} - reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure - _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node}, checks all active states, risk=guarded (operator approval required) executor.py: - Handle disk_cleanup action type via _execute_disk_cleanup() - Commands come from action payload; safety gate rejects any command touching /opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf / hosts/*/services.yaml: - Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra - Add node-agent to chelsty-ha (previously missing) - Add cleanup policy notes to LTE node comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
node-agent:
role: node-stability-monitor
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
runtime:
feat(node-agent): implement health monitor and safe cleanup policy scripts/monitor/health-monitor.sh (new): - Standalone bash health monitor: disk/RAM/CPU checks + docker container health - Per-node-type cleanup policy enforced: lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops sd_card (piha, saturn): dangling images + containers, rate-limited once/24h ai_node (solaria): dangling + containers + build cache, NEVER -a standard (vps): dangling + containers + build cache + CP filesystem rotation - VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d, events >3d AND past observer checkpoint - Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu, containers_not_running, healthcheck_failed) services/node-agent/ (new): - Python daemon (node_agent.py): same policy as bash script, Docker SDK for container checks and cleanup, /proc for system metrics - Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var) - Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0 - docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only observer.py: - Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure - Handle disk_pressure: record severity on node, clear when healthy - Handle high_memory / high_cpu: record pressure level for correlation supervisor.py: - Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha} - reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure - _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node}, checks all active states, risk=guarded (operator approval required) executor.py: - Handle disk_cleanup action type via _execute_disk_cleanup() - Commands come from action payload; safety gate rejects any command touching /opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf / hosts/*/services.yaml: - Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra - Add node-agent to chelsty-ha (previously missing) - Add cleanup policy notes to LTE node comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
config_path: /opt/homelab/config/node-agent
data_path: /opt/homelab/state
logs_path: /opt/homelab/events
brain-watchdog:
role: control-plane-watchdog
deployment_model: docker-compose
exposure: private
offline_required: false
depends_on:
local: []
external: [control-plane]
runtime:
config_path: /opt/homelab/config/brain-watchdog
vikunja:
role: task-management
deployment_model: docker-compose
exposure: private # public via npm reverse proxy at vikunja.okit.pl
offline_required: false
depends_on:
local: []
external: [forgejo] # OIDC identity provider
ports:
- name: http
container_port: 3456
protocol: tcp
runtime:
# secrets (.env) + OIDC config (config.yml) live alongside the compose file
config_path: services/vikunja
# data is in Docker named volumes: vikunja_vikunja_db, vikunja_vikunja_files
llm-gateway:
role: llm-router # FastAPI proxy -> Ollama @ SOLARIA :11434
deployment_model: docker-compose
exposure: private # Tailscale-only bind (TAILSCALE_BIND_IP); no public ingress
offline_required: false
depends_on:
local: []
external: [ollama] # SOLARIA may be powered down -> routes 502, health stays ok
ports:
- name: http
container_port: 8080
protocol: tcp
runtime:
# .env (TAILSCALE_BIND_IP) lives alongside the compose file; stateless, no data path
config_path: services/llm-gateway
feat(kb): faza 3 krok 5 — cykliczny ingest (systemd timer) + alerting documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py): orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic, claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA (availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem. Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors, llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis, last_success_timestamp trzymany z poprzedniego pliku przy failu) do /opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu). systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer (OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do /opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge. fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical) + KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml). node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override (--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) + topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA. Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń, 0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany). Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge), zdecydowanie czy stability-agent też idzie na per-host przy okazji. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 15:05:18 +02:00
node_exporter:
role: metrics-exporter
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
runtime:
# textfile collector reads /opt/homelab/state/node-exporter (module 5 phase 3 step 5,
# docs/kb/modules/05-faza3-plan.md §7.2 — kb-ingest.prom) via the existing /:/host:ro
# mount, see hosts/piha/runtime/node_exporter/docker-compose.override.yml.
data_path: /opt/homelab/state/node-exporter
kb-postgres:
role: kb-database # KB spine: Postgres 16 + pgvector (always-on)
deployment_model: docker-compose
exposure: local-only # Tailscale-accessible; no public ingress
offline_required: false
depends_on:
local: []
external: []
ports:
- name: postgres
host_port: 5433
protocol: tcp
runtime:
config_path: /opt/homelab/config/kb-postgres
# data is in Docker named volume kb_postgres_data — must land on the NVMe
# (Docker data-root on /home); see hosts/piha/runtime/kb-postgres override.
feat(kb): add kb-query service skeleton (search API, no ingress yet) Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first user-facing HTTP entry point to the KB. FastAPI wrapping kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source links) and GET /healthz. Search API only, no answer synthesis (phase 5) and no server-side dist filtering — the 0.45/0.55 colour thresholds are a frontend concern (plan §7, a later step). Hard startup invariant (plan §2 decision 2): refuses to start unless the configured EMBED_MODEL is present in both document_chunk.model and document_summary.embedding_model. Note the latter: document_summary.model is the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the embedder — checked live against kb-postgres@PIHA before writing this, see app/startup.py's docstring. Verified end-to-end with a live docker run: the invariant crash-loops on a mismatched EMBED_MODEL and passes through to a real /search hit against the live corpus with a correct model. Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and a failure surfaces as 503, not a crash. Also: scripts/deploy/deploy.sh's gate now builds each service via `docker compose build` instead of a raw `docker build <svc_dir>`, so a service whose docker-compose.yml declares a repo-root build context (needed here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention already documented in CLAUDE.md) resolves the same way in the gate as it does at real deploy time (deploy-node.sh's `docker compose ... up --build`). No behavior change for existing single-context services — verified against llm-gateway's compose file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 16:06:18 +02:00
kb-query:
role: kb-search-api # module 5 phase 4: FastAPI wrapper over kb_retrieval cascade/flat query
deployment_model: docker-compose
exposure: private # LAN bind (LAN_BIND_IP); npm@PIHA vhost + OIDC is a later step
offline_required: false
depends_on:
feat(kb-query): active embed fallback SOLARIA→PIHA (module 5 phase 4, plan §2/§5) Last missing core piece of KB phase 4: kb-query no longer hard-fails /search when Ollama@SOLARIA is unreachable. app/fallback.py implements the plan's circuit-breaker exactly (30s cached health probe, 3s hard embed timeout on SOLARIA, one-shot same-request switch to a new local ollama-piha@PIHA container on timeout/error). sol_status in /healthz and /search now reflects the real breaker state instead of a hardcoded "up". New services/ollama-piha (bge-m3, OLLAMA_KEEP_ALIVE=0, arm64/no-GPU) is the local fallback leg. Live calibration on PIHA (2026-07-27, normal load): embed latency 4.2-5.2s, RAM peak ~983MiB against a 2.5GiB ceiling -- both inside the plan's go-bar, so the fallback is enabled by default rather than gated behind a flag. Calibration also surfaced and disabled (not removed) a previously-undocumented orphaned native ollama.service on PIHA that had been conflicting with the container's port. The embed-model invariant (query embedding == document_chunk.model) still enforces once at startup, since both fallback legs share one EMBED_MODEL constant by construction; a redundant per-request DB check was deliberately skipped and the invariant is instead proven structurally by test. retrieval_eval.py gains --transport http (plan §2 decision 6/§9), previously unimplemented. Verified live: HTTP transport is bit-identical to direct transport against the same live SOLARIA (0 mismatches), and a live sol-down simulation (kb-query's own OLLAMA_URL pointed at a dead address, no other Ollama consumer touched) shows the PIHA fallback answering with the same hit@3 gate outcome and dist within ~3e-4 of the SOLARIA baseline. Zero changes to DB schema or kb_retrieval's retrieval logic -- only the embed + health layer, per task constraints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 18:57:46 +02:00
local: [kb-postgres, ollama-piha] # ollama-piha is the fallback leg, not hard-required at boot
external: [ollama] # SOLARIA may be offline -> active fallback to ollama-piha, plan §2/§5
feat(kb): add kb-query service skeleton (search API, no ingress yet) Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first user-facing HTTP entry point to the KB. FastAPI wrapping kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source links) and GET /healthz. Search API only, no answer synthesis (phase 5) and no server-side dist filtering — the 0.45/0.55 colour thresholds are a frontend concern (plan §7, a later step). Hard startup invariant (plan §2 decision 2): refuses to start unless the configured EMBED_MODEL is present in both document_chunk.model and document_summary.embedding_model. Note the latter: document_summary.model is the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the embedder — checked live against kb-postgres@PIHA before writing this, see app/startup.py's docstring. Verified end-to-end with a live docker run: the invariant crash-loops on a mismatched EMBED_MODEL and passes through to a real /search hit against the live corpus with a correct model. Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and a failure surfaces as 503, not a crash. Also: scripts/deploy/deploy.sh's gate now builds each service via `docker compose build` instead of a raw `docker build <svc_dir>`, so a service whose docker-compose.yml declares a repo-root build context (needed here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention already documented in CLAUDE.md) resolves the same way in the gate as it does at real deploy time (deploy-node.sh's `docker compose ... up --build`). No behavior change for existing single-context services — verified against llm-gateway's compose file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 16:06:18 +02:00
ports:
- name: http
container_port: 8080
host_port: 8230
protocol: tcp
runtime:
feat(kb-query): active embed fallback SOLARIA→PIHA (module 5 phase 4, plan §2/§5) Last missing core piece of KB phase 4: kb-query no longer hard-fails /search when Ollama@SOLARIA is unreachable. app/fallback.py implements the plan's circuit-breaker exactly (30s cached health probe, 3s hard embed timeout on SOLARIA, one-shot same-request switch to a new local ollama-piha@PIHA container on timeout/error). sol_status in /healthz and /search now reflects the real breaker state instead of a hardcoded "up". New services/ollama-piha (bge-m3, OLLAMA_KEEP_ALIVE=0, arm64/no-GPU) is the local fallback leg. Live calibration on PIHA (2026-07-27, normal load): embed latency 4.2-5.2s, RAM peak ~983MiB against a 2.5GiB ceiling -- both inside the plan's go-bar, so the fallback is enabled by default rather than gated behind a flag. Calibration also surfaced and disabled (not removed) a previously-undocumented orphaned native ollama.service on PIHA that had been conflicting with the container's port. The embed-model invariant (query embedding == document_chunk.model) still enforces once at startup, since both fallback legs share one EMBED_MODEL constant by construction; a redundant per-request DB check was deliberately skipped and the invariant is instead proven structurally by test. retrieval_eval.py gains --transport http (plan §2 decision 6/§9), previously unimplemented. Verified live: HTTP transport is bit-identical to direct transport against the same live SOLARIA (0 mismatches), and a live sol-down simulation (kb-query's own OLLAMA_URL pointed at a dead address, no other Ollama consumer touched) shows the PIHA fallback answering with the same hit@3 gate outcome and dist within ~3e-4 of the SOLARIA baseline. Zero changes to DB schema or kb_retrieval's retrieval logic -- only the embed + health layer, per task constraints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 18:57:46 +02:00
# .env (KB_DSN, LAN_BIND_IP, OLLAMA_PIHA_URL) lives alongside the compose file; stateless, no data path
feat(kb): add kb-query service skeleton (search API, no ingress yet) Module 5 phase 4 step 1 (docs/kb/modules/05-faza4-plan.md, §4): first user-facing HTTP entry point to the KB. FastAPI wrapping kb_retrieval.cascade_query/flat_query — GET /search (query_text -> embed via Ollama@SOLARIA -> cascade/flat -> envelope join -> JSON with per-source links) and GET /healthz. Search API only, no answer synthesis (phase 5) and no server-side dist filtering — the 0.45/0.55 colour thresholds are a frontend concern (plan §7, a later step). Hard startup invariant (plan §2 decision 2): refuses to start unless the configured EMBED_MODEL is present in both document_chunk.model and document_summary.embedding_model. Note the latter: document_summary.model is the LLM that *wrote* the summary (claude-haiku-4-5/gemma3:12b), not the embedder — checked live against kb-postgres@PIHA before writing this, see app/startup.py's docstring. Verified end-to-end with a live docker run: the invariant crash-loops on a mismatched EMBED_MODEL and passes through to a real /search hit against the live corpus with a correct model. Repo-only: no deploy, no npm/OIDC/DNS wiring (plan §8, later step), no local embed fallback (plan §5, later step) — Ollama@SOLARIA is called directly and a failure surfaces as 503, not a crash. Also: scripts/deploy/deploy.sh's gate now builds each service via `docker compose build` instead of a raw `docker build <svc_dir>`, so a service whose docker-compose.yml declares a repo-root build context (needed here to COPY packages/kb-retrieval/, the packages/ Dockerfile convention already documented in CLAUDE.md) resolves the same way in the gate as it does at real deploy time (deploy-node.sh's `docker compose ... up --build`). No behavior change for existing single-context services — verified against llm-gateway's compose file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 16:06:18 +02:00
config_path: services/kb-query
feat(kb-query): active embed fallback SOLARIA→PIHA (module 5 phase 4, plan §2/§5) Last missing core piece of KB phase 4: kb-query no longer hard-fails /search when Ollama@SOLARIA is unreachable. app/fallback.py implements the plan's circuit-breaker exactly (30s cached health probe, 3s hard embed timeout on SOLARIA, one-shot same-request switch to a new local ollama-piha@PIHA container on timeout/error). sol_status in /healthz and /search now reflects the real breaker state instead of a hardcoded "up". New services/ollama-piha (bge-m3, OLLAMA_KEEP_ALIVE=0, arm64/no-GPU) is the local fallback leg. Live calibration on PIHA (2026-07-27, normal load): embed latency 4.2-5.2s, RAM peak ~983MiB against a 2.5GiB ceiling -- both inside the plan's go-bar, so the fallback is enabled by default rather than gated behind a flag. Calibration also surfaced and disabled (not removed) a previously-undocumented orphaned native ollama.service on PIHA that had been conflicting with the container's port. The embed-model invariant (query embedding == document_chunk.model) still enforces once at startup, since both fallback legs share one EMBED_MODEL constant by construction; a redundant per-request DB check was deliberately skipped and the invariant is instead proven structurally by test. retrieval_eval.py gains --transport http (plan §2 decision 6/§9), previously unimplemented. Verified live: HTTP transport is bit-identical to direct transport against the same live SOLARIA (0 mismatches), and a live sol-down simulation (kb-query's own OLLAMA_URL pointed at a dead address, no other Ollama consumer touched) shows the PIHA fallback answering with the same hit@3 gate outcome and dist within ~3e-4 of the SOLARIA baseline. Zero changes to DB schema or kb_retrieval's retrieval logic -- only the embed + health layer, per task constraints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 18:57:46 +02:00
ollama-piha:
role: kb-embed-fallback # module 5 phase 4 plan §2 decision 2 / §5: local CPU embed
# fallback for kb-query when Ollama@SOLARIA is down/times out
deployment_model: docker-compose
exposure: private # LAN bind (LAN_BIND_IP); consumed only by kb-query@PIHA
offline_required: true # PIHA-local, never depends on SOLARIA/VPS/Forgejo at runtime
depends_on:
local: []
external: []
ports:
- name: http
container_port: 11434
host_port: 11434
protocol: tcp
runtime:
# .env (LAN_BIND_IP) lives alongside the compose file
config_path: services/ollama-piha
# bge-m3 model files persist in the Docker named volume-equivalent bind
# /opt/homelab/data/ollama-piha — see hosts/piha/runtime/ollama-piha override.