homelab-codex-ws/hosts/piha/services.yaml

107 lines
3.2 KiB
YAML
Raw Normal View History

host: piha
services:
ha-diag-agent:
role: ha-diagnostic-agent
deployment_model: docker-compose
exposure: local-only
offline_required: false
depends_on:
local: []
external: [homeassistant]
config:
target_url: http://localhost:8123
location_tag: "ken"
events_dir: /opt/homelab/events/piha
runtime:
config_path: /opt/homelab/config/ha-diag-agent
data_path: /var/lib/ha-diag-agent
feat(node-agent): implement health monitor and safe cleanup policy scripts/monitor/health-monitor.sh (new): - Standalone bash health monitor: disk/RAM/CPU checks + docker container health - Per-node-type cleanup policy enforced: lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops sd_card (piha, saturn): dangling images + containers, rate-limited once/24h ai_node (solaria): dangling + containers + build cache, NEVER -a standard (vps): dangling + containers + build cache + CP filesystem rotation - VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d, events >3d AND past observer checkpoint - Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu, containers_not_running, healthcheck_failed) services/node-agent/ (new): - Python daemon (node_agent.py): same policy as bash script, Docker SDK for container checks and cleanup, /proc for system metrics - Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var) - Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0 - docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only observer.py: - Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure - Handle disk_pressure: record severity on node, clear when healthy - Handle high_memory / high_cpu: record pressure level for correlation supervisor.py: - Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha} - reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure - _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node}, checks all active states, risk=guarded (operator approval required) executor.py: - Handle disk_cleanup action type via _execute_disk_cleanup() - Commands come from action payload; safety gate rejects any command touching /opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf / hosts/*/services.yaml: - Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra - Add node-agent to chelsty-ha (previously missing) - Add cleanup policy notes to LTE node comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
node-agent:
role: node-stability-monitor
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
runtime:
feat(node-agent): implement health monitor and safe cleanup policy scripts/monitor/health-monitor.sh (new): - Standalone bash health monitor: disk/RAM/CPU checks + docker container health - Per-node-type cleanup policy enforced: lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops sd_card (piha, saturn): dangling images + containers, rate-limited once/24h ai_node (solaria): dangling + containers + build cache, NEVER -a standard (vps): dangling + containers + build cache + CP filesystem rotation - VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d, events >3d AND past observer checkpoint - Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu, containers_not_running, healthcheck_failed) services/node-agent/ (new): - Python daemon (node_agent.py): same policy as bash script, Docker SDK for container checks and cleanup, /proc for system metrics - Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var) - Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0 - docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only observer.py: - Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure - Handle disk_pressure: record severity on node, clear when healthy - Handle high_memory / high_cpu: record pressure level for correlation supervisor.py: - Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha} - reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure - _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node}, checks all active states, risk=guarded (operator approval required) executor.py: - Handle disk_cleanup action type via _execute_disk_cleanup() - Commands come from action payload; safety gate rejects any command touching /opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf / hosts/*/services.yaml: - Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra - Add node-agent to chelsty-ha (previously missing) - Add cleanup policy notes to LTE node comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
config_path: /opt/homelab/config/node-agent
data_path: /opt/homelab/state
logs_path: /opt/homelab/events
brain-watchdog:
role: control-plane-watchdog
deployment_model: docker-compose
exposure: private
offline_required: false
depends_on:
local: []
external: [control-plane]
runtime:
config_path: /opt/homelab/config/brain-watchdog
vikunja:
role: task-management
deployment_model: docker-compose
exposure: private # public via npm reverse proxy at vikunja.okit.pl
offline_required: false
depends_on:
local: []
external: [forgejo] # OIDC identity provider
ports:
- name: http
container_port: 3456
protocol: tcp
runtime:
# secrets (.env) + OIDC config (config.yml) live alongside the compose file
config_path: services/vikunja
# data is in Docker named volumes: vikunja_vikunja_db, vikunja_vikunja_files
llm-gateway:
role: llm-router # FastAPI proxy -> Ollama @ SOLARIA :11434
deployment_model: docker-compose
exposure: private # Tailscale-only bind (TAILSCALE_BIND_IP); no public ingress
offline_required: false
depends_on:
local: []
external: [ollama] # SOLARIA may be powered down -> routes 502, health stays ok
ports:
- name: http
container_port: 8080
protocol: tcp
runtime:
# .env (TAILSCALE_BIND_IP) lives alongside the compose file; stateless, no data path
config_path: services/llm-gateway
feat(kb): faza 3 krok 5 — cykliczny ingest (systemd timer) + alerting documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py): orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic, claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA (availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem. Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors, llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis, last_success_timestamp trzymany z poprzedniego pliku przy failu) do /opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu). systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer (OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do /opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge. fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical) + KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml). node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override (--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) + topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA. Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń, 0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany). Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge), zdecydowanie czy stability-agent też idzie na per-host przy okazji. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 15:05:18 +02:00
node_exporter:
role: metrics-exporter
deployment_model: docker-compose
exposure: local-only
offline_required: true
depends_on:
local: []
external: []
runtime:
# textfile collector reads /opt/homelab/state/node-exporter (module 5 phase 3 step 5,
# docs/kb/modules/05-faza3-plan.md §7.2 — kb-ingest.prom) via the existing /:/host:ro
# mount, see hosts/piha/runtime/node_exporter/docker-compose.override.yml.
data_path: /opt/homelab/state/node-exporter
kb-postgres:
role: kb-database # KB spine: Postgres 16 + pgvector (always-on)
deployment_model: docker-compose
exposure: local-only # Tailscale-accessible; no public ingress
offline_required: false
depends_on:
local: []
external: []
ports:
- name: postgres
host_port: 5433
protocol: tcp
runtime:
config_path: /opt/homelab/config/kb-postgres
# data is in Docker named volume kb_postgres_data — must land on the NVMe
# (Docker data-root on /home); see hosts/piha/runtime/kb-postgres override.