documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py):
orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic,
claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA
(availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu
etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na
kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał
tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla
cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem.
Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors,
llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie
blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis,
last_success_timestamp trzymany z poprzedniego pliku przy failu) do
/opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu).
systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer
(OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User
oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do
/opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w
README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge.
fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical)
+ KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem
brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml).
node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override
(--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) +
topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter
owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent
zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA.
Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run
złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym
taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń,
0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY
dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany).
Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge),
zdecydowanie czy stability-agent też idzie na per-host przy okazji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Kod zyl tylko na dysku PIHA w /opt/llm-gateway (bez gita). Przeniesiony do
services/llm-gateway/ pod pelny wzorzec homelaba:
- app/main.py: kod 1:1 z PIHA + OLLAMA_URL/CHAT_MODEL/CODE_MODEL
nadpisywalne przez env (defaulty bez zmian)
- docker-compose.yml: bind TYLKO do Tailscale IP (${TAILSCALE_BIND_IP},
wzorzec fleet-prometheus), nie 0.0.0.0 jak w starym compose
- service.yaml, env.example (bez sekretow), healthcheck.sh, README, testy
- hosts/piha/runtime/llm-gateway: mem_limit 256m (PIHA jest RAM-bound)
- rejestracja w hosts/piha/services.yaml i inventory/topology.yaml
Zepsuty /opt/llm-gateway/docker-compose.yml (zduplikowany klucz ports)
celowo NIE przeniesiony.
DoD: pytest 4 passed; docker build + smoke run OK (GET / -> gateway ok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Declares the real OS-level hostname for each node so deploy-node.sh
can resolve the correct hosts/ directory without assuming
OS-hostname == logical name. Only VPS differs: its OS hostname is
ubuntu-4gb-hel1-1 while the repo directory is hosts/vps/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wraps the existing manually-run Vikunja instance on PIHA in the standard
service layout. No deploy — definition only, for review.
- services/vikunja/: docker-compose.yml (vikunja + postgres:16-alpine),
service.yaml, README, healthcheck, config.yml (OIDC, secret-free),
.env.example. Real .env stays gitignored.
- Pins EXISTING named volumes (vikunja_vikunja_db, vikunja_vikunja_files)
so DB + uploaded files survive cutover.
- extra_hosts forgejo.okit.pl=192.168.31.5 (npm on PIHA) so OIDC discovery
resolves over LAN instead of flaky public DNS.
- OIDC client secret injected via env (VIKUNJA_AUTH_OPENID_PROVIDERS_
FORGEJO_CLIENTSECRET); config.yml keeps trailing-slash authurl/redirecturl.
- Registers vikunja in hosts/piha/services.yaml + inventory/topology.yaml,
plus hosts/piha/runtime/vikunja override (advisory mem_limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container runs as uid 1000 (homelab), HOME=/home/homelab. ssh without -i
looks for $HOME/.ssh — mounting to /root/.ssh was never visible to the
process user and caused silent Permission denied on event shipping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Polls /summary on VPS over Tailscale every 60s; computes freshness
locally from last_update epoch (never trusts self-reported status).
Alerts via Telegram Bot API directly after 3 consecutive failures;
sends recovery message on heal. State (fail_count, alerted) persisted
to volume so debounce survives restarts.
- services/brain-watchdog/: Python service, no external deps (stdlib only)
- hosts/piha/runtime/brain-watchdog/: override with mem_limit 64m
- hosts/piha/services.yaml + inventory/topology.yaml: manifest entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- new per-host service, follows node-agent pattern
- 7 new HA event types defined (routing in supervisor — Phase 5)
- HeartbeatCheck as pipeline validator (pings /api/, emits ha_websocket_dead)
- service.yaml + host configs for piha (ken) and chelsty-infra (chelsty)
- test scaffolding with aiohttp/aiosqlite mocks (15/15 passing)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
services/agent-system/runtime-materializer/materializer.py:
- Add materialize_from_api() that fetches all world-state endpoints
from the control-plane HTTP API (CONTROL_PLANE_URL env var)
- When CONTROL_PLANE_URL is set, use API as source of truth instead of Redis
- Redis path preserved as fallback for backward compat
hosts/piha/runtime/agent-system/docker-compose.override.yml (new):
- Inject CONTROL_PLANE_URL=http://100.95.58.48:18180 for runtime-materializer
- piha webui /snapshot now mirrors VPS observer output (clean, ghost-free)
Root cause: materializer read from Redis which held 80 stale service entries
with hash-prefixed ghost keys (e.g. 0ccb8a88e079_control-plane-supervisor).
Redis is never updated by the current observer pipeline; the control-plane API
is the single authoritative world-state source.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
100.108.208.3 is piha's Tailscale IP (piha hosts Forgejo+Redis).
VPS's actual Tailscale IP is 100.95.58.48. All three node-agent
overrides were pointing at piha itself, causing containers to SSH
to their own host and fail auth.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes ship events to VPS via rsync+SSH. The container runs as root
and uses the default SSH identity, which must be at /root/.ssh/.
Mount /home/oskar/.ssh from the host read-only so the existing
authorized key is available inside the container.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- piha: NODE_TYPE=sd_card (rate-limited docker prune, once per day)
- solaria: NODE_TYPE=ai_node (dangling+containers+build cache; never -a to preserve Ollama images)
- chelsty-infra: NODE_TYPE=lte_node (NO cleanup, events-only)
- All three: VPS_EVENTS_HOST set for event shipping via rsync+SSH
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala