solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.
Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:
- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
dormant_nodes and duty_cycle_nodes from one topology read. dormant
behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
are logged at INFO and return; no action is written.
duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.
Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.
Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.
Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified 2026-07-30 in the vps event store: lustro's node_offline events fire
once a day, ~24 h apart (~23:30 local), matching solaria's scheduled power-off
pattern. Documented so daily stale/offline/online cycles read as expected
behavior, not faults.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
topology.yaml (per its own rule that hosts/*/services.yaml are
authoritative, recon F20.1):
- add status field; chelsty-infra + chelsty-ha -> dormant (site hardware
down since ~2026-06-01, revival planned)
- add lustro as a full active node (runs node-agent, ships events, F20.11)
- drop per-node service lists (vps list contradicted hosts/vps, F20.1;
piha/solaria lists were stale too, F20.8/F20.9) — node-level truth only
- deployment.mode pull -> push: every deploy script SSH-pushes from
saturn (F20.10)
Dormant semantics in code:
- observer (scripts/observer/observer.py): loads status from topology;
_prune_stale_world skips dormant nodes — last-known world state stays
frozen, no node_offline/node_stale/node_online events emitted
- supervisor (services/control-plane/src/supervisor.py): reloads dormant
set each reconcile; dormant hosts' services excluded from desired state
(existing pending actions auto-cancel via
service_removed_from_desired_state), disk_cleanup skipped, node/HA
events from dormant nodes not routed to alerts
Tests: services/control-plane/tests/test_dormant_nodes.py (9 cases);
full suite 145 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
documents-ingest-cyclic (jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py):
orkiestruje paperless_adapter -> chunk_embed -> summarize(--backend anthropic,
claude-haiku-4-5) -> summarize(--embed-summaries) bez zmian w samych jobach. Ollama@SOLARIA
(availability_target: medium) jest tolerowana offline: probe GET /api/tags przed obu
etapami embed, brak -> pominięcie, nie fail (oba embed passy idempotentne, nadrobią się na
kolejnym ticku). Czwarty etap (embed-summaries) dopisany ponad plan §7.1 (który wymieniał
tylko 3 kroki) — bez niego nowe streszczenia miałyby embedding=NULL i byłyby niewidoczne dla
cascade_query (bramka kroku 4, WHERE embedding IS NOT NULL); potwierdzone z Oskarem.
Predykaty pass/fail każdego etapu 1:1 z exit-checkiem danego joba (chunks_errors,
llm_errors, stats-balance itd.) — etapy izolowane, nie fail-fast (wcześniejszy fail nie
blokuje kolejnych, tak jak joby izolują błędy per wiersz). Metryki .prom (atomowy zapis,
last_success_timestamp trzymany z poprzedniego pliku przy failu) do
/opt/homelab/state/node-exporter/kb-ingest.prom. 36 nowych testów (202/202 pakietu).
systemd (jobs/documents-ingest/systemd/): pierwszy systemd-timer w repo — kb-ingest.timer
(OnCalendar=*-*-* 03:30, Persistent=true, plan §7.1) + kb-ingest.service (host-level, User
oskar, EnvironmentFile /opt/homelab/kb/.env) + kb-ingest-run.sh (log per-run do
/opt/homelab/logs/kb-ingest/, konwencja repo). Instalacja i sekrety udokumentowane w
README (Faza 3 krok 5) — instalacja na PIHA dopiero po merge.
fleet-prometheus (rules/kb-ingest.yml): KbIngestStale (>172800s od last_success, critical)
+ KbEmbedBacklogGrowing (backlog>0 przez 72h, warning) — dostawa istniejącym torem
brain-watchdog->Telegram, bez Alertmanagera (konwencja liveness.yml).
node_exporter: owner_node vps -> per-host (service.yaml) + wpis + override
(--collector.textfile.directory, bez nowego mountu — czyta przez istniejący /:/host:ro) +
topology.yaml dla PIHA. Domyka pozycję z docs/backlog.md "stability-agent / node_exporter
owner_node single, biegaja wielomiejscowo -> per-host" (połowę — node_exporter; stability-agent
zostaje osobnym follow-upem) w ramach paczki B inwentaryzacji monitoringu dla PIHA.
Test end-to-end na żywo na PIHA (2× --apply, po potwierdzeniu z Oskarem): pierwszy run
złapał 5 dokumentów faktycznie nowych w Paperless (nieoczekiwane, niezwiązane z tym
taskiem) -> 82 nowe chunki (2 ocr_junk), 5 nowych streszczeń, 5 embeddingów streszczeń,
0 błędów, metryki zapisane. Drugi run: pełna idempotencja, wszystko 0. ANTHROPIC_API_KEY
dodany przez Oskara ręcznie do /opt/homelab/kb/.env (nigdy nie logowany/generowany).
Co dalej: prawdziwa instalacja systemd (services.yaml już przygotowany, po merge),
zdecydowanie czy stability-agent też idzie na per-host przy okazji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Nextcloud zostaje prywatny (mesh/kapala.org); Gokapi to osobny publiczny
serwis do wysylania linkow do plikow na zewnatrz (Firefox Send alt).
owner_node=vps, storage lokalny dysk (nie S3), E2E encryption ON, port
53842 bindowany tylko na TAILSCALE_BIND_IP (hairpin NAT przez npm@VPS,
nigdy 0.0.0.0). Cutover checklist w README (DNS, wildcard *.okit.pl na
npm@VPS, setup wizard) — nie zdeployowane w tym commicie.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Kod zyl tylko na dysku PIHA w /opt/llm-gateway (bez gita). Przeniesiony do
services/llm-gateway/ pod pelny wzorzec homelaba:
- app/main.py: kod 1:1 z PIHA + OLLAMA_URL/CHAT_MODEL/CODE_MODEL
nadpisywalne przez env (defaulty bez zmian)
- docker-compose.yml: bind TYLKO do Tailscale IP (${TAILSCALE_BIND_IP},
wzorzec fleet-prometheus), nie 0.0.0.0 jak w starym compose
- service.yaml, env.example (bez sekretow), healthcheck.sh, README, testy
- hosts/piha/runtime/llm-gateway: mem_limit 256m (PIHA jest RAM-bound)
- rejestracja w hosts/piha/services.yaml i inventory/topology.yaml
Zepsuty /opt/llm-gateway/docker-compose.yml (zduplikowany klucz ports)
celowo NIE przeniesiony.
DoD: pytest 4 passed; docker build + smoke run OK (GET / -> gateway ok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- hosts/vps/services.yaml: fleet-prometheus entry (role: fleet-liveness-source,
exposure: tailscale-internal, port 9090, runtime paths, depends_on node_exporter)
- inventory/topology.yaml: add fleet-prometheus to the vps services list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wraps the existing manually-run Vikunja instance on PIHA in the standard
service layout. No deploy — definition only, for review.
- services/vikunja/: docker-compose.yml (vikunja + postgres:16-alpine),
service.yaml, README, healthcheck, config.yml (OIDC, secret-free),
.env.example. Real .env stays gitignored.
- Pins EXISTING named volumes (vikunja_vikunja_db, vikunja_vikunja_files)
so DB + uploaded files survive cutover.
- extra_hosts forgejo.okit.pl=192.168.31.5 (npm on PIHA) so OIDC discovery
resolves over LAN instead of flaky public DNS.
- OIDC client secret injected via env (VIKUNJA_AUTH_OPENID_PROVIDERS_
FORGEJO_CLIENTSECRET); config.yml keeps trailing-slash authurl/redirecturl.
- Registers vikunja in hosts/piha/services.yaml + inventory/topology.yaml,
plus hosts/piha/runtime/vikunja override (advisory mem_limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Polls /summary on VPS over Tailscale every 60s; computes freshness
locally from last_update epoch (never trusts self-reported status).
Alerts via Telegram Bot API directly after 3 consecutive failures;
sends recovery message on heal. State (fail_count, alerted) persisted
to volume so debounce survives restarts.
- services/brain-watchdog/: Python service, no external deps (stdlib only)
- hosts/piha/runtime/brain-watchdog/: override with mem_limit 64m
- hosts/piha/services.yaml + inventory/topology.yaml: manifest entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala