71eaab0025
solaria (powered off ~16 h/day by design) and lustro (nightly display power-off) generated node_offline/node_stale/node_online alerts on every daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18, unapproved. Because an unapproved pending action suppresses its own dedup ID indefinitely (recon D14, supervisor.py pending/approved/running check), those stale alerts also meant a *real* future outage on either node would generate nothing at all. Suppression is data-driven from inventory/topology.yaml, not a hardcoded node-name check: - topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and lustro, mirroring the existing dormant/dormant_reason shape. vps and piha deliberately do not carry it — an offline 24/7 node is a real incident. - supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both dormant_nodes and duty_cycle_nodes from one topology read. dormant behavior is byte-for-byte unchanged. - supervisor: one guard in _route_node_event. Duty-cycle liveness events are logged at INFO and return; no action is written. duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully active: its services are still reconciled (missing_service -> redeploy), its disk pressure still generates disk_cleanup, and its ha_* events still route. Only the liveness alert is suppressed. Regression tests pin all three. Fail-loud: an unreadable topology leaves both sets empty, which disables suppression and lets alerts through. A broken topology must never silently mute the fleet. Accepted trade-off: a genuine permanent outage of solaria or lustro no longer alerts. It stays visible in the operator UI (which computes liveness independently at read time) and in the event feed. An "offline longer than the expected window" escalation is the natural follow-up and needs a schedule in the topology field rather than a bare marker. Tests: 169 passed in services/control-plane/tests (was 157; +12). Runtime deployment is deliberately NOT part of this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
133 lines
4.2 KiB
YAML
133 lines
4.2 KiB
YAML
topology:
|
|
mesh: tailscale
|
|
git_provider: forgejo
|
|
|
|
deployment:
|
|
# Every deploy script SSH-pushes from saturn to the target node (recon
|
|
# F20.10); "pull" never matched reality.
|
|
mode: push
|
|
orchestrator: saturn
|
|
|
|
# Ingress dla usług domowych przez NPM @ PIHA + wildcard cert kapala.org.
|
|
# Od 2026-06-30. Poprzednia ścieżka (add-on Tailscale na ha-ken) PORZUCONA.
|
|
# DNS kapala.org: Cloudflare, wildcard *.kapala.org via DNS-01 (auto-renew).
|
|
# Mail: Fastmail (MX/DKIM/SPF na CF, DNS only).
|
|
# Subdomeny mesh: A → 100.108.208.3 (Tailscale PIHA), DNS only.
|
|
ingress:
|
|
dns_provider: cloudflare # kapala.org od 2026-06-30
|
|
wildcard_cert:
|
|
domain: "*.kapala.org"
|
|
method: dns-01 # Cloudflare DNS-01, auto-renew
|
|
proxy: npm-piha # Nginx Proxy Manager @ PIHA
|
|
services:
|
|
home_assistant:
|
|
host: 192.168.31.7
|
|
port: 8123
|
|
ingress: ha.kapala.org # NPM @ PIHA, cert *.kapala.org, mesh-only
|
|
access: tailscale # 100.108.208.3
|
|
cert: wildcard-kapala # DNS-01 Cloudflare, auto-renew
|
|
immich:
|
|
host: 192.168.31.5
|
|
port: 2283
|
|
ingress: immich.kapala.org # NPM @ PIHA, cert *.kapala.org, mesh-only
|
|
access: tailscale
|
|
cert: wildcard-kapala
|
|
|
|
# Node-level truth ONLY. Per-node service lists live in hosts/<node>/services.yaml
|
|
# (authoritative — the lists formerly duplicated here contradicted them, recon F20.1).
|
|
#
|
|
# status:
|
|
# active — node is monitored and remediated normally.
|
|
# dormant — node exists but is expected offline: the observer keeps its
|
|
# last-known world state and emits NO liveness events for it; the
|
|
# supervisor generates NO actions for it. See ARCHITEKTURA.md.
|
|
#
|
|
# duty_cycle:
|
|
# Present (any non-empty value) => the node powers down on a schedule BY
|
|
# DESIGN. Its node_offline / node_stale / node_online events are still
|
|
# emitted by the observer and still appear in the event feed, but the
|
|
# supervisor LOGS them instead of generating an alert_only action.
|
|
# Everything else is unchanged: the node is `active`, its services are
|
|
# reconciled and its disk pressure is acted on exactly like a 24/7 node —
|
|
# duty_cycle is NOT dormant. Absent => a liveness transition is a real
|
|
# incident (vps, piha).
|
|
nodes:
|
|
|
|
saturn:
|
|
status: active
|
|
roles:
|
|
- control
|
|
- development
|
|
|
|
piha:
|
|
status: active
|
|
roles:
|
|
- infra
|
|
- monitoring
|
|
|
|
solaria:
|
|
# Powered off ~16 h/day BY DESIGN (on-demand compute desktop): one daily
|
|
# offline/online liveness cycle is expected, not a fault. Nothing that
|
|
# must run 24/7 may live here.
|
|
status: active
|
|
duty_cycle: nightly
|
|
duty_cycle_reason: "on-demand compute desktop, powered off ~16 h/day"
|
|
roles:
|
|
- compute
|
|
- ai
|
|
|
|
vps:
|
|
status: active
|
|
roles:
|
|
- edge
|
|
- ingress
|
|
- control-plane
|
|
|
|
lustro:
|
|
# MagicMirror Raspberry Pi — full monitored node: runs node-agent and
|
|
# ships events to the control plane (recon F20.11).
|
|
# Duty cycle (checked 2026-07-30 in the vps event store): node_offline
|
|
# fires once a day, ~24 h apart, around 23:30 local — a scheduled nightly
|
|
# display power-off like solaria's. Daily stale/offline/online liveness
|
|
# cycles are expected here, not faults.
|
|
status: active
|
|
duty_cycle: nightly
|
|
duty_cycle_reason: "MagicMirror display power-off ~23:30 local (verified 2026-07-30)"
|
|
roles:
|
|
- edge
|
|
- display
|
|
|
|
chelsty-infra:
|
|
status: dormant
|
|
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
|
dormant_since: "2026-06-01"
|
|
site: chelsty
|
|
roles:
|
|
- remote
|
|
- hypervisor
|
|
- infra
|
|
- staging
|
|
connectivity:
|
|
uplink: lte
|
|
intermittent: true
|
|
home_automation:
|
|
offline_operation_required: true
|
|
coordinator:
|
|
model: SLZB-06U
|
|
connection: network
|
|
usb: false
|
|
|
|
chelsty-ha:
|
|
status: dormant
|
|
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
|
dormant_since: "2026-06-01"
|
|
site: chelsty
|
|
roles:
|
|
- remote
|
|
- homeassistant
|
|
connectivity:
|
|
uplink: lte
|
|
intermittent: true
|
|
home_automation:
|
|
offline_operation_required: true
|