Commit graph

1 commit

Author SHA1 Message Date
oskar 71eaab0025 feat(supervisor): duty-cycle nodes — liveness transitions logged, not actioned
solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.

Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:

- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
  lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
  deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
  dormant_nodes and duty_cycle_nodes from one topology read. dormant
  behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
  are logged at INFO and return; no action is written.

duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.

Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.

Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.

Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:48:29 +02:00