2026-05-10 22:05:16 +02:00
|
|
|
topology:
|
|
|
|
|
mesh: tailscale
|
|
|
|
|
git_provider: forgejo
|
|
|
|
|
|
|
|
|
|
deployment:
|
2026-07-29 18:47:01 +02:00
|
|
|
# Every deploy script SSH-pushes from saturn to the target node (recon
|
|
|
|
|
# F20.10); "pull" never matched reality.
|
|
|
|
|
mode: push
|
2026-05-10 22:05:16 +02:00
|
|
|
orchestrator: saturn
|
|
|
|
|
|
2026-06-30 16:23:57 +02:00
|
|
|
# Ingress dla usług domowych przez NPM @ PIHA + wildcard cert kapala.org.
|
|
|
|
|
# Od 2026-06-30. Poprzednia ścieżka (add-on Tailscale na ha-ken) PORZUCONA.
|
|
|
|
|
# DNS kapala.org: Cloudflare, wildcard *.kapala.org via DNS-01 (auto-renew).
|
|
|
|
|
# Mail: Fastmail (MX/DKIM/SPF na CF, DNS only).
|
|
|
|
|
# Subdomeny mesh: A → 100.108.208.3 (Tailscale PIHA), DNS only.
|
|
|
|
|
ingress:
|
|
|
|
|
dns_provider: cloudflare # kapala.org od 2026-06-30
|
|
|
|
|
wildcard_cert:
|
|
|
|
|
domain: "*.kapala.org"
|
|
|
|
|
method: dns-01 # Cloudflare DNS-01, auto-renew
|
|
|
|
|
proxy: npm-piha # Nginx Proxy Manager @ PIHA
|
|
|
|
|
services:
|
|
|
|
|
home_assistant:
|
|
|
|
|
host: 192.168.31.7
|
|
|
|
|
port: 8123
|
|
|
|
|
ingress: ha.kapala.org # NPM @ PIHA, cert *.kapala.org, mesh-only
|
|
|
|
|
access: tailscale # 100.108.208.3
|
|
|
|
|
cert: wildcard-kapala # DNS-01 Cloudflare, auto-renew
|
|
|
|
|
immich:
|
|
|
|
|
host: 192.168.31.5
|
|
|
|
|
port: 2283
|
|
|
|
|
ingress: immich.kapala.org # NPM @ PIHA, cert *.kapala.org, mesh-only
|
|
|
|
|
access: tailscale
|
|
|
|
|
cert: wildcard-kapala
|
|
|
|
|
|
2026-07-29 18:47:01 +02:00
|
|
|
# Node-level truth ONLY. Per-node service lists live in hosts/<node>/services.yaml
|
|
|
|
|
# (authoritative — the lists formerly duplicated here contradicted them, recon F20.1).
|
|
|
|
|
#
|
|
|
|
|
# status:
|
|
|
|
|
# active — node is monitored and remediated normally.
|
|
|
|
|
# dormant — node exists but is expected offline: the observer keeps its
|
|
|
|
|
# last-known world state and emits NO liveness events for it; the
|
|
|
|
|
# supervisor generates NO actions for it. See ARCHITEKTURA.md.
|
feat(supervisor): duty-cycle nodes — liveness transitions logged, not actioned
solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.
Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:
- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
dormant_nodes and duty_cycle_nodes from one topology read. dormant
behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
are logged at INFO and return; no action is written.
duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.
Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.
Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.
Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:48:29 +02:00
|
|
|
#
|
|
|
|
|
# duty_cycle:
|
|
|
|
|
# Present (any non-empty value) => the node powers down on a schedule BY
|
|
|
|
|
# DESIGN. Its node_offline / node_stale / node_online events are still
|
|
|
|
|
# emitted by the observer and still appear in the event feed, but the
|
|
|
|
|
# supervisor LOGS them instead of generating an alert_only action.
|
|
|
|
|
# Everything else is unchanged: the node is `active`, its services are
|
|
|
|
|
# reconciled and its disk pressure is acted on exactly like a 24/7 node —
|
|
|
|
|
# duty_cycle is NOT dormant. Absent => a liveness transition is a real
|
|
|
|
|
# incident (vps, piha).
|
2026-05-10 22:05:16 +02:00
|
|
|
nodes:
|
|
|
|
|
|
|
|
|
|
saturn:
|
2026-07-29 18:47:01 +02:00
|
|
|
status: active
|
2026-05-10 22:05:16 +02:00
|
|
|
roles:
|
|
|
|
|
- control
|
|
|
|
|
- development
|
|
|
|
|
|
|
|
|
|
piha:
|
2026-07-29 18:47:01 +02:00
|
|
|
status: active
|
2026-05-10 22:05:16 +02:00
|
|
|
roles:
|
|
|
|
|
- infra
|
|
|
|
|
- monitoring
|
|
|
|
|
|
|
|
|
|
solaria:
|
2026-07-29 18:47:01 +02:00
|
|
|
# Powered off ~16 h/day BY DESIGN (on-demand compute desktop): one daily
|
|
|
|
|
# offline/online liveness cycle is expected, not a fault. Nothing that
|
|
|
|
|
# must run 24/7 may live here.
|
|
|
|
|
status: active
|
feat(supervisor): duty-cycle nodes — liveness transitions logged, not actioned
solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.
Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:
- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
dormant_nodes and duty_cycle_nodes from one topology read. dormant
behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
are logged at INFO and return; no action is written.
duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.
Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.
Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.
Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:48:29 +02:00
|
|
|
duty_cycle: nightly
|
|
|
|
|
duty_cycle_reason: "on-demand compute desktop, powered off ~16 h/day"
|
2026-05-10 22:05:16 +02:00
|
|
|
roles:
|
|
|
|
|
- compute
|
|
|
|
|
- ai
|
|
|
|
|
|
|
|
|
|
vps:
|
2026-07-29 18:47:01 +02:00
|
|
|
status: active
|
2026-05-10 22:05:16 +02:00
|
|
|
roles:
|
|
|
|
|
- edge
|
|
|
|
|
- ingress
|
2026-05-21 11:40:55 +02:00
|
|
|
- control-plane
|
2026-07-29 18:47:01 +02:00
|
|
|
|
|
|
|
|
lustro:
|
|
|
|
|
# MagicMirror Raspberry Pi — full monitored node: runs node-agent and
|
|
|
|
|
# ships events to the control plane (recon F20.11).
|
2026-07-30 15:36:13 +02:00
|
|
|
# Duty cycle (checked 2026-07-30 in the vps event store): node_offline
|
|
|
|
|
# fires once a day, ~24 h apart, around 23:30 local — a scheduled nightly
|
|
|
|
|
# display power-off like solaria's. Daily stale/offline/online liveness
|
|
|
|
|
# cycles are expected here, not faults.
|
2026-07-29 18:47:01 +02:00
|
|
|
status: active
|
feat(supervisor): duty-cycle nodes — liveness transitions logged, not actioned
solaria (powered off ~16 h/day by design) and lustro (nightly display
power-off) generated node_offline/node_stale/node_online alerts on every
daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18,
unapproved. Because an unapproved pending action suppresses its own dedup
ID indefinitely (recon D14, supervisor.py pending/approved/running check),
those stale alerts also meant a *real* future outage on either node would
generate nothing at all.
Suppression is data-driven from inventory/topology.yaml, not a hardcoded
node-name check:
- topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and
lustro, mirroring the existing dormant/dormant_reason shape. vps and piha
deliberately do not carry it — an offline 24/7 node is a real incident.
- supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both
dormant_nodes and duty_cycle_nodes from one topology read. dormant
behavior is byte-for-byte unchanged.
- supervisor: one guard in _route_node_event. Duty-cycle liveness events
are logged at INFO and return; no action is written.
duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully
active: its services are still reconciled (missing_service -> redeploy),
its disk pressure still generates disk_cleanup, and its ha_* events still
route. Only the liveness alert is suppressed. Regression tests pin all
three.
Fail-loud: an unreadable topology leaves both sets empty, which disables
suppression and lets alerts through. A broken topology must never silently
mute the fleet.
Accepted trade-off: a genuine permanent outage of solaria or lustro no
longer alerts. It stays visible in the operator UI (which computes liveness
independently at read time) and in the event feed. An "offline longer than
the expected window" escalation is the natural follow-up and needs a
schedule in the topology field rather than a bare marker.
Tests: 169 passed in services/control-plane/tests (was 157; +12).
Runtime deployment is deliberately NOT part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:48:29 +02:00
|
|
|
duty_cycle: nightly
|
|
|
|
|
duty_cycle_reason: "MagicMirror display power-off ~23:30 local (verified 2026-07-30)"
|
2026-07-29 18:47:01 +02:00
|
|
|
roles:
|
|
|
|
|
- edge
|
|
|
|
|
- display
|
2026-05-10 22:05:16 +02:00
|
|
|
|
2026-05-20 14:23:57 +02:00
|
|
|
chelsty-infra:
|
2026-07-29 18:47:01 +02:00
|
|
|
status: dormant
|
|
|
|
|
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
|
|
|
|
dormant_since: "2026-06-01"
|
2026-05-20 14:23:57 +02:00
|
|
|
site: chelsty
|
2026-05-10 22:05:16 +02:00
|
|
|
roles:
|
2026-05-11 13:26:23 +02:00
|
|
|
- remote
|
2026-05-10 22:05:16 +02:00
|
|
|
- hypervisor
|
2026-05-20 14:23:57 +02:00
|
|
|
- infra
|
2026-05-10 22:05:16 +02:00
|
|
|
- staging
|
2026-05-11 13:26:23 +02:00
|
|
|
connectivity:
|
|
|
|
|
uplink: lte
|
|
|
|
|
intermittent: true
|
|
|
|
|
home_automation:
|
|
|
|
|
offline_operation_required: true
|
|
|
|
|
coordinator:
|
|
|
|
|
model: SLZB-06U
|
|
|
|
|
connection: network
|
|
|
|
|
usb: false
|
2026-05-20 14:23:57 +02:00
|
|
|
|
|
|
|
|
chelsty-ha:
|
2026-07-29 18:47:01 +02:00
|
|
|
status: dormant
|
|
|
|
|
dormant_reason: "site hardware down since ~2026-06-01; will be revived later"
|
|
|
|
|
dormant_since: "2026-06-01"
|
2026-05-20 14:23:57 +02:00
|
|
|
site: chelsty
|
|
|
|
|
roles:
|
|
|
|
|
- remote
|
|
|
|
|
- homeassistant
|
|
|
|
|
connectivity:
|
|
|
|
|
uplink: lte
|
|
|
|
|
intermittent: true
|
|
|
|
|
home_automation:
|
|
|
|
|
offline_operation_required: true
|
