71eaab0025
solaria (powered off ~16 h/day by design) and lustro (nightly display power-off) generated node_offline/node_stale/node_online alerts on every daily cycle. Six of them have sat in actions/pending/ since 2026-06-17/18, unapproved. Because an unapproved pending action suppresses its own dedup ID indefinitely (recon D14, supervisor.py pending/approved/running check), those stale alerts also meant a *real* future outage on either node would generate nothing at all. Suppression is data-driven from inventory/topology.yaml, not a hardcoded node-name check: - topology.yaml: new `duty_cycle` (+ `duty_cycle_reason`) on solaria and lustro, mirroring the existing dormant/dormant_reason shape. vps and piha deliberately do not carry it — an offline 24/7 node is a real incident. - supervisor: _load_dormant_nodes() -> _load_node_policy(), loading both dormant_nodes and duty_cycle_nodes from one topology read. dormant behavior is byte-for-byte unchanged. - supervisor: one guard in _route_node_event. Duty-cycle liveness events are logged at INFO and return; no action is written. duty_cycle is deliberately NOT dormant. A duty-cycle node stays fully active: its services are still reconciled (missing_service -> redeploy), its disk pressure still generates disk_cleanup, and its ha_* events still route. Only the liveness alert is suppressed. Regression tests pin all three. Fail-loud: an unreadable topology leaves both sets empty, which disables suppression and lets alerts through. A broken topology must never silently mute the fleet. Accepted trade-off: a genuine permanent outage of solaria or lustro no longer alerts. It stays visible in the operator UI (which computes liveness independently at read time) and in the event feed. An "offline longer than the expected window" escalation is the natural follow-up and needs a schedule in the topology field rather than a bare marker. Tests: 169 passed in services/control-plane/tests (was 157; +12). Runtime deployment is deliberately NOT part of this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .claude/skills | ||
| backups/zigbee | ||
| docs/sessions | ||
| dotfiles | ||
| hardware/esp/ir-ac-ha-integration | ||
| hosts | ||
| inventory | ||
| jobs | ||
| kb | ||
| packages | ||
| scripts | ||
| services | ||
| .codex | ||
| .gitignore | ||
| .mcp.json | ||
| CLAUDE.md | ||
| codex_context | ||
| codex_context.yaml | ||
| deploy_agent.py | ||
| ollama_client.py | ||
| README.md | ||
| start-aider.sh | ||
| start-codex.sh | ||
| sync-context.sh | ||
| update-context.md | ||
Homelab Codex
GitOps-lite orchestration for a distributed homelab environment.
Architecture
The homelab consists of several nodes connected via a Tailscale internal mesh.
| Host | Role | Description |
|---|---|---|
| SATURN | Primary Node | Development, orchestration, and git source of truth (commit node). |
| SOLARIA | Compute Node | GPU, inference, and heavy compute workloads. |
| PIHA | Infra Node | Core infrastructure services, automation, and monitoring. |
| VPS | Edge Node | Public ingress, reverse proxy, and edge services. |
Agent System
The homelab uses a multi-agent orchestration model with human-in-the-loop for destructive actions:
| Agent | Node | Role |
|---|---|---|
| stability-agent | all nodes | Per-node watchdog — monitors Docker, disk, Tailscale, MQTT; emits events |
| node-agent | all nodes | Publishes container health events to Redis pub/sub |
| observer | VPS | Synthesizes world state from events into /opt/homelab/world/*.json |
| supervisor | VPS | Detects drift between desired and actual state; writes pending actions |
| planner-agent | SOLARIA | LLM-powered diagnosis — listens to Redis, proposes remediation actions |
| executor | VPS | Executes actions only after operator approval |
| operator-ui + telegram-bot | VPS / PIHA | Operator reviews and approves/rejects pending actions |
Action approval flow: pending/ → operator approves → approved/ → executor runs.
Repository Structure
docs/: Infrastructure Standards and Deployment Conventions.kb/phases/subsystem-a-naprawa.md: Current Maintenance Plan (Control Plane).hosts/: Host-specific configurations and service assignments.services/: Reusable Docker Compose service definitions.scripts/: Deployment and management scripts.
Getting Started
- Standardization: Follow the Infrastructure Standards.
- Deployment: See Deployment Conventions for how to roll out changes.
- SATURN: Remember that SATURN is the only node where commits should be made.
Documentation Index
- Current Maintenance Plan (Control Plane)
- Infrastructure Standards
- Agent Operating Procedures (For AI/Non-Human Agents)
- Deployment Conventions
- Hardware
- Networking
- Services
- Node Capabilities
- Action Model
Note: This repository documents the state of the homelab. Runtime state lives outside the repository in /opt/homelab.