5f1528e4ab
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by events and never expired, so a node that crashed/lost connectivity stayed "online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing that flipped status to offline was a node_offline event, which an unreachable node can never emit. Now node status is derived from freshness (now - last_seen), recomputed every observer cycle (incl. cycles with no new events): - always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat) - remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s Thresholds + tier logic live in ONE shared helper, services/control-plane/src/ liveness.py, imported by the observer and both operator UIs (bind-mounted into the agent-system webui image). No 3x copy. Transitions are not silent: the observer emits node_stale / node_offline / node_online (recovery) events tagged source=observer (skipped on re-ingest so they never reset last_seen), routed by the supervisor to alert_only actions. Read-time safety net: both UIs recompute liveness from last_seen at request time, so a stalled observer still surfaces dead nodes. Services inherit their node's liveness (cascade, variant B) without mutating services.json. Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip. Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline + self-event skip, operator_ui read-time net + cascade, supervisor node-event routing. 89 passed. docker compose config valid for both stacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| runtime-materializer | ||
| scripts | ||
| telegram-bot | ||
| webui | ||
| action-model.md | ||
| deploy.sh | ||
| docker-compose.yml | ||
| env.example | ||
| README.md | ||
Agent System
Central runtime materializer and Operator Control Plane UI.
Components
- Redis: Central state store (on PIHA).
- Runtime Materializer: Converts Redis state to JSON files in
/opt/homelab/world. - Web UI: Exposes API endpoints and serving the Operator UI.
- Telegram Bot: Provides operator commands and action approvals via Telegram.
Configuration
Environment variables should be set in .env (see env.example).
Key variables for the Telegram Bot:
TELEGRAM_BOT_TOKEN: Your bot token from @BotFather.TELEGRAM_ALLOWED_USER_IDS: Comma-separated list of authorized Telegram User IDs.CONTROL_PLANE_URL: URL to theagent-system-webui(default:http://webui:8080).
Telegram Commands
/status: Check bot and API connectivity./summary: System health overview./nodes: List homelab nodes and their status./services: Summary of services across nodes./unhealthy: List all unhealthy components./incidents: View active incidents./actions: Summary of operator actions./help: List all commands.
Deployment (on PIHA)
cd services/agent-system
./deploy.sh
Deployment (on CHELSTY)
cd services/stability-agent
docker compose up -d --build
Verification
The deploy.sh script automatically verifies the local endpoints.
You can also manually check:
# Check runtime summary
curl http://localhost:18180/summary
# Check discovered nodes
curl http://localhost:18180/nodes
# Check discovered services
curl http://localhost:18180/services
Directory Structure
/opt/homelab/world: Contains materialized JSON state./opt/homelab/state: Contains operator configuration and local heartbeats.