5f1528e4ab
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by events and never expired, so a node that crashed/lost connectivity stayed "online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing that flipped status to offline was a node_offline event, which an unreachable node can never emit. Now node status is derived from freshness (now - last_seen), recomputed every observer cycle (incl. cycles with no new events): - always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat) - remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s Thresholds + tier logic live in ONE shared helper, services/control-plane/src/ liveness.py, imported by the observer and both operator UIs (bind-mounted into the agent-system webui image). No 3x copy. Transitions are not silent: the observer emits node_stale / node_offline / node_online (recovery) events tagged source=observer (skipped on re-ingest so they never reset last_seen), routed by the supervisor to alert_only actions. Read-time safety net: both UIs recompute liveness from last_seen at request time, so a stalled observer still surfaces dead nodes. Services inherit their node's liveness (cascade, variant B) without mutating services.json. Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip. Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline + self-event skip, operator_ui read-time net + cascade, supervisor node-event routing. 89 passed. docker compose config valid for both stacks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
1.5 KiB
YAML
52 lines
1.5 KiB
YAML
services:
|
|
redis:
|
|
image: redis:7
|
|
container_name: agent-system-redis
|
|
ports:
|
|
- "6379:6379"
|
|
restart: unless-stopped
|
|
|
|
webui:
|
|
build: ./webui
|
|
container_name: agent-system-webui
|
|
ports:
|
|
- "18180:8080"
|
|
volumes:
|
|
- /opt/homelab:/opt/homelab
|
|
# Shared liveness logic — single source of truth, bind-mounted read-only
|
|
# so this image does not carry a second copy of the thresholds/tier logic.
|
|
# web.py imports it defensively; absence degrades to observer status only.
|
|
- ../control-plane/src/liveness.py:/app/liveness.py:ro
|
|
depends_on:
|
|
- redis
|
|
restart: unless-stopped
|
|
|
|
runtime-materializer:
|
|
build: ./runtime-materializer
|
|
container_name: agent-system-runtime-materializer
|
|
environment:
|
|
REDIS_HOST: redis
|
|
REDIS_PORT: "6379"
|
|
HOMELAB_WORLD_ROOT: /opt/homelab/world
|
|
WORLD_DIR: /opt/homelab/world
|
|
MATERIALIZE_INTERVAL: "10"
|
|
volumes:
|
|
- /opt/homelab:/opt/homelab
|
|
depends_on:
|
|
- redis
|
|
restart: unless-stopped
|
|
|
|
telegram-bot:
|
|
build: ./telegram-bot
|
|
container_name: agent-system-telegram-bot
|
|
environment:
|
|
TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
|
|
TELEGRAM_ALLOWED_USER_IDS: ${TELEGRAM_ALLOWED_USER_IDS}
|
|
CONTROL_PLANE_URL: ${CONTROL_PLANE_URL:-http://webui:8080}
|
|
ENABLE_LLM_FALLBACK: ${ENABLE_LLM_FALLBACK:-false}
|
|
OPENCLAW_BASE_URL: ${OPENCLAW_BASE_URL}
|
|
ACTIONS_ROOT: /opt/homelab/actions
|
|
volumes:
|
|
- /opt/homelab:/opt/homelab
|
|
restart: on-failure
|