f2ba81bcc1
An "unknown"/empty node_name silently produced evt-unknown-* event files. One landed in events/piha/ and, being lexically greater than every evt-piha-* name, pinned the observer's (then lexical) checkpoint and blocked PIHA for ~34 days. Defense in depth, two layers, both refuse rather than emit poison: - config.py: node_name field_validator rejects ""/"unknown" (case-insensitive, trimmed); validate_default=True so the "unknown" default itself is rejected when NODE_NAME never reaches the process. main() catches ValidationError and exits 1 with a clear FATAL message instead of a raw traceback. - event_emitter.py: EventEmitter.__init__ guards node_name at the exact poison site (node_name is embedded in the filename), so no future call path can reintroduce evt-unknown-*. Precedence unchanged and correct: Settings.load() uses os.environ.setdefault, so env NODE_NAME wins over YAML; YAML only supplies node_name when env is absent. api.py never writes event files (only /health, /trigger), so its _node_name global cannot produce poison — verified, left as-is. Tests (services/ha-diag-agent/tests/): new test_config.py (env load, env>YAML precedence, fail-fast on unknown/empty/whitespace/default/YAML-unknown) and EventEmitter guard tests. Full unit suite: 139 passed, 0 regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| src/ha_diag | ||
| tests | ||
| DEPLOY.md | ||
| docker-compose.yml | ||
| Dockerfile | ||
| env.example | ||
| healthcheck.sh | ||
| pyproject.toml | ||
| README.md | ||
| service.yaml | ||
ha-diag-agent
Per-host Home Assistant diagnostic agent. Polls HA REST API on a schedule,
emits structured events to /opt/homelab/events/<node>/, and exposes an
HTTP API for health checks and manual check triggers.
Follows the same event-pipeline pattern as node-agent: filesystem-first,
no direct supervisor integration, events processed by the VPS observer.
Architecture
APScheduler (interval-based REST checks)
├─ HeartbeatCheck → pings /api/, emits ha_websocket_dead on failure
├─ UnavailableEntitiesCheck → entity unavailable > threshold
├─ SystemHealthCheck → /api/system_health per-integration status
├─ AutomationFailuresCheck → automation last-run error traces
└─ UpdatesAvailableCheck → pending HA/integration updates
WebSocketMonitor (persistent, long-running — Phase 4b)
└─ Maintains a live WS subscription to state_changed events
Any traffic = HA is alive. Watchdog fires ha_websocket_dead on
silence > 5min or on disconnect. Emits ha_websocket_recovered
when the connection is restored after a dead alert.
FastAPI (port 8087, internal only — no host port mapping)
GET /health → liveness probe (includes ws_connected field)
POST /trigger/<check> → run a named check on demand
SQLite (/data/ha_diag.db)
entity_baseline → last-known entity states
check_history → per-check run log
alerts_sent → dedup gate for alert events
The WebSocketMonitor is the only persistent-connection component; all other checks are APScheduler intervals (stateless REST polls).
Event Types
| Type | Severity | Trigger |
|---|---|---|
ha_websocket_dead |
error | WS disconnect, silence > 5min, or /api/ unreachable |
ha_websocket_recovered |
info | WS reconnected after a dead alert (clears incident) |
ha_integration_failed |
error | Integration in error state |
ha_entity_unavailable_long |
warning | Entity unavailable > threshold |
ha_automation_failing |
warning | Automation last run errored |
ha_update_available |
info | HA or integration update pending |
ha_recorder_lag |
warning | Recorder write lag > threshold |
ha_system_health_degraded |
warning | System health check failed |
Event routing in supervisor (Phase 5) maps these to notify actions.
ha_websocket_recovered should be routed to clear any active ha_websocket_dead incident.
First-time deployment
See DEPLOY.md for the full procedure: HA token creation,
per-host .env config, deploy commands, verification steps, 48h shadow-mode
observation, and rollback.
Shadow mode (HA_DIAG_SHADOW_MODE, default true on the control-plane):
ha_websocket_dead events are downgraded to alert_only with a [SHADOW MODE]
note instead of queuing an automatic container_restart. Set to false in
/opt/homelab/config/control-plane/.env on the VPS when ready for live actions.
Deployment model
The agent is deployed per-host but targets a potentially remote HA instance:
| Node | Agent runs on | HA lives on | HA URL |
|---|---|---|---|
| piha | piha | piha (localhost) | http://localhost:8123 |
| chelsty-infra | chelsty-infra | chelsty-ha (HAOS VM, separate machine) | http://100.70.180.90:8123 |
chelsty-infra note: Home Assistant runs on chelsty-ha, a dedicated Home Assistant
OS VM. chelsty-infra is the hypervisor but does not run HA itself. The agent on
chelsty-infra reaches HA over the Tailscale network (100.70.180.90:8123). If chelsty-ha
gets a new Tailscale IP, update HA_URL in /opt/homelab/config/ha-diag-agent/.env on
chelsty-infra.
Deployment
# 1. Create config on target node
ssh oskar@<node-ip>
mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent
cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF'
HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra
HA_TOKEN=<long-lived-token>
NODE_NAME=piha # or chelsty-infra
LOCATION_TAG=ken # or chelsty
CHECK_INTERVAL=60
EOF
# 2. Deploy
scripts/deploy/deploy.sh --service ha-diag-agent
# 3. Verify
docker ps --filter name=ha-diag-agent
docker exec ha-diag-agent python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8087/health', timeout=5).read().decode())"
chelsty-infra note
chelsty-infra runs docker-compose v1 (1.29.2). Use docker-compose (hyphenated):
docker-compose -f docker-compose.yml up -d --build
HA long-lived token
In HA UI: Profile → Long-Lived Access Tokens → Create token.
Running Tests
cd services/ha-diag-agent
pip install -e ".[dev]"
pytest tests/ -v
Optional YAML config
Place /opt/homelab/config/ha-diag-agent/ha-diag-agent.yaml on the node.
Values there are defaults; env vars take priority.
ha_url: http://homeassistant.local:8123
location_tag: ken
check_interval: 60