Oskar Kapala
31b48d162a
- persistent WS connection to HA with auth + state_changed subscription - watchdog detects silence > 5min → emits ha_websocket_dead - immediate ha_websocket_dead on disconnect, exponential reconnect with jitter - cooldown prevents alert spam (10min repeat window while HA stays down) - ha_websocket_recovered emitted on reconnect after a dead alert (allows supervisor to clear active incidents in Phase 5) - new monitors/ subpackage for long-running tasks (vs interval checks/) - /health endpoint now includes ws_connected field - 26 unit tests, 3 integration tests (real HA + container stop/restart) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
121 lines
4.3 KiB
Markdown
121 lines
4.3 KiB
Markdown
# ha-diag-agent
|
|
|
|
Per-host Home Assistant diagnostic agent. Polls HA REST API on a schedule,
|
|
emits structured events to `/opt/homelab/events/<node>/`, and exposes an
|
|
HTTP API for health checks and manual check triggers.
|
|
|
|
Follows the same event-pipeline pattern as `node-agent`: filesystem-first,
|
|
no direct supervisor integration, events processed by the VPS observer.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
APScheduler (interval-based REST checks)
|
|
├─ HeartbeatCheck → pings /api/, emits ha_websocket_dead on failure
|
|
├─ UnavailableEntitiesCheck → entity unavailable > threshold
|
|
├─ SystemHealthCheck → /api/system_health per-integration status
|
|
├─ AutomationFailuresCheck → automation last-run error traces
|
|
└─ UpdatesAvailableCheck → pending HA/integration updates
|
|
|
|
WebSocketMonitor (persistent, long-running — Phase 4b)
|
|
└─ Maintains a live WS subscription to state_changed events
|
|
Any traffic = HA is alive. Watchdog fires ha_websocket_dead on
|
|
silence > 5min or on disconnect. Emits ha_websocket_recovered
|
|
when the connection is restored after a dead alert.
|
|
|
|
FastAPI (port 8087)
|
|
GET /health → liveness probe (includes ws_connected field)
|
|
POST /trigger/<check> → run a named check on demand
|
|
|
|
SQLite (/data/ha_diag.db)
|
|
entity_baseline → last-known entity states
|
|
check_history → per-check run log
|
|
alerts_sent → dedup gate for alert events
|
|
```
|
|
|
|
The WebSocketMonitor is the only persistent-connection component; all other
|
|
checks are APScheduler intervals (stateless REST polls).
|
|
|
|
## Event Types
|
|
|
|
| Type | Severity | Trigger |
|
|
|------|----------|---------|
|
|
| `ha_websocket_dead` | error | WS disconnect, silence > 5min, or /api/ unreachable |
|
|
| `ha_websocket_recovered` | info | WS reconnected after a dead alert (clears incident) |
|
|
| `ha_integration_failed` | error | Integration in error state |
|
|
| `ha_entity_unavailable_long` | warning | Entity unavailable > threshold |
|
|
| `ha_automation_failing` | warning | Automation last run errored |
|
|
| `ha_update_available` | info | HA or integration update pending |
|
|
| `ha_recorder_lag` | warning | Recorder write lag > threshold |
|
|
| `ha_system_health_degraded` | warning | System health check failed |
|
|
|
|
Event routing in supervisor (Phase 5) maps these to `notify` actions.
|
|
`ha_websocket_recovered` should be routed to clear any active `ha_websocket_dead` incident.
|
|
|
|
## Deployment model
|
|
|
|
The agent is deployed **per-host** but targets a potentially remote HA instance:
|
|
|
|
| Node | Agent runs on | HA lives on | HA URL |
|
|
|------|--------------|-------------|--------|
|
|
| piha | piha | piha (localhost) | `http://localhost:8123` |
|
|
| chelsty-infra | chelsty-infra | chelsty-ha (HAOS VM, separate machine) | `http://100.70.180.90:8123` |
|
|
|
|
**chelsty-infra note:** Home Assistant runs on `chelsty-ha`, a dedicated Home Assistant
|
|
OS VM. `chelsty-infra` is the hypervisor but does not run HA itself. The agent on
|
|
`chelsty-infra` reaches HA over the Tailscale network (`100.70.180.90:8123`). If `chelsty-ha`
|
|
gets a new Tailscale IP, update `HA_URL` in `/opt/homelab/config/ha-diag-agent/.env` on
|
|
`chelsty-infra`.
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
# 1. Create config on target node
|
|
ssh oskar@<node-ip>
|
|
mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent
|
|
cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF'
|
|
HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra
|
|
HA_TOKEN=<long-lived-token>
|
|
NODE_NAME=piha # or chelsty-infra
|
|
LOCATION_TAG=ken # or chelsty
|
|
CHECK_INTERVAL=60
|
|
EOF
|
|
|
|
# 2. Deploy
|
|
scripts/deploy/deploy.sh --service ha-diag-agent
|
|
|
|
# 3. Verify
|
|
docker ps --filter name=ha-diag-agent
|
|
curl http://localhost:8087/health
|
|
```
|
|
|
|
### chelsty-infra note
|
|
|
|
`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated):
|
|
```bash
|
|
docker-compose -f docker-compose.yml up -d --build
|
|
```
|
|
|
|
### HA long-lived token
|
|
|
|
In HA UI: Profile → Long-Lived Access Tokens → Create token.
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
cd services/ha-diag-agent
|
|
pip install -e ".[dev]"
|
|
pytest tests/ -v
|
|
```
|
|
|
|
## Optional YAML config
|
|
|
|
Place `/opt/homelab/config/ha-diag-agent/ha-diag-agent.yaml` on the node.
|
|
Values there are defaults; env vars take priority.
|
|
|
|
```yaml
|
|
ha_url: http://homeassistant.local:8123
|
|
location_tag: ken
|
|
check_interval: 60
|
|
```
|