Oskar Kapala
52607a7cdd
- HA_DIAG_SHADOW_MODE env flag in supervisor (default true) - shadow_mode downgrades container_restart actions to alert_only with [SHADOW MODE] note; same action_id and 30-min cooldown apply - alert_only events unaffected (always routed normally) - 3 new tests: shadow on/off for ha_websocket_dead, alert-only unaffected - DEPLOY.md with token gen, per-host config, verification, 48h observation, production-mode enablement, rollback - README.md updated with shadow mode flag summary and DEPLOY.md link Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
132 lines
4.8 KiB
Markdown
132 lines
4.8 KiB
Markdown
# ha-diag-agent
|
|
|
|
Per-host Home Assistant diagnostic agent. Polls HA REST API on a schedule,
|
|
emits structured events to `/opt/homelab/events/<node>/`, and exposes an
|
|
HTTP API for health checks and manual check triggers.
|
|
|
|
Follows the same event-pipeline pattern as `node-agent`: filesystem-first,
|
|
no direct supervisor integration, events processed by the VPS observer.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
APScheduler (interval-based REST checks)
|
|
├─ HeartbeatCheck → pings /api/, emits ha_websocket_dead on failure
|
|
├─ UnavailableEntitiesCheck → entity unavailable > threshold
|
|
├─ SystemHealthCheck → /api/system_health per-integration status
|
|
├─ AutomationFailuresCheck → automation last-run error traces
|
|
└─ UpdatesAvailableCheck → pending HA/integration updates
|
|
|
|
WebSocketMonitor (persistent, long-running — Phase 4b)
|
|
└─ Maintains a live WS subscription to state_changed events
|
|
Any traffic = HA is alive. Watchdog fires ha_websocket_dead on
|
|
silence > 5min or on disconnect. Emits ha_websocket_recovered
|
|
when the connection is restored after a dead alert.
|
|
|
|
FastAPI (port 8087)
|
|
GET /health → liveness probe (includes ws_connected field)
|
|
POST /trigger/<check> → run a named check on demand
|
|
|
|
SQLite (/data/ha_diag.db)
|
|
entity_baseline → last-known entity states
|
|
check_history → per-check run log
|
|
alerts_sent → dedup gate for alert events
|
|
```
|
|
|
|
The WebSocketMonitor is the only persistent-connection component; all other
|
|
checks are APScheduler intervals (stateless REST polls).
|
|
|
|
## Event Types
|
|
|
|
| Type | Severity | Trigger |
|
|
|------|----------|---------|
|
|
| `ha_websocket_dead` | error | WS disconnect, silence > 5min, or /api/ unreachable |
|
|
| `ha_websocket_recovered` | info | WS reconnected after a dead alert (clears incident) |
|
|
| `ha_integration_failed` | error | Integration in error state |
|
|
| `ha_entity_unavailable_long` | warning | Entity unavailable > threshold |
|
|
| `ha_automation_failing` | warning | Automation last run errored |
|
|
| `ha_update_available` | info | HA or integration update pending |
|
|
| `ha_recorder_lag` | warning | Recorder write lag > threshold |
|
|
| `ha_system_health_degraded` | warning | System health check failed |
|
|
|
|
Event routing in supervisor (Phase 5) maps these to `notify` actions.
|
|
`ha_websocket_recovered` should be routed to clear any active `ha_websocket_dead` incident.
|
|
|
|
## First-time deployment
|
|
|
|
See **[DEPLOY.md](DEPLOY.md)** for the full procedure: HA token creation,
|
|
per-host `.env` config, deploy commands, verification steps, 48h shadow-mode
|
|
observation, and rollback.
|
|
|
|
**Shadow mode** (`HA_DIAG_SHADOW_MODE`, default `true` on the control-plane):
|
|
`ha_websocket_dead` events are downgraded to `alert_only` with a `[SHADOW MODE]`
|
|
note instead of queuing an automatic `container_restart`. Set to `false` in
|
|
`/opt/homelab/config/control-plane/.env` on the VPS when ready for live actions.
|
|
|
|
## Deployment model
|
|
|
|
The agent is deployed **per-host** but targets a potentially remote HA instance:
|
|
|
|
| Node | Agent runs on | HA lives on | HA URL |
|
|
|------|--------------|-------------|--------|
|
|
| piha | piha | piha (localhost) | `http://localhost:8123` |
|
|
| chelsty-infra | chelsty-infra | chelsty-ha (HAOS VM, separate machine) | `http://100.70.180.90:8123` |
|
|
|
|
**chelsty-infra note:** Home Assistant runs on `chelsty-ha`, a dedicated Home Assistant
|
|
OS VM. `chelsty-infra` is the hypervisor but does not run HA itself. The agent on
|
|
`chelsty-infra` reaches HA over the Tailscale network (`100.70.180.90:8123`). If `chelsty-ha`
|
|
gets a new Tailscale IP, update `HA_URL` in `/opt/homelab/config/ha-diag-agent/.env` on
|
|
`chelsty-infra`.
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
# 1. Create config on target node
|
|
ssh oskar@<node-ip>
|
|
mkdir -p /opt/homelab/config/ha-diag-agent /var/lib/ha-diag-agent
|
|
cat > /opt/homelab/config/ha-diag-agent/.env << 'EOF'
|
|
HA_URL=http://homeassistant.local:8123 # or http://100.70.180.90:8123 for chelsty-infra
|
|
HA_TOKEN=<long-lived-token>
|
|
NODE_NAME=piha # or chelsty-infra
|
|
LOCATION_TAG=ken # or chelsty
|
|
CHECK_INTERVAL=60
|
|
EOF
|
|
|
|
# 2. Deploy
|
|
scripts/deploy/deploy.sh --service ha-diag-agent
|
|
|
|
# 3. Verify
|
|
docker ps --filter name=ha-diag-agent
|
|
curl http://localhost:8087/health
|
|
```
|
|
|
|
### chelsty-infra note
|
|
|
|
`chelsty-infra` runs docker-compose v1 (1.29.2). Use `docker-compose` (hyphenated):
|
|
```bash
|
|
docker-compose -f docker-compose.yml up -d --build
|
|
```
|
|
|
|
### HA long-lived token
|
|
|
|
In HA UI: Profile → Long-Lived Access Tokens → Create token.
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
cd services/ha-diag-agent
|
|
pip install -e ".[dev]"
|
|
pytest tests/ -v
|
|
```
|
|
|
|
## Optional YAML config
|
|
|
|
Place `/opt/homelab/config/ha-diag-agent/ha-diag-agent.yaml` on the node.
|
|
Values there are defaults; env vars take priority.
|
|
|
|
```yaml
|
|
ha_url: http://homeassistant.local:8123
|
|
location_tag: ken
|
|
check_interval: 60
|
|
```
|