homelab-codex-ws/docs/observer-runtime.md

# Observer Runtime

The Observer Runtime is a lightweight agent responsible for synthesizing the operational world state of the homelab from raw events, logs, and state files.

## Architecture

The observer follows a filesystem-first approach, consuming append-only events and generating a normalized world model. It is designed to be idempotent, resumable, and resilient to intermittent node connectivity.

### Inputs
- `/opt/homelab/events/`: Normalized JSON events (one `.json` file per event, organized by date and node).
- `/opt/homelab/state/observer_checkpoint.json`: Per-node checkpoint dict (see below).
- Repository Inventory: `inventory/topology.yaml` and `hosts/*/services.yaml`.

### World Model Output
Generated under `/opt/homelab/world/`:
- `nodes.json`: Current node availability, roles, disk/memory pressure, last seen timestamps. Dict keyed by node name.
- `services.json`: Service health status and links to active incidents. Dict keyed by `"node/service"`.
- `deployments.json`: Tracking of active and historical deployment runs by `correlation_id`.
- `incidents.json`: Correlated operational issues, including repeat failures and resolution status.
- `runtime-summary.json`: High-level overview for dashboards and planner agents.

## Checkpoint Format

The observer tracks per-node progress to avoid silently skipping event directories:

```json
{
  "node_checkpoints": {
    "vps":            "/opt/homelab/events/2026-05-27/vps/evt-vps-1234.json",
    "piha":           "/opt/homelab/events/2026-05-27/piha/evt-piha-5678.json",
    "chelsty-infra":  "/opt/homelab/events/2026-05-27/chelsty-infra/evt-chelsty-infra-9012.json"
  }
}
```

A single global checkpoint (`last_processed_file`) was replaced with this per-node dict because the old approach silently skipped any node directory that sorts alphabetically before the last-seen node (e.g. `piha/` would be skipped when the checkpoint pointed to `vps/`).

**Reset:** Delete `/opt/homelab/state/observer_checkpoint.json`. The observer will reprocess all events and rebuild world state from scratch.

## Event Types

### Negative events (create/escalate incidents)
- `service_unhealthy`, `healthcheck_failed` — open or increment an active incident
- `deployment_failed` — record failure in deployments.json

### Positive events (resolve state)
- `service_healthy` — marks service status as `healthy` **and** resolves any active incident for that service
- `service_recovered` — alias, same effect
- `deployment_completed` — marks deployment as completed

### Node events
- `node_online`, `node_offline` — update node status in nodes.json
- `disk_pressure_*` — set `disk_pressure` field on the node record

## Node Liveness (TTL)

Node `status` is derived from **freshness**, not from the last status event.
Every reconcile cycle (`_prune_stale_world`, runs even with no new events) the
observer computes `now - last_seen` per node and classifies it:

| tier | age (always-on) | age (remote / LTE) | status written | panel health |
|---|---|---|---|---|
| fresh | ≤ 180 s | ≤ 900 s | `online` | nominal |
| stale | 180–600 s | 900–3600 s | `stale` | degraded |
| dead | > 600 s | > 3600 s | `offline` | error |
| unknown | `last_seen` missing | — | (unchanged) | (unchanged) |

Thresholds (tied to the 60 s node-agent heartbeat: fresh = 3× interval) live in
**one place**: `services/control-plane/src/liveness.py`, imported by the observer
and both operator UIs (`compute_liveness` / `ttls_for` / `node_health`). Remote
nodes (topology role `remote`, or `chelsty-*`) use the wider TTLs above.
Override via env: `LIVENESS_TTL_FRESH`, `LIVENESS_TTL_DEAD`,
`LIVENESS_REMOTE_TTL_FRESH`, `LIVENESS_REMOTE_TTL_DEAD`.

This fixes the "dead node shown NOMINAL" silent outage: previously `status`
stayed `online` forever because the only thing that flipped it to offline was a
`node_offline` event, which a crashed/partitioned node can never emit.

**Transitions** are not silent. On crossing a boundary the observer writes an
event (`node_stale`, `node_offline`, or `node_online` on recovery) tagged
`source: "observer"` with the affected node in both `node` and
`payload.affected_node`. These are **skipped on re-ingest** (so they never reset
`last_seen`) and routed by the supervisor to `alert_only` actions (Telegram).

**Read-time safety net:** both operator UIs recompute liveness from `last_seen`
at request time using the same helper, so even a *stalled observer* (frozen
`nodes.json`) still surfaces a dead node. Services inherit their node's liveness
(a service on a dead/stale node is never shown nominal) — computed read-time,
`services.json` is not mutated.

## Incident Lifecycle

1.  **Detection**: A `service_unhealthy` or `healthcheck_failed` event creates or increments an active incident.
2.  **Correlation**: Multiple failure events for the same `node/service` are collapsed into one incident, incrementing `occurrence_count`.
3.  **Resolution**: A `service_healthy` or `service_recovered` event resolves any active incident for that service, setting `status: resolved` and `resolved_at`.
4.  **Expiry**: Resolved incidents older than 7 days are pruned from world state by `_prune_stale_world()`.

### Example Incident JSON
```json
{
  "inc-1715518800-vps-observer": {
    "id": "inc-1715518800-vps-observer",
    "node": "vps",
    "service": "observer",
    "status": "resolved",
    "severity": "error",
    "started_at": 1715518800.0,
    "last_occurrence": 1715518860.0,
    "occurrence_count": 2,
    "trigger_type": "containers_not_running",
    "resolved_at": 1715519100.0
  }
}
```

## World State Pruning

`_prune_stale_world()` runs every reconcile cycle and removes:

1. **Stale nodes** — nodes not present in `inventory/topology.yaml` (e.g. ghost nodes created when `NODE_NAME` was unset and fell back to the container's 12-char hex ID).
2. **Services of stale nodes** — all `node/service` keys whose node was pruned.
3. **Ghost service keys** — service keys whose service-name portion matches the pattern `<12hexchars>_<name>` (Docker internal stale-state artifacts, created when node-agent used `c.name` instead of the compose label).
4. **Expired incidents** — resolved incidents older than 7 days.

## Runtime Behavior

### Idempotency
The observer processes events in order. Deleting the checkpoint and restarting replays all events and produces the same world state.

### Deployment Tracking
Deployments are tracked via `correlation_id`. The observer synthesizes the start, end, and status of each deployment run from events.

### Topology Filtering
Events from nodes not listed in `inventory/topology.yaml` are discarded during pruning. This prevents transient bootstrap noise from polluting world state.