homelab-codex-ws/docs/observer-runtime.md

Observer Runtime

The Observer Runtime is a lightweight agent responsible for synthesizing the operational world state of the homelab from raw events, logs, and state files.

Architecture

The observer follows a filesystem-first approach, consuming append-only events and generating a normalized world model. It is designed to be idempotent, resumable, and resilient to intermittent node connectivity.

Inputs

• /opt/homelab/events/: Normalized JSON events (one .json file per event, organized by date and node).
• /opt/homelab/state/observer_checkpoint.json: Per-node checkpoint dict (see below).
• Repository Inventory: inventory/topology.yaml and hosts/*/services.yaml.

World Model Output

Generated under /opt/homelab/world/:

• nodes.json: Current node availability, roles, disk/memory pressure, last seen timestamps. Dict keyed by node name.
• services.json: Service health status and links to active incidents. Dict keyed by "node/service".
• deployments.json: Tracking of active and historical deployment runs by correlation_id.
• incidents.json: Correlated operational issues, including repeat failures and resolution status.
• runtime-summary.json: High-level overview for dashboards and planner agents.

Checkpoint Format

The observer tracks per-node progress to avoid silently skipping event directories:

{
  "node_checkpoints": {
    "vps":            "/opt/homelab/events/2026-05-27/vps/evt-vps-1234.json",
    "piha":           "/opt/homelab/events/2026-05-27/piha/evt-piha-5678.json",
    "chelsty-infra":  "/opt/homelab/events/2026-05-27/chelsty-infra/evt-chelsty-infra-9012.json"
  }
}

A single global checkpoint (last_processed_file) was replaced with this per-node dict because the old approach silently skipped any node directory that sorts alphabetically before the last-seen node (e.g. piha/ would be skipped when the checkpoint pointed to vps/).

Reset: Delete /opt/homelab/state/observer_checkpoint.json. The observer will reprocess all events and rebuild world state from scratch.

Event Types

Negative events (create/escalate incidents)

• service_unhealthy, healthcheck_failed — open or increment an active incident
• deployment_failed — record failure in deployments.json

Positive events (resolve state)

• service_healthy — marks service status as healthy and resolves any active incident for that service
• service_recovered — alias, same effect
• deployment_completed — marks deployment as completed

Node events

• node_online, node_offline — update node status in nodes.json
• disk_pressure_* — set disk_pressure field on the node record

Node Liveness (TTL)

Node status is derived from freshness, not from the last status event. Every reconcile cycle (_prune_stale_world, runs even with no new events) the observer computes now - last_seen per node and classifies it:

tier	age (always-on)	age (remote / LTE)	status written	panel health
fresh	≤ 180 s	≤ 900 s	online	nominal
stale	180–600 s	900–3600 s	stale	degraded
dead	> 600 s	> 3600 s	offline	error
unknown	last_seen missing	—	(unchanged)	(unchanged)

Thresholds (tied to the 60 s node-agent heartbeat: fresh = 3× interval) live in one place: services/control-plane/src/liveness.py, imported by the observer and both operator UIs (compute_liveness / ttls_for / node_health). Remote nodes (topology role remote, or chelsty-*) use the wider TTLs above. Override via env: LIVENESS_TTL_FRESH, LIVENESS_TTL_DEAD, LIVENESS_REMOTE_TTL_FRESH, LIVENESS_REMOTE_TTL_DEAD.

This fixes the "dead node shown NOMINAL" silent outage: previously status stayed online forever because the only thing that flipped it to offline was a node_offline event, which a crashed/partitioned node can never emit.

Transitions are not silent. On crossing a boundary the observer writes an event (node_stale, node_offline, or node_online on recovery) tagged source: "observer" with the affected node in both node and payload.affected_node. These are skipped on re-ingest (so they never reset last_seen) and routed by the supervisor to alert_only actions (Telegram).

Read-time safety net: both operator UIs recompute liveness from last_seen at request time using the same helper, so even a stalled observer (frozen nodes.json) still surfaces a dead node. Services inherit their node's liveness (a service on a dead/stale node is never shown nominal) — computed read-time, services.json is not mutated.

Incident Lifecycle

1. Detection: A service_unhealthy or healthcheck_failed event creates or increments an active incident.
2. Correlation: Multiple failure events for the same node/service are collapsed into one incident, incrementing occurrence_count.
3. Resolution: A service_healthy or service_recovered event resolves any active incident for that service, setting status: resolved and resolved_at.
4. Expiry: Resolved incidents older than 7 days are pruned from world state by _prune_stale_world().

Example Incident JSON

{
  "inc-1715518800-vps-observer": {
    "id": "inc-1715518800-vps-observer",
    "node": "vps",
    "service": "observer",
    "status": "resolved",
    "severity": "error",
    "started_at": 1715518800.0,
    "last_occurrence": 1715518860.0,
    "occurrence_count": 2,
    "trigger_type": "containers_not_running",
    "resolved_at": 1715519100.0
  }
}

World State Pruning

_prune_stale_world() runs every reconcile cycle and removes:

1. Stale nodes — nodes not present in inventory/topology.yaml (e.g. ghost nodes created when NODE_NAME was unset and fell back to the container's 12-char hex ID).
2. Services of stale nodes — all node/service keys whose node was pruned.
3. Ghost service keys — service keys whose service-name portion matches the pattern <12hexchars>_<name> (Docker internal stale-state artifacts, created when node-agent used c.name instead of the compose label).
4. Expired incidents — resolved incidents older than 7 days.

Runtime Behavior

Idempotency

The observer processes events in order. Deleting the checkpoint and restarting replays all events and produces the same world state.

Deployment Tracking

Deployments are tracked via correlation_id. The observer synthesizes the start, end, and status of each deployment run from events.

Topology Filtering

Events from nodes not listed in inventory/topology.yaml are discarded during pruning. This prevents transient bootstrap noise from polluting world state.