5c262ca25d
public (wzorce/schematy, bez IP/portow/sciezek hostow): observer, capability-model, event-system, standards, agent-operating-procedures, service-model, action-approval-model. private: recon-multiagent, fleet-inventory, fleet-inventory-verify, kb-mail-pillar, kb-documents-pillar, topology, agent-system. deprecated (martwe stuby z 2026-04-15) — visibility private wg rozstrzygniecia 6: access-model, core-stack, legacy-services-list, networking. git mv + frontmatter, tresc nietknieta. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
144 lines
6.7 KiB
Markdown
144 lines
6.7 KiB
Markdown
---
|
||
okf: "0.1"
|
||
type: subsystem
|
||
visibility: public
|
||
status: active
|
||
updated: 2026-06-17
|
||
links: []
|
||
---
|
||
|
||
# Observer Runtime
|
||
|
||
The Observer Runtime is a lightweight agent responsible for synthesizing the operational world state of the homelab from raw events, logs, and state files.
|
||
|
||
## Architecture
|
||
|
||
The observer follows a filesystem-first approach, consuming append-only events and generating a normalized world model. It is designed to be idempotent, resumable, and resilient to intermittent node connectivity.
|
||
|
||
### Inputs
|
||
- `/opt/homelab/events/`: Normalized JSON events (one `.json` file per event, organized by date and node).
|
||
- `/opt/homelab/state/observer_checkpoint.json`: Per-node checkpoint dict (see below).
|
||
- Repository Inventory: `inventory/topology.yaml` and `hosts/*/services.yaml`.
|
||
|
||
### World Model Output
|
||
Generated under `/opt/homelab/world/`:
|
||
- `nodes.json`: Current node availability, roles, disk/memory pressure, last seen timestamps. Dict keyed by node name.
|
||
- `services.json`: Service health status and links to active incidents. Dict keyed by `"node/service"`.
|
||
- `deployments.json`: Tracking of active and historical deployment runs by `correlation_id`.
|
||
- `incidents.json`: Correlated operational issues, including repeat failures and resolution status.
|
||
- `runtime-summary.json`: High-level overview for dashboards and planner agents.
|
||
|
||
## Checkpoint Format
|
||
|
||
The observer tracks per-node progress to avoid silently skipping event directories:
|
||
|
||
```json
|
||
{
|
||
"node_checkpoints": {
|
||
"vps": "/opt/homelab/events/2026-05-27/vps/evt-vps-1234.json",
|
||
"piha": "/opt/homelab/events/2026-05-27/piha/evt-piha-5678.json",
|
||
"chelsty-infra": "/opt/homelab/events/2026-05-27/chelsty-infra/evt-chelsty-infra-9012.json"
|
||
}
|
||
}
|
||
```
|
||
|
||
A single global checkpoint (`last_processed_file`) was replaced with this per-node dict because the old approach silently skipped any node directory that sorts alphabetically before the last-seen node (e.g. `piha/` would be skipped when the checkpoint pointed to `vps/`).
|
||
|
||
**Reset:** Delete `/opt/homelab/state/observer_checkpoint.json`. The observer will reprocess all events and rebuild world state from scratch.
|
||
|
||
## Event Types
|
||
|
||
### Negative events (create/escalate incidents)
|
||
- `service_unhealthy`, `healthcheck_failed` — open or increment an active incident
|
||
- `deployment_failed` — record failure in deployments.json
|
||
|
||
### Positive events (resolve state)
|
||
- `service_healthy` — marks service status as `healthy` **and** resolves any active incident for that service
|
||
- `service_recovered` — alias, same effect
|
||
- `deployment_completed` — marks deployment as completed
|
||
|
||
### Node events
|
||
- `node_online`, `node_offline` — update node status in nodes.json
|
||
- `disk_pressure_*` — set `disk_pressure` field on the node record
|
||
|
||
## Node Liveness (TTL)
|
||
|
||
Node `status` is derived from **freshness**, not from the last status event.
|
||
Every reconcile cycle (`_prune_stale_world`, runs even with no new events) the
|
||
observer computes `now - last_seen` per node and classifies it:
|
||
|
||
| tier | age (always-on) | age (remote / LTE) | status written | panel health |
|
||
|---|---|---|---|---|
|
||
| fresh | ≤ 180 s | ≤ 900 s | `online` | nominal |
|
||
| stale | 180–600 s | 900–3600 s | `stale` | degraded |
|
||
| dead | > 600 s | > 3600 s | `offline` | error |
|
||
| unknown | `last_seen` missing | — | (unchanged) | (unchanged) |
|
||
|
||
Thresholds (tied to the 60 s node-agent heartbeat: fresh = 3× interval) live in
|
||
**one place**: `services/control-plane/src/liveness.py`, imported by the observer
|
||
and both operator UIs (`compute_liveness` / `ttls_for` / `node_health`). Remote
|
||
nodes (topology role `remote`, or `chelsty-*`) use the wider TTLs above.
|
||
Override via env: `LIVENESS_TTL_FRESH`, `LIVENESS_TTL_DEAD`,
|
||
`LIVENESS_REMOTE_TTL_FRESH`, `LIVENESS_REMOTE_TTL_DEAD`.
|
||
|
||
This fixes the "dead node shown NOMINAL" silent outage: previously `status`
|
||
stayed `online` forever because the only thing that flipped it to offline was a
|
||
`node_offline` event, which a crashed/partitioned node can never emit.
|
||
|
||
**Transitions** are not silent. On crossing a boundary the observer writes an
|
||
event (`node_stale`, `node_offline`, or `node_online` on recovery) tagged
|
||
`source: "observer"` with the affected node in both `node` and
|
||
`payload.affected_node`. These are **skipped on re-ingest** (so they never reset
|
||
`last_seen`) and routed by the supervisor to `alert_only` actions (Telegram).
|
||
|
||
**Read-time safety net:** both operator UIs recompute liveness from `last_seen`
|
||
at request time using the same helper, so even a *stalled observer* (frozen
|
||
`nodes.json`) still surfaces a dead node. Services inherit their node's liveness
|
||
(a service on a dead/stale node is never shown nominal) — computed read-time,
|
||
`services.json` is not mutated.
|
||
|
||
## Incident Lifecycle
|
||
|
||
1. **Detection**: A `service_unhealthy` or `healthcheck_failed` event creates or increments an active incident.
|
||
2. **Correlation**: Multiple failure events for the same `node/service` are collapsed into one incident, incrementing `occurrence_count`.
|
||
3. **Resolution**: A `service_healthy` or `service_recovered` event resolves any active incident for that service, setting `status: resolved` and `resolved_at`.
|
||
4. **Expiry**: Resolved incidents older than 7 days are pruned from world state by `_prune_stale_world()`.
|
||
|
||
### Example Incident JSON
|
||
```json
|
||
{
|
||
"inc-1715518800-vps-observer": {
|
||
"id": "inc-1715518800-vps-observer",
|
||
"node": "vps",
|
||
"service": "observer",
|
||
"status": "resolved",
|
||
"severity": "error",
|
||
"started_at": 1715518800.0,
|
||
"last_occurrence": 1715518860.0,
|
||
"occurrence_count": 2,
|
||
"trigger_type": "containers_not_running",
|
||
"resolved_at": 1715519100.0
|
||
}
|
||
}
|
||
```
|
||
|
||
## World State Pruning
|
||
|
||
`_prune_stale_world()` runs every reconcile cycle and removes:
|
||
|
||
1. **Stale nodes** — nodes not present in `inventory/topology.yaml` (e.g. ghost nodes created when `NODE_NAME` was unset and fell back to the container's 12-char hex ID).
|
||
2. **Services of stale nodes** — all `node/service` keys whose node was pruned.
|
||
3. **Ghost service keys** — service keys whose service-name portion matches the pattern `<12hexchars>_<name>` (Docker internal stale-state artifacts, created when node-agent used `c.name` instead of the compose label).
|
||
4. **Expired incidents** — resolved incidents older than 7 days.
|
||
|
||
## Runtime Behavior
|
||
|
||
### Idempotency
|
||
The observer processes events in order. Deleting the checkpoint and restarting replays all events and produces the same world state.
|
||
|
||
### Deployment Tracking
|
||
Deployments are tracked via `correlation_id`. The observer synthesizes the start, end, and status of each deployment run from events.
|
||
|
||
### Topology Filtering
|
||
Events from nodes not listed in `inventory/topology.yaml` are discarded during pruning. This prevents transient bootstrap noise from polluting world state.
|