(A1, A2, B7, D15). Source read at master @ `473bf8e`; this branch is cut from
master @ `0650eb8`. The two intervening merges (`0650eb8` ha-mcp, `cb8a19d`
kb-query tests) touch none of the audited paths — `services/node-agent/`,
`services/stability-agent/`, `scripts/observer/`, `services/control-plane/` — so
every finding and line citation below holds at `0650eb8`. Runtime evidence
collected 2026-07-30 ~14:30 UTC from vps, piha and solaria (solaria = the host
this recon ran on).
---
## FINDING FIRST
**stability-agent's filesystem event output is not merely mis-tagged — it is
structurally unreadable by the entire control plane, and always has been.**
Recon D15 identified one break (`service=None`). There are five, stacked, and
any one of them alone is fatal:
| # | Break | Evidence |
|---|---|---|
| 1 | **Wrong path subtree.** stability-agent writes `/opt/homelab/events/<YYYY-MM-DD>/<node>/events.jsonl`. node-agent writes `/opt/homelab/events/<node>/evt-*.json`. These are sibling subtrees, not the same store. | `stability_agent.py:47-51` vs `node_agent.py:261,287` |
| 2 | **Wrong file extension.** Observer and supervisor both discover events with `glob("**/*.json")`. `events.jsonl` does not match `*.json`. | `observer.py:912`, `supervisor.py:605` |
| 3 | **Wrong file format.** Even if matched, the observer does `json.load(f)` on the *whole file* — a multi-line JSONL append log is not a JSON document. | `observer.py:938` |
| 4 | **Never shipped off-node.** node-agent's rsync ships only `EVENTS_DIR/<node>/`. The date-partitioned subtree is outside that path, so piha's and solaria's stability-agent events never reach the VPS at all. | `node_agent.py:788` |
| 5 | **Wrong payload key + `service=None`** (recon D15). stability-agent writes `details`; the observer reads `payload`. And the *deployed* build still emits one aggregate event with no `service` key. | `stability_agent.py:42` vs `observer.py:701` |
Break #4 is confirmed empirically: on VPS, every date-partitioned directory
contains **only `vps/`** — there is no `piha/` or `solaria/` subdir in any of the
13 date dirs, despite both nodes having written thousands of local lines.
Break #5's repo-side fix exists but **is not deployed**: commit `f92e161`
("tag containers_not_running events with compose service") is dated **2026-07-29**;
the stability-agent image running on solaria was built **2026-05-17**. 100 % of the
5,056 deployed events sampled across three nodes carry `service: None`. So D15's
claim **is still true on current runtime**, while being *stale on current master* —
and it is moot either way, because breaks #1–#4 mean the observer never opens the
file.
Consequence: **all six of stability-agent's filesystem event types are inert** —
| **Deploy path** | **declarative** — in `hosts/*/services.yaml` (7 hosts) → `deploy.sh` / `deploy-node.sh` | **shadow** — `scripts/deploy/deploy-stability-agent.sh <node> [--ssh]` (hardcoded TS IPs) → `services/stability-agent/deploy-local.sh` → `compose up -d --build --force-recreate`. **Explicitly skipped** by `deploy-node.sh:93` because it ships a `deploy-local.sh` |
| **In desired state?** | yes, all 7 hosts | yes on `hosts/solaria` + `hosts/vps` (added in the 2026-07 truth cleanup — **recon B7 is stale here**), but the entry is inert: the deploy loop skips it |
| **Service contract files** | ❌ **no `service.yaml`, no README, no env.example, no healthcheck.sh** — violates the CLAUDE.md service layout | ✅ all four present |
| 1 | **Redis publishing** → `homelab:nodes:*`, `homelab:services:*`, `homelab:events` stream | ✅ **YES — and exclusive** | `stability_agent.py:330-372` → `agent-system-redis` on piha (100.108.208.3) → `services/agent-system/runtime-materializer/materializer.py:122,138,155` reads all three → writes the agent-system WebUI's world dir. Verified live: node keys refreshed 2026-07-30T14:28, service keys 14:29. **Not an incident path** — it feeds a UI, never the supervisor. node-agent has no Redis code at all. |
| 2 | `containers_not_running` (its own) | ❌ dead | Breaks #1–#5 above. Additionally the deployed build emits the pre-fix aggregate form. Never opens an incident; `observer.py:770` would handle it *if* the file were ever read. |
| 3 | `disk_usage_high` | ❌ dead ×2 | Unreadable (#1–#4), **and** the type appears nowhere in `observer.process_event`'s if/elif chain — the observer only knows `disk_pressure`. Even hand-fed a well-formed file, it would bump `last_seen` and fall through. Not in any supervisor trigger set. |
| 4 | `docker_api_error` | ❌ dead ×2 | Same: unreadable, and no branch in `process_event`, no supervisor trigger. Despite being the **2nd-most-emitted type on VPS (496 events)**. |
| 5 | `agent_error` | ❌ dead ×2 | Same. Self-diagnostics that nothing collects. |
| 6 | `redis_publish_error` | ❌ dead ×2 | Same. |
| 7 | `mqtt_unreachable` (TCP probe) | ❌ dead ×3 | Unreadable; **`MQTT_HOST` is unset on piha/solaria/vps** so the probe never even runs there (only the never-applied chelsty-infra override sets it); and the type was **removed from `CONTAINER_RESTART_TRIGGERS` on 2026-07-28** (`supervisor.py:47-51`) precisely because the observer never creates such an incident. |
| 8 | Tailscale presence check | ❌ inert | `stability_agent.py:189` — result goes to `state/stability-agent.json` only. No event emitted. Grep confirms **no consumer** of that file anywhere in the repo outside its own docs. |
| 9 | Zigbee2MQTT presence/running check | ❌ inert | `stability_agent.py:297-310` — same status-file-only fate. Also a naive `"zigbee2mqtt" in name` substring match. |
Two structural properties also unique to it, neither a "capability":
- **`:ro` sock mount** — a safety property. It is the only sock-holder on the fleet that is read-only by mount (recon A1). Preserving it means preserving a container whose *outputs* are dead.
- **Broader container coverage** — no restart-policy filter, so it sees one-shot and manually-run containers that node-agent deliberately ignores. Inert, since the events go nowhere.
**Bonus:** `scripts/lib/events.sh` — the documented shell event emitter — writes to
the *same* dead date-partitioned JSONL path (`scripts/lib/events.sh:18-20`). Every
shell-emitted event in the system shares stability-agent's fate. CLAUDE.md
("Events are append-only JSON lines at `/opt/homelab/events/YYYY-MM-DD/<node>/events.jsonl`")
documents the dead format as the canonical one.
---
## 3. UNIQUE TO node-agent
| # | Capability | Notes |
|---|---|---|
| 1 | **Dispatch execution** (`container_restart`) | The on-node hands of the HITL loop. Type whitelist of exactly one, self-restart guard, idempotency markers, node-scoping check (`:921-991`). No equivalent exists in stability-agent — and adding one would require flipping its sock to rw. |
| 2 | **`action_result` reporting** | Closes the executor's loop (`:993`); without it actions hang in `running/`. |
| 3 | **Event shipping to VPS** | rsync `--remove-source-files` (`:777`). This is *the* mechanism by which any off-VPS node reaches the control plane. stability-agent has no shipping whatsoever. |
| 4 | **An event format the control plane can read** | One `*.json` file per event, `evt-<node>-<ts>-<type>-<svc>` naming that carries the checkpoint ordering key, `payload` field. Everything downstream is built around this shape. |
| 5 | **RAM monitoring** (`high_memory`) | Genuinely absent from stability-agent. Matters most on the 4 GiB no-swap VPS. |
| 6 | **CPU monitoring** (`high_cpu`) | Ditto. 121 events on vps, 129 on piha, 48 on lustro in the retained window. |
| 7 | **Crash-loop detection** | `RestartCount ≥ 3` threshold distinguishes a real flap from deploy churn (`:482-516`). stability-agent cannot tell them apart. |
| 8 | **Docker healthcheck-status reading** → `healthcheck_failed` | 3,375 events on piha. The single largest actionable signal in the fleet after `containers_not_running`. |
| 9 | **`service_healthy` positive confirmation** | Keeps `services.json` populated so the supervisor's drift loop doesn't classify every desired service as `missing_service` and flood the action queue (`:528-549`, `observer.py:762`). Load-bearing. |
| 10 | **`node_health` heartbeat** | Drives observer node liveness (`node_offline`/`node_stale`/`node_online`). The single highest-volume event type fleet-wide. |
| 11 | **Docker cleanup, per-node-type** | The whole `lte_node`/`sd_card`/`ai_node`/`standard` policy (`:646-667`). SD-card write-cycle protection and the "never `-a` on ai_node" rule live only here. |
| 12 | **Control-plane FS rotation** (vps) | Old actions, deploy logs, checkpointed noise-event sweep (`:673`). The fix for the 358k-file backlog that paralyzed the supervisor. |
| 13 | **Control-plane HTTP health probe** (vps) | The only thing that maps the 4-container control-plane stack to the single `control-plane` key in `services.yaml` (`:1016`). |
| 14 | **Declarative deploy path** | In `hosts/*/services.yaml`, deployed by the normal pipeline. |
Node-agent's own gaps, for symmetry: no `service.yaml`/README/env.example/healthcheck.sh
(CLAUDE.md violation); no memory limit declared on VPS despite the VPS rule; rw sock
with **no allowlist of restartable container names** (recon A2); and on solaria the
docker half is broken by a group_add gid mismatch — though note that as of today
solaria's node-agent container reports `Up 28 minutes (healthy)`, so this is worth
re-verifying separately before relying on either state.
---
## 4. RUNTIME EVIDENCE
Collected 2026-07-30 ~14:30. VPS event store = `/opt/homelab/events/`.
### Container liveness
| Node | stability-agent | node-agent |
|---|---|---|
| vps | running, **Up 7 weeks** (healthy) | running, Up 13 days (healthy) |
| piha | running, Up 13 days (healthy) | running, Up 7 days (healthy) |
| solaria | running, Up ~1 h (healthy) | running, Up 28 min (healthy) |
Both are running everywhere. Neither is crashed. This is a delivery problem, not an
availability problem.
### Events reaching the VPS control plane (node-agent subtree, `events/<node>/`)
Every one of those keys is written by stability-agent and by nothing else. The
service-key set also shows the deployed build's staleness — entries like
`homelab:services:vps:dc8be279ea1a_node-agent` carry the docker stale-state hash
prefix that `container_service_name()` was written to strip.
---
## 5. COST OF EACH OPTION
### (a) Retire stability-agent entirely
**Monitoring capability lost:**
- Nothing that currently reaches an incident, an action, or an operator alert. All six filesystem event types are already inert (§2).
- MQTT TCP reachability probe — the only fleet component that probes a broker port. Currently unconfigured on every online node and untriggerable, so the loss is *prospective*, not actual. Relevant to chelsty (offline-first, Mosquitto is load-bearing) if that site is revived.
- Tailscale presence and Zigbee2MQTT presence checks — inert today, but they are the only code that looks at either.
- Container visibility without a restart-policy filter (one-shot/manual containers).
- Disk monitoring of `/` as distinct from `/opt/homelab` — on any node where those are separate mounts, node-agent does not watch `/`.
**What breaks — the real cost:**
- **The agent-system WebUI goes blank.** `homelab:nodes:*`, `homelab:services:*` and the `homelab:events` stream lose their only writer. `runtime-materializer` continues polling and materializes empty sets. This is a live, working, user-facing surface — 130 service keys refreshed a minute before this recon was written.
- Any Telegram-bot or WebUI view built on that materialized world dir degrades with it (not separately verified in this recon).
- The fleet loses its only `:ro`-sock container; every remaining sock holder is rw (recon A1).
-`hosts/vps/runtime/stability-agent/` was allotted 96 MiB of the 3,200 MiB VPS budget — reclaimed.
- **Not covered by the cleanup:** the Redis publisher must be rehomed first or the UI loss is permanent.
### (b) Merge stability-agent's unique checks into node-agent
**Which checks would actually move — four, and only one is non-trivial:**
| Check | Surface area |
|---|---|
| **Redis publishing** (the load-bearing one) | ~60 lines: a `RedisClient` (or the `redis` package, already a dependency of agent-system), plus the `homelab:nodes:*` / `homelab:services:*` HSET loop and the `homelab:events` XADD. node-agent already computes every input — disk/mem/cpu in `run_once()`, container state in `check_containers()`. It needs a publish hook, `REDIS_HOST/PORT/ENABLED` env vars in the compose and in 4–7 host overrides, and a decision on whether an offline-first node may block on a piha-hosted Redis (must be fail-open; stability-agent's version already is). **This alone preserves the only functional output.** |
| **MQTT TCP probe** | ~15 lines: `socket.create_connection((MQTT_HOST, MQTT_PORT), timeout=5)` → emit `mqtt_unreachable`. Plus `MQTT_HOST`/`MQTT_PORT` env. Note: to be *useful* it also needs an observer branch and a supervisor trigger — the type was deliberately removed from `CONTAINER_RESTART_TRIGGERS` on 2026-07-28 because no incident path existed. So the honest surface is 15 lines in node-agent **plus** an observer `process_event` branch **plus** a supervisor mapping. |
| **Tailscale presence** | ~10 lines, two `os.path.exists` calls. Needs a new event type and observer/supervisor wiring to be worth anything; as a status-file value it is pointless. |
| **Zigbee2MQTT presence** | ~5 lines, but it is a site-specific check on a node that is offline. Arguably belongs in a chelsty-specific probe, not the fleet sensor. |
| **`/` disk check alongside `/opt/homelab`** | ~10 lines: a second `shutil.disk_usage("/")` and a mount label in the payload. |
| **Drop the restart-policy filter** | Not a merge — a behavior change to `check_containers()` with a known noise cost; the filter exists on purpose. |
already logs the equivalents; promoting them to events adds volume with no consumer
unless observer branches are added too.
Rough total: **~100–120 lines in `node_agent.py`**, env additions across
`services/node-agent/docker-compose.yml` and 4–7 host overrides, plus tests
(Definition of Done: docker build + smoke run + pytest — node-agent has 5 test files
to extend). The merged agent keeps its rw sock, so the fleet's last `:ro` sock holder
disappears either way. Requires a coordinated cutover: node-agent must be publishing
to Redis *before* stability-agent stops, or the WebUI gaps.
### (c) Keep both
**Duplicated cost, as measured:**
| Dimension | Cost |
|---|---|
| **Deploy paths** | Two mechanisms for the same job. stability-agent's is the shadow path — hardcoded Tailscale IPs, explicitly skipped by `deploy-node.sh:93`, a node-name whitelist that doesn't match the repo's own host dirs (`chelsty` vs `chelsty-infra`), and `--force-recreate` on every run. Its `hosts/*/services.yaml` entries are decorative: the pipeline skips them, so desired state claims coverage the pipeline does not provide. |
| **Undeployed fixes** | The gap is not theoretical. `f92e161` (2026-07-29) is in master; the vps image is from 2026-05-17 and has been running 7 weeks. Nobody noticed, because fixing the `service` tag changes nothing while breaks #1–#4 stand. A second deploy path is a second thing to forget. |
| **Sock exposure** | Two containers per node hold `/var/run/docker.sock`. stability-agent's is `:ro` and its output is dead — the fleet pays the attack surface of a sock mount on every node for a channel with zero consumers. |
| **Event volume** | ~7,300 dead events written to disk across three nodes, ~5 MB, in 13–20 date directories per node that no retention sweep touches (node-agent's `_cleanup_control_plane_fs` only prunes `service_healthy`/`node_health` under `events/<node>/`, never the date subtree). It grows without bound and nothing reads it. |
| **Confusion — the compounding cost** | CLAUDE.md documents the dead JSONL path as *the* canonical event format (line 100). `scripts/lib/events.sh` — the sanctioned shell emitter — writes to it. Recon D15 diagnosed one symptom (`service=None`) and a fix was written and merged for it, when four upstream breaks made that fix inert. The 2026-07-06 prometheus recon reached the correct conclusion and it was not carried forward. Two sensors emitting the same event type name into two incompatible stores means every future "why didn't this alert fire" investigation starts by disambiguating which agent, which store, which format. |
| **What it costs to keep it honest** | Nothing about (c) is stable as-is: keeping both means either accepting a permanently dead channel, or fixing breaks #1–#4 — which is a rewrite of stability-agent's emitter into node-agent's format, i.e. option (b) performed inside a second codebase. |
---
## DECISION NEEDED
The choice is whether the Redis→agent-system-WebUI publisher is worth a second
per-node daemon with a docker.sock mount and a parallel deploy path, or whether
it moves into node-agent. Everything else about stability-agent is already dead:
all six of its filesystem event types are unreadable by the observer and
supervisor for four independent structural reasons beyond the one recon D15
named, its events have never opened an incident on any node in the service's
lifetime, and the two nodes that are not the VPS have never shipped a single one
of them anywhere. Against that, it is the *only* writer of the 4 node keys, 130
service keys and 31,458-entry stream that the agent-system UI renders — refreshed
one minute before this recon was written — and node-agent has no Redis code at
all. So (a) retire is the cheapest cleanup but blanks a live UI unless the
publisher is rehomed first; (b) merge is ~100–120 lines plus env plumbing across
4–7 host overrides plus a coordinated cutover, and ends with one sensor, one
deploy path, one sock, one event format; (c) keep both preserves the UI at zero
engineering cost today, but the duplicated cost is not static — it is a
permanently dead event channel that the repo's own CLAUDE.md documents as
canonical, a shadow deploy path that has already silently withheld a merged fix
for 7 weeks, and a second sock mount on every node buying nothing. The key
trade-off is therefore not "monitoring coverage" — node-agent strictly dominates
on every check that reaches a consumer — but **whether to pay a one-time
migration of the Redis publisher now, or keep paying the ambiguity of two
sensors, two event stores and two deploy paths indefinitely.**
