homelab-codex-ws/docs/architecture/RECON-czujniki-2026-07-30.md

# Recon — node-agent vs stability-agent (2026-07-30)

Read-only recon. Ground truth: `docs/architecture/RECON-multiagent-2026-07-27.md`
(A1, A2, B7, D15). Source read at master @ `473bf8e`; this branch is cut from
master @ `0650eb8`. The two intervening merges (`0650eb8` ha-mcp, `cb8a19d`
kb-query tests) touch none of the audited paths — `services/node-agent/`,
`services/stability-agent/`, `scripts/observer/`, `services/control-plane/` — so
every finding and line citation below holds at `0650eb8`. Runtime evidence
collected 2026-07-30 ~14:30 UTC from vps, piha and solaria (solaria = the host
this recon ran on).

---

## FINDING FIRST

**stability-agent's filesystem event output is not merely mis-tagged — it is
structurally unreadable by the entire control plane, and always has been.**
Recon D15 identified one break (`service=None`). There are five, stacked, and
any one of them alone is fatal:

| # | Break | Evidence |
|---|---|---|
| 1 | **Wrong path subtree.** stability-agent writes `/opt/homelab/events/<YYYY-MM-DD>/<node>/events.jsonl`. node-agent writes `/opt/homelab/events/<node>/evt-*.json`. These are sibling subtrees, not the same store. | `stability_agent.py:47-51` vs `node_agent.py:261,287` |
| 2 | **Wrong file extension.** Observer and supervisor both discover events with `glob("**/*.json")`. `events.jsonl` does not match `*.json`. | `observer.py:912`, `supervisor.py:605` |
| 3 | **Wrong file format.** Even if matched, the observer does `json.load(f)` on the *whole file* — a multi-line JSONL append log is not a JSON document. | `observer.py:938` |
| 4 | **Never shipped off-node.** node-agent's rsync ships only `EVENTS_DIR/<node>/`. The date-partitioned subtree is outside that path, so piha's and solaria's stability-agent events never reach the VPS at all. | `node_agent.py:788` |
| 5 | **Wrong payload key + `service=None`** (recon D15). stability-agent writes `details`; the observer reads `payload`. And the *deployed* build still emits one aggregate event with no `service` key. | `stability_agent.py:42` vs `observer.py:701` |

Break #4 is confirmed empirically: on VPS, every date-partitioned directory
contains **only `vps/`** — there is no `piha/` or `solaria/` subdir in any of the
13 date dirs, despite both nodes having written thousands of local lines.

Break #5's repo-side fix exists but **is not deployed**: commit `f92e161`
("tag containers_not_running events with compose service") is dated **2026-07-29**;
the stability-agent image running on solaria was built **2026-05-17**. 100 % of the
5,056 deployed events sampled across three nodes carry `service: None`. So D15's
claim **is still true on current runtime**, while being *stale on current master* —
and it is moot either way, because breaks #1–#4 mean the observer never opens the
file.

Consequence: **all six of stability-agent's filesystem event types are inert** —
`containers_not_running`, `disk_usage_high`, `mqtt_unreachable`, `docker_api_error`,
`agent_error`, `redis_publish_error`. None reaches an incident. None reaches the
supervisor. This is not a per-type gap; it is the whole channel.

**The counterweight:** stability-agent has a *second*, entirely separate output
channel that node-agent does not have at all — Redis publishing to
`agent-system-redis` on piha — and **that channel is live, fresh, and exclusive**.
It is the sole populator of the agent-system WebUI's runtime view (4 node keys,
130 service keys, a 31,458-entry event stream, last write 14:29 today). Retiring
stability-agent blanks that UI.

So the two sensors do not overlap the way they appear to. They overlap on
*measurement* and diverge completely on *delivery*: node-agent feeds the
control plane, stability-agent feeds the agent-system UI, and stability-agent's
half of the event store is a write-only archive nothing has ever read.

A prior recon reached the same conclusion about the event path on 2026-07-06
(`docs/infra/prometheus-cutover-recon-2026-07-06.md:88-91`); it has not been acted on.

---

## 1. FEATURE MATRIX

| Capability | node-agent | stability-agent |
|---|---|---|
| **Source** | `services/node-agent/src/node_agent.py` (1116 ln) | `services/stability-agent/src/stability_agent.py` (386 ln) |
| **Cycle** | 60 s (`CHECK_INTERVAL`) | 60 s (`STABILITY_CHECK_INTERVAL`) |
| **Docker access** | docker-py SDK, `from_env()` | hand-rolled `UnixHTTPConnection` → `GET /containers/json?all=1` |
| **Sock mode** | **rw** | **ro** |
| **Why that mode** | Genuinely needs write: `container.restart()` (`:984`), `images.prune`, `containers.prune`, `api.prune_builds` (`:619-641`) | Read-only by construction — issues only HTTP GETs; no write code path exists |
| **Disk check** | `shutil.disk_usage(RUNTIME_PATH)` → the `/opt/homelab` partition. Warn 75 %, crit 85 % (hardcoded) | `shutil.disk_usage("/")` → **root fs, not the runtime partition**. Single threshold, `DISK_THRESHOLD_PCT` (default 90) |
| **RAM check** | ✅ `/proc/meminfo`, warn 85 % / crit 95 % | ❌ none |
| **CPU check** | ✅ two-sample `/proc/stat` delta, threshold 90 % | ❌ none |
| **Container state** | `list(all=True)`; skips `created`; **filters to managed only** (`restart_policy ∈ {unless-stopped, always, on-failure}`); reads docker `Health.Status` and `RestartCount` | all containers; skips `created`; **no restart-policy filter** (reports one-shot/manual containers too); no health-status read, no RestartCount |
| **Crash-loop detection** | ✅ `restarting` + `RestartCount ≥ 3` → `containers_not_running`; below threshold → `container_restarting` | ❌ `restarting` is simply "not running" |
| **Docker healthcheck status** | ✅ `running` + `unhealthy` → `healthcheck_failed` | ❌ not read |
| **MQTT probe** | ❌ none | ✅ TCP connect to `MQTT_HOST:MQTT_PORT`, 5 s timeout |
| **Tailscale check** | ❌ none | ✅ socket / `tailscale0` iface presence — **status file only, no event** |
| **Zigbee2MQTT check** | ❌ none | ✅ substring match over container list — **status file only, no event** |
| **Docker API error** | logged, no event | ✅ `docker_api_error` event |
| **Self-error reporting** | logged only | ✅ `agent_error` event |
| **Control-plane HTTP probe** | ✅ vps-only, `localhost:18180/summary` (`:1016`) | ❌ |
| **Docker cleanup / prune** | ✅ per-node-type policy (`:646`) | ❌ none — pure observer |
| **Control-plane FS rotation** | ✅ vps-only: old actions >7 d, deploy logs >30 d, checkpointed noise events >3 d (`:673`) | ❌ |
| **Dispatch execution** | ✅ pulls `actions/dispatch/<node>/`, executes `container_restart` (`:859-991`) | ❌ no action code path at all |
| **Event shipping to VPS** | ✅ rsync `--remove-source-files` (`:777`) | ❌ **none** — events stay on the node forever |
| **Event file layout** | `events/<node>/evt-<node>-<ts>-<type>-<svc>.json`, one file per event, key `payload` | `events/<date>/<node>/events.jsonl`, append-only JSONL, key `details` |
| **Read by observer?** | ✅ | ❌ (breaks #1–#5) |
| **Redis publishing** | ❌ none | ✅ `HSET homelab:nodes:<node>`, `HSET homelab:services:<node>:<container>`, `XADD homelab:events` |
| **Heartbeat** | `state/node-agent.heartbeat` (touch) | `state/stability-agent.heartbeat` + `state/stability-agent.json` status summary |
| **Deploy path** | **declarative** — in `hosts/*/services.yaml` (7 hosts) → `deploy.sh` / `deploy-node.sh` | **shadow** — `scripts/deploy/deploy-stability-agent.sh <node> [--ssh]` (hardcoded TS IPs) → `services/stability-agent/deploy-local.sh` → `compose up -d --build --force-recreate`. **Explicitly skipped** by `deploy-node.sh:93` because it ships a `deploy-local.sh` |
| **In desired state?** | yes, all 7 hosts | yes on `hosts/solaria` + `hosts/vps` (added in the 2026-07 truth cleanup — **recon B7 is stale here**), but the entry is inert: the deploy loop skips it |
| **Service contract files** | ❌ **no `service.yaml`, no README, no env.example, no healthcheck.sh** — violates the CLAUDE.md service layout | ✅ all four present |
| **Config surface** | `NODE_NAME`, `NODE_TYPE`, `VPS_EVENTS_HOST/USER/PATH`, `CHECK_INTERVAL`, `CRASH_LOOP_RESTART_THRESHOLD`, `RUNTIME_PATH`, `REPO_ROOT`. Thresholds hardcoded | `NODE_NAME`, `STABILITY_CHECK_INTERVAL`, `DISK_THRESHOLD_PCT`, `MQTT_HOST/PORT`, `REDIS_HOST/PORT/ENABLED` |
| **Per-node overrides** | `hosts/<node>/runtime/node-agent/` | `hosts/<node>/runtime/stability-agent/` (4 files: chelsty-infra, piha, solaria, vps) |
| **Per-node-type behavior** | ✅ real: `lte_node` no cleanup at all · `sd_card` dangling images + stopped containers, max 1×/24 h, no builder prune · `ai_node` + build cache, never `-a` · `standard` + control-plane FS rotation | ❌ **none.** No node-type concept. Only env values differ |
| **Memory limit (vps)** | not set in `hosts/vps/runtime/node-agent/` | `mem_limit: 96m`, `oom_score_adj: -900` |

### Event types, with trigger condition

**node-agent** — 11 types:

| Event | Severity | Condition |
|---|---|---|
| `disk_pressure` | high / medium | `/opt/homelab` usage ≥ 85 % / ≥ 75 % |
| `high_memory` | high / medium | MemAvailable-derived usage ≥ 95 % / ≥ 85 % |
| `high_cpu` | medium | CPU ≥ 90 % |
| `containers_not_running` | high | managed container `exited`/`dead`, **or** `restarting` with `RestartCount ≥ 3` |
| `container_restarting` | low | `restarting` with `RestartCount < 3` (observational, deliberately unwired) |
| `healthcheck_failed` | high | `running` + docker health `unhealthy` |
| `container_state_unexpected` | medium | `paused`, or any unhandled docker state |
| `service_healthy` | info | `running`, emitted **only** on the ¬healthy→healthy transition |
| `service_unhealthy` | high | vps-only: control-plane HTTP non-200 or unreachable |
| `node_health` | info | every cycle — carries disk/mem/cpu |
| `action_result` | info / high | after executing (or refusing) a dispatched action |

**stability-agent** — 6 types:

| Event | Severity | Condition |
|---|---|---|
| `containers_not_running` | warning | any container with `State ∉ {running, created}` |
| `disk_usage_high` | warning | `/` usage > `DISK_THRESHOLD_PCT` (90) |
| `mqtt_unreachable` | error | TCP connect to `MQTT_HOST:MQTT_PORT` fails — **only fires if `MQTT_HOST` is set** |
| `docker_api_error` | warning | sock exists but API request returns None |
| `agent_error` | error | unhandled exception in the main loop |
| `redis_publish_error` | warning | Redis publish threw |

### Config-surface defect found in passing

`deploy-local.sh` validates the node name against `piha|chelsty|solaria|vps`. The
repo has **`hosts/chelsty-infra/`**, not `hosts/chelsty/`. Deploying the chelsty
node therefore finds no override file and falls through to the compose default
`NODE_NAME=chelsty`. Live proof: Redis holds a `homelab:nodes:chelsty` key,
stale since 2026-06-02, alongside correctly-named `vps`/`piha`/`solaria` keys.
`hosts/chelsty-infra/runtime/stability-agent/docker-compose.override.yml` — the
only override carrying `MQTT_HOST` and a tuned `DISK_THRESHOLD_PCT=85` — has
never been applied.

---

## 2. UNIQUE TO stability-agent

Nine items. **One is functional.**

| # | Capability | Functional end-to-end? | Code path / reason |
|---|---|---|---|
| 1 | **Redis publishing** → `homelab:nodes:*`, `homelab:services:*`, `homelab:events` stream | ✅ **YES — and exclusive** | `stability_agent.py:330-372` → `agent-system-redis` on piha (100.108.208.3) → `services/agent-system/runtime-materializer/materializer.py:122,138,155` reads all three → writes the agent-system WebUI's world dir. Verified live: node keys refreshed 2026-07-30T14:28, service keys 14:29. **Not an incident path** — it feeds a UI, never the supervisor. node-agent has no Redis code at all. |
| 2 | `containers_not_running` (its own) | ❌ dead | Breaks #1–#5 above. Additionally the deployed build emits the pre-fix aggregate form. Never opens an incident; `observer.py:770` would handle it *if* the file were ever read. |
| 3 | `disk_usage_high` | ❌ dead ×2 | Unreadable (#1–#4), **and** the type appears nowhere in `observer.process_event`'s if/elif chain — the observer only knows `disk_pressure`. Even hand-fed a well-formed file, it would bump `last_seen` and fall through. Not in any supervisor trigger set. |
| 4 | `docker_api_error` | ❌ dead ×2 | Same: unreadable, and no branch in `process_event`, no supervisor trigger. Despite being the **2nd-most-emitted type on VPS (496 events)**. |
| 5 | `agent_error` | ❌ dead ×2 | Same. Self-diagnostics that nothing collects. |
| 6 | `redis_publish_error` | ❌ dead ×2 | Same. |
| 7 | `mqtt_unreachable` (TCP probe) | ❌ dead ×3 | Unreadable; **`MQTT_HOST` is unset on piha/solaria/vps** so the probe never even runs there (only the never-applied chelsty-infra override sets it); and the type was **removed from `CONTAINER_RESTART_TRIGGERS` on 2026-07-28** (`supervisor.py:47-51`) precisely because the observer never creates such an incident. |
| 8 | Tailscale presence check | ❌ inert | `stability_agent.py:189` — result goes to `state/stability-agent.json` only. No event emitted. Grep confirms **no consumer** of that file anywhere in the repo outside its own docs. |
| 9 | Zigbee2MQTT presence/running check | ❌ inert | `stability_agent.py:297-310` — same status-file-only fate. Also a naive `"zigbee2mqtt" in name` substring match. |

Two structural properties also unique to it, neither a "capability":

- **`:ro` sock mount** — a safety property. It is the only sock-holder on the fleet that is read-only by mount (recon A1). Preserving it means preserving a container whose *outputs* are dead.
- **Broader container coverage** — no restart-policy filter, so it sees one-shot and manually-run containers that node-agent deliberately ignores. Inert, since the events go nowhere.

**Bonus:** `scripts/lib/events.sh` — the documented shell event emitter — writes to
the *same* dead date-partitioned JSONL path (`scripts/lib/events.sh:18-20`). Every
shell-emitted event in the system shares stability-agent's fate. CLAUDE.md
("Events are append-only JSON lines at `/opt/homelab/events/YYYY-MM-DD/<node>/events.jsonl`")
documents the dead format as the canonical one.

---

## 3. UNIQUE TO node-agent

| # | Capability | Notes |
|---|---|---|
| 1 | **Dispatch execution** (`container_restart`) | The on-node hands of the HITL loop. Type whitelist of exactly one, self-restart guard, idempotency markers, node-scoping check (`:921-991`). No equivalent exists in stability-agent — and adding one would require flipping its sock to rw. |
| 2 | **`action_result` reporting** | Closes the executor's loop (`:993`); without it actions hang in `running/`. |
| 3 | **Event shipping to VPS** | rsync `--remove-source-files` (`:777`). This is *the* mechanism by which any off-VPS node reaches the control plane. stability-agent has no shipping whatsoever. |
| 4 | **An event format the control plane can read** | One `*.json` file per event, `evt-<node>-<ts>-<type>-<svc>` naming that carries the checkpoint ordering key, `payload` field. Everything downstream is built around this shape. |
| 5 | **RAM monitoring** (`high_memory`) | Genuinely absent from stability-agent. Matters most on the 4 GiB no-swap VPS. |
| 6 | **CPU monitoring** (`high_cpu`) | Ditto. 121 events on vps, 129 on piha, 48 on lustro in the retained window. |
| 7 | **Crash-loop detection** | `RestartCount ≥ 3` threshold distinguishes a real flap from deploy churn (`:482-516`). stability-agent cannot tell them apart. |
| 8 | **Docker healthcheck-status reading** → `healthcheck_failed` | 3,375 events on piha. The single largest actionable signal in the fleet after `containers_not_running`. |
| 9 | **`service_healthy` positive confirmation** | Keeps `services.json` populated so the supervisor's drift loop doesn't classify every desired service as `missing_service` and flood the action queue (`:528-549`, `observer.py:762`). Load-bearing. |
| 10 | **`node_health` heartbeat** | Drives observer node liveness (`node_offline`/`node_stale`/`node_online`). The single highest-volume event type fleet-wide. |
| 11 | **Docker cleanup, per-node-type** | The whole `lte_node`/`sd_card`/`ai_node`/`standard` policy (`:646-667`). SD-card write-cycle protection and the "never `-a` on ai_node" rule live only here. |
| 12 | **Control-plane FS rotation** (vps) | Old actions, deploy logs, checkpointed noise-event sweep (`:673`). The fix for the 358k-file backlog that paralyzed the supervisor. |
| 13 | **Control-plane HTTP health probe** (vps) | The only thing that maps the 4-container control-plane stack to the single `control-plane` key in `services.yaml` (`:1016`). |
| 14 | **Declarative deploy path** | In `hosts/*/services.yaml`, deployed by the normal pipeline. |

Node-agent's own gaps, for symmetry: no `service.yaml`/README/env.example/healthcheck.sh
(CLAUDE.md violation); no memory limit declared on VPS despite the VPS rule; rw sock
with **no allowlist of restartable container names** (recon A2); and on solaria the
docker half is broken by a group_add gid mismatch — though note that as of today
solaria's node-agent container reports `Up 28 minutes (healthy)`, so this is worth
re-verifying separately before relying on either state.

---

## 4. RUNTIME EVIDENCE

Collected 2026-07-30 ~14:30. VPS event store = `/opt/homelab/events/`.

### Container liveness

| Node | stability-agent | node-agent |
|---|---|---|
| vps | running, **Up 7 weeks** (healthy) | running, Up 13 days (healthy) |
| piha | running, Up 13 days (healthy) | running, Up 7 days (healthy) |
| solaria | running, Up ~1 h (healthy) | running, Up 28 min (healthy) |

Both are running everywhere. Neither is crashed. This is a delivery problem, not an
availability problem.

### Events reaching the VPS control plane (node-agent subtree, `events/<node>/`)

| Node | Files | Top types |
|---|---|---|
| piha | **33,058** | `ha_entity_unavailable_long` 22,897 · `node_health` 4,112 · `healthcheck_failed` 3,375 · `containers_not_running` 2,488 · `high_cpu` 129 · `ha_*` 47 · `action_result` 2 |
| lustro | **11,935** | `service_healthy` 8,893 · `node_health` 2,940 · `high_cpu` 48 · liveness 54 |
| vps | **4,520** | `node_health` 4,232 · `disk_pressure` 132 · `high_cpu` 121 · `service_unhealthy` 34 · `containers_not_running` 1 |
| solaria | **1,260** | `node_health` 1,211 · liveness 42 · `service_healthy` 5 · `containers_not_running` 2 |
| chelsty-infra | **0** | node offline |
| **Total** | **50,773** | all from node-agent / ha-diag-agent / observer |

### Events from stability-agent (date-partitioned subtree)

| Node | Reaches VPS? | Local volume | Types | Last event |
|---|---|---|---|---|
| vps | same fs, but unreadable | **2,240** all-time (13 date dirs) | `containers_not_running` 1,744 · `docker_api_error` 496 | **2026-07-16** — nothing for 14 days |
| piha | ❌ **never shipped** | **2,528** in July alone | `containers_not_running` 2,527 · `docker_api_error` 1 | 2026-07-27 |
| solaria | ❌ **never shipped** | 2,549 all-time / **288** in July | `containers_not_running` 287 · `docker_api_error` 1 | 2026-07-23 |

`service` field across all 5,056 events sampled: **`None`, 100 %.** No exceptions on
any node.

Confirmation of break #4: the 13 date directories on VPS contain **only a `vps/`
subdirectory** — no `piha/`, no `solaria/`, ever. piha's and solaria's ~5,000 lines
have never left their nodes.

### Does any stability-agent event produce an incident? — **No. Zero. On any node.**

`world/incidents.json` on VPS contains exactly **one** incident:

```
inc-1784736375-piha-homeassistant
  node=piha  service=homeassistant  status=active
  trigger_type=containers_not_running  occurrence_count=1307
```

Traced to the **node-agent** subtree (`events/piha/evt-*.json`) — it has a
`service` field, which is the precondition `observer.py:754` requires to key a
service and open an incident. stability-agent contributed nothing to it.

**Quantified answer to the question asked:** over the full retained window,
node-agent's channel produced 50,773 events and 1 incident. stability-agent's
channel produced 7,317 events and **0** incidents. Its incident yield is not
low — it is structurally zero, and has been since the service was first deployed
(2026-05-17).

### The Redis channel — live and exclusive

| Metric | Value |
|---|---|
| `homelab:nodes:*` keys | 4 (vps, piha, solaria + stale `chelsty` from 2026-06-02) |
| `homelab:services:*` keys | 130 |
| `homelab:incidents:*` keys | 1 |
| `homelab:events` stream | 31,458 entries |
| Freshness | `vps` 14:28:29, `piha` 14:28:27, `solaria` 14:27:52, services 14:29:27 — **all today** |
| Consumers running on piha | `agent-system-redis`, `agent-system-runtime-materializer`, `agent-system-webui`, `agent-system-telegram-bot` |

Every one of those keys is written by stability-agent and by nothing else. The
service-key set also shows the deployed build's staleness — entries like
`homelab:services:vps:dc8be279ea1a_node-agent` carry the docker stale-state hash
prefix that `container_service_name()` was written to strip.

---

## 5. COST OF EACH OPTION

### (a) Retire stability-agent entirely

**Monitoring capability lost:**
- Nothing that currently reaches an incident, an action, or an operator alert. All six filesystem event types are already inert (§2).
- MQTT TCP reachability probe — the only fleet component that probes a broker port. Currently unconfigured on every online node and untriggerable, so the loss is *prospective*, not actual. Relevant to chelsty (offline-first, Mosquitto is load-bearing) if that site is revived.
- Tailscale presence and Zigbee2MQTT presence checks — inert today, but they are the only code that looks at either.
- Container visibility without a restart-policy filter (one-shot/manual containers).
- Disk monitoring of `/` as distinct from `/opt/homelab` — on any node where those are separate mounts, node-agent does not watch `/`.

**What breaks — the real cost:**
- **The agent-system WebUI goes blank.** `homelab:nodes:*`, `homelab:services:*` and the `homelab:events` stream lose their only writer. `runtime-materializer` continues polling and materializes empty sets. This is a live, working, user-facing surface — 130 service keys refreshed a minute before this recon was written.
- Any Telegram-bot or WebUI view built on that materialized world dir degrades with it (not separately verified in this recon).
- The fleet loses its only `:ro`-sock container; every remaining sock holder is rw (recon A1).
- `hosts/vps/runtime/stability-agent/` was allotted 96 MiB of the 3,200 MiB VPS budget — reclaimed.

**Deploy/config cleanup entailed:**
- Delete `services/stability-agent/` (8 files), `scripts/deploy/deploy-stability-agent.sh`, 4 × `hosts/*/runtime/stability-agent/docker-compose.override.yml`.
- Remove entries from `hosts/solaria/services.yaml` and `hosts/vps/services.yaml`.
- `docker compose down` on vps, piha, solaria (chelsty-infra when reachable).
- Purge or archive `/opt/homelab/events/2026-*/` on all four nodes (~5 MB on piha, mostly May).
- Update CLAUDE.md (agent-system architecture §1, event-path claim at line 100), `docs/chelsty-stability-agent.md`, recon A1/A2/B7.
- **Not covered by the cleanup:** the Redis publisher must be rehomed first or the UI loss is permanent.

### (b) Merge stability-agent's unique checks into node-agent

**Which checks would actually move — four, and only one is non-trivial:**

| Check | Surface area |
|---|---|
| **Redis publishing** (the load-bearing one) | ~60 lines: a `RedisClient` (or the `redis` package, already a dependency of agent-system), plus the `homelab:nodes:*` / `homelab:services:*` HSET loop and the `homelab:events` XADD. node-agent already computes every input — disk/mem/cpu in `run_once()`, container state in `check_containers()`. It needs a publish hook, `REDIS_HOST/PORT/ENABLED` env vars in the compose and in 4–7 host overrides, and a decision on whether an offline-first node may block on a piha-hosted Redis (must be fail-open; stability-agent's version already is). **This alone preserves the only functional output.** |
| **MQTT TCP probe** | ~15 lines: `socket.create_connection((MQTT_HOST, MQTT_PORT), timeout=5)` → emit `mqtt_unreachable`. Plus `MQTT_HOST`/`MQTT_PORT` env. Note: to be *useful* it also needs an observer branch and a supervisor trigger — the type was deliberately removed from `CONTAINER_RESTART_TRIGGERS` on 2026-07-28 because no incident path existed. So the honest surface is 15 lines in node-agent **plus** an observer `process_event` branch **plus** a supervisor mapping. |
| **Tailscale presence** | ~10 lines, two `os.path.exists` calls. Needs a new event type and observer/supervisor wiring to be worth anything; as a status-file value it is pointless. |
| **Zigbee2MQTT presence** | ~5 lines, but it is a site-specific check on a node that is offline. Arguably belongs in a chelsty-specific probe, not the fleet sensor. |
| **`/` disk check alongside `/opt/homelab`** | ~10 lines: a second `shutil.disk_usage("/")` and a mount label in the payload. |
| **Drop the restart-policy filter** | Not a merge — a behavior change to `check_containers()` with a known noise cost; the filter exists on purpose. |

**Not moving:** `docker_api_error`, `agent_error`, `redis_publish_error` — node-agent
already logs the equivalents; promoting them to events adds volume with no consumer
unless observer branches are added too.

Rough total: **~100–120 lines in `node_agent.py`**, env additions across
`services/node-agent/docker-compose.yml` and 4–7 host overrides, plus tests
(Definition of Done: docker build + smoke run + pytest — node-agent has 5 test files
to extend). The merged agent keeps its rw sock, so the fleet's last `:ro` sock holder
disappears either way. Requires a coordinated cutover: node-agent must be publishing
to Redis *before* stability-agent stops, or the WebUI gaps.

### (c) Keep both

**Duplicated cost, as measured:**

| Dimension | Cost |
|---|---|
| **Deploy paths** | Two mechanisms for the same job. stability-agent's is the shadow path — hardcoded Tailscale IPs, explicitly skipped by `deploy-node.sh:93`, a node-name whitelist that doesn't match the repo's own host dirs (`chelsty` vs `chelsty-infra`), and `--force-recreate` on every run. Its `hosts/*/services.yaml` entries are decorative: the pipeline skips them, so desired state claims coverage the pipeline does not provide. |
| **Undeployed fixes** | The gap is not theoretical. `f92e161` (2026-07-29) is in master; the vps image is from 2026-05-17 and has been running 7 weeks. Nobody noticed, because fixing the `service` tag changes nothing while breaks #1–#4 stand. A second deploy path is a second thing to forget. |
| **Sock exposure** | Two containers per node hold `/var/run/docker.sock`. stability-agent's is `:ro` and its output is dead — the fleet pays the attack surface of a sock mount on every node for a channel with zero consumers. |
| **Event volume** | ~7,300 dead events written to disk across three nodes, ~5 MB, in 13–20 date directories per node that no retention sweep touches (node-agent's `_cleanup_control_plane_fs` only prunes `service_healthy`/`node_health` under `events/<node>/`, never the date subtree). It grows without bound and nothing reads it. |
| **Confusion — the compounding cost** | CLAUDE.md documents the dead JSONL path as *the* canonical event format (line 100). `scripts/lib/events.sh` — the sanctioned shell emitter — writes to it. Recon D15 diagnosed one symptom (`service=None`) and a fix was written and merged for it, when four upstream breaks made that fix inert. The 2026-07-06 prometheus recon reached the correct conclusion and it was not carried forward. Two sensors emitting the same event type name into two incompatible stores means every future "why didn't this alert fire" investigation starts by disambiguating which agent, which store, which format. |
| **What it costs to keep it honest** | Nothing about (c) is stable as-is: keeping both means either accepting a permanently dead channel, or fixing breaks #1–#4 — which is a rewrite of stability-agent's emitter into node-agent's format, i.e. option (b) performed inside a second codebase. |

---

## DECISION NEEDED

The choice is whether the Redis→agent-system-WebUI publisher is worth a second
per-node daemon with a docker.sock mount and a parallel deploy path, or whether
it moves into node-agent. Everything else about stability-agent is already dead:
all six of its filesystem event types are unreadable by the observer and
supervisor for four independent structural reasons beyond the one recon D15
named, its events have never opened an incident on any node in the service's
lifetime, and the two nodes that are not the VPS have never shipped a single one
of them anywhere. Against that, it is the *only* writer of the 4 node keys, 130
service keys and 31,458-entry stream that the agent-system UI renders — refreshed
one minute before this recon was written — and node-agent has no Redis code at
all. So (a) retire is the cheapest cleanup but blanks a live UI unless the
publisher is rehomed first; (b) merge is ~100–120 lines plus env plumbing across
4–7 host overrides plus a coordinated cutover, and ends with one sensor, one
deploy path, one sock, one event format; (c) keep both preserves the UI at zero
engineering cost today, but the duplicated cost is not static — it is a
permanently dead event channel that the repo's own CLAUDE.md documents as
canonical, a shadow deploy path that has already silently withheld a merged fix
for 7 weeks, and a second sock mount on every node buying nothing. The key
trade-off is therefore not "monitoring coverage" — node-agent strictly dominates
on every check that reaches a consumer — but **whether to pay a one-time
migration of the Redis publisher now, or keep paying the ambiguity of two
sensors, two event stores and two deploy paths indefinitely.**