homelab-codex-ws/kb/audits/czujniki-2026-07-30.md
oskar 9f77a723e8 feat(kb): 5 audytow/reconow -> kb/audits/ (type: audit, as_of)
czujniki-2026-07-30 (node-agent vs stability-agent)
lustro-shipping-2026-07-16 (event=dead prom=up, 1507 mismatchy)
prometheus-cutover-2026-07-06 (recon starego toru livenesci)
piha-slim-2026-07-02 (audyt odchudzania PIHA)
vps-stacki-2026-07-27 (audyt niezarzadzanych stackow na VPS)

ODSTEPSTWO OD RECONU — swiadome. Recon typowal te 5 plikow jako SPLIT
(audit+decision / audit+incident / audit+phase). Rozstrzygniecie 2 wprowadza
typ `audit` z polem as_of i mapuje kazdy z nich na JEDNA sciezke
kb/audits/<obszar>-<data>.md. Audyt jest spojna migawka stanu z konkretna
data — rozbicie go na "ustalenia" i "rekomendacje" rozerwaloby ten kontekst
i wymagaloby redakcji tresci, czego etap 2 zabrania. Zostaja w calosci.

Efekt: 29 SPLIT-ow z reconu realizowane jako 24 (10 service+runbook,
14 wielotypowych), 5 zamienionych na caloscowe dokumenty type: audit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:58:46 +02:00

30 KiB
Raw Blame History

okf type visibility status updated as_of links
0.1 audit private active 2026-07-30 2026-07-30

Recon — node-agent vs stability-agent (2026-07-30)

Read-only recon. Ground truth: docs/architecture/RECON-multiagent-2026-07-27.md (A1, A2, B7, D15). Source read at master @ 473bf8e; this branch is cut from master @ 0650eb8. The two intervening merges (0650eb8 ha-mcp, cb8a19d kb-query tests) touch none of the audited paths — services/node-agent/, services/stability-agent/, scripts/observer/, services/control-plane/ — so every finding and line citation below holds at 0650eb8. Runtime evidence collected 2026-07-30 ~14:30 UTC from vps, piha and solaria (solaria = the host this recon ran on).


FINDING FIRST

stability-agent's filesystem event output is not merely mis-tagged — it is structurally unreadable by the entire control plane, and always has been. Recon D15 identified one break (service=None). There are five, stacked, and any one of them alone is fatal:

# Break Evidence
1 Wrong path subtree. stability-agent writes /opt/homelab/events/<YYYY-MM-DD>/<node>/events.jsonl. node-agent writes /opt/homelab/events/<node>/evt-*.json. These are sibling subtrees, not the same store. stability_agent.py:47-51 vs node_agent.py:261,287
2 Wrong file extension. Observer and supervisor both discover events with glob("**/*.json"). events.jsonl does not match *.json. observer.py:912, supervisor.py:605
3 Wrong file format. Even if matched, the observer does json.load(f) on the whole file — a multi-line JSONL append log is not a JSON document. observer.py:938
4 Never shipped off-node. node-agent's rsync ships only EVENTS_DIR/<node>/. The date-partitioned subtree is outside that path, so piha's and solaria's stability-agent events never reach the VPS at all. node_agent.py:788
5 Wrong payload key + service=None (recon D15). stability-agent writes details; the observer reads payload. And the deployed build still emits one aggregate event with no service key. stability_agent.py:42 vs observer.py:701

Break #4 is confirmed empirically: on VPS, every date-partitioned directory contains only vps/ — there is no piha/ or solaria/ subdir in any of the 13 date dirs, despite both nodes having written thousands of local lines.

Break #5's repo-side fix exists but is not deployed: commit f92e161 ("tag containers_not_running events with compose service") is dated 2026-07-29; the stability-agent image running on solaria was built 2026-05-17. 100 % of the 5,056 deployed events sampled across three nodes carry service: None. So D15's claim is still true on current runtime, while being stale on current master — and it is moot either way, because breaks #1#4 mean the observer never opens the file.

Consequence: all six of stability-agent's filesystem event types are inertcontainers_not_running, disk_usage_high, mqtt_unreachable, docker_api_error, agent_error, redis_publish_error. None reaches an incident. None reaches the supervisor. This is not a per-type gap; it is the whole channel.

The counterweight: stability-agent has a second, entirely separate output channel that node-agent does not have at all — Redis publishing to agent-system-redis on piha — and that channel is live, fresh, and exclusive. It is the sole populator of the agent-system WebUI's runtime view (4 node keys, 130 service keys, a 31,458-entry event stream, last write 14:29 today). Retiring stability-agent blanks that UI.

So the two sensors do not overlap the way they appear to. They overlap on measurement and diverge completely on delivery: node-agent feeds the control plane, stability-agent feeds the agent-system UI, and stability-agent's half of the event store is a write-only archive nothing has ever read.

A prior recon reached the same conclusion about the event path on 2026-07-06 (docs/infra/prometheus-cutover-recon-2026-07-06.md:88-91); it has not been acted on.


1. FEATURE MATRIX

Capability node-agent stability-agent
Source services/node-agent/src/node_agent.py (1116 ln) services/stability-agent/src/stability_agent.py (386 ln)
Cycle 60 s (CHECK_INTERVAL) 60 s (STABILITY_CHECK_INTERVAL)
Docker access docker-py SDK, from_env() hand-rolled UnixHTTPConnectionGET /containers/json?all=1
Sock mode rw ro
Why that mode Genuinely needs write: container.restart() (:984), images.prune, containers.prune, api.prune_builds (:619-641) Read-only by construction — issues only HTTP GETs; no write code path exists
Disk check shutil.disk_usage(RUNTIME_PATH) → the /opt/homelab partition. Warn 75 %, crit 85 % (hardcoded) shutil.disk_usage("/")root fs, not the runtime partition. Single threshold, DISK_THRESHOLD_PCT (default 90)
RAM check /proc/meminfo, warn 85 % / crit 95 % none
CPU check two-sample /proc/stat delta, threshold 90 % none
Container state list(all=True); skips created; filters to managed only (restart_policy ∈ {unless-stopped, always, on-failure}); reads docker Health.Status and RestartCount all containers; skips created; no restart-policy filter (reports one-shot/manual containers too); no health-status read, no RestartCount
Crash-loop detection restarting + RestartCount ≥ 3containers_not_running; below threshold → container_restarting restarting is simply "not running"
Docker healthcheck status running + unhealthyhealthcheck_failed not read
MQTT probe none TCP connect to MQTT_HOST:MQTT_PORT, 5 s timeout
Tailscale check none socket / tailscale0 iface presence — status file only, no event
Zigbee2MQTT check none substring match over container list — status file only, no event
Docker API error logged, no event docker_api_error event
Self-error reporting logged only agent_error event
Control-plane HTTP probe vps-only, localhost:18180/summary (:1016)
Docker cleanup / prune per-node-type policy (:646) none — pure observer
Control-plane FS rotation vps-only: old actions >7 d, deploy logs >30 d, checkpointed noise events >3 d (:673)
Dispatch execution pulls actions/dispatch/<node>/, executes container_restart (:859-991) no action code path at all
Event shipping to VPS rsync --remove-source-files (:777) none — events stay on the node forever
Event file layout events/<node>/evt-<node>-<ts>-<type>-<svc>.json, one file per event, key payload events/<date>/<node>/events.jsonl, append-only JSONL, key details
Read by observer? (breaks #1#5)
Redis publishing none HSET homelab:nodes:<node>, HSET homelab:services:<node>:<container>, XADD homelab:events
Heartbeat state/node-agent.heartbeat (touch) state/stability-agent.heartbeat + state/stability-agent.json status summary
Deploy path declarative — in hosts/*/services.yaml (7 hosts) → deploy.sh / deploy-node.sh shadowscripts/deploy/deploy-stability-agent.sh <node> [--ssh] (hardcoded TS IPs) → services/stability-agent/deploy-local.shcompose up -d --build --force-recreate. Explicitly skipped by deploy-node.sh:93 because it ships a deploy-local.sh
In desired state? yes, all 7 hosts yes on hosts/solaria + hosts/vps (added in the 2026-07 truth cleanup — recon B7 is stale here), but the entry is inert: the deploy loop skips it
Service contract files no service.yaml, no README, no env.example, no healthcheck.sh — violates the CLAUDE.md service layout all four present
Config surface NODE_NAME, NODE_TYPE, VPS_EVENTS_HOST/USER/PATH, CHECK_INTERVAL, CRASH_LOOP_RESTART_THRESHOLD, RUNTIME_PATH, REPO_ROOT. Thresholds hardcoded NODE_NAME, STABILITY_CHECK_INTERVAL, DISK_THRESHOLD_PCT, MQTT_HOST/PORT, REDIS_HOST/PORT/ENABLED
Per-node overrides hosts/<node>/runtime/node-agent/ hosts/<node>/runtime/stability-agent/ (4 files: chelsty-infra, piha, solaria, vps)
Per-node-type behavior real: lte_node no cleanup at all · sd_card dangling images + stopped containers, max 1×/24 h, no builder prune · ai_node + build cache, never -a · standard + control-plane FS rotation none. No node-type concept. Only env values differ
Memory limit (vps) not set in hosts/vps/runtime/node-agent/ mem_limit: 96m, oom_score_adj: -900

Event types, with trigger condition

node-agent — 11 types:

Event Severity Condition
disk_pressure high / medium /opt/homelab usage ≥ 85 % / ≥ 75 %
high_memory high / medium MemAvailable-derived usage ≥ 95 % / ≥ 85 %
high_cpu medium CPU ≥ 90 %
containers_not_running high managed container exited/dead, or restarting with RestartCount ≥ 3
container_restarting low restarting with RestartCount < 3 (observational, deliberately unwired)
healthcheck_failed high running + docker health unhealthy
container_state_unexpected medium paused, or any unhandled docker state
service_healthy info running, emitted only on the ¬healthy→healthy transition
service_unhealthy high vps-only: control-plane HTTP non-200 or unreachable
node_health info every cycle — carries disk/mem/cpu
action_result info / high after executing (or refusing) a dispatched action

stability-agent — 6 types:

Event Severity Condition
containers_not_running warning any container with State ∉ {running, created}
disk_usage_high warning / usage > DISK_THRESHOLD_PCT (90)
mqtt_unreachable error TCP connect to MQTT_HOST:MQTT_PORT fails — only fires if MQTT_HOST is set
docker_api_error warning sock exists but API request returns None
agent_error error unhandled exception in the main loop
redis_publish_error warning Redis publish threw

Config-surface defect found in passing

deploy-local.sh validates the node name against piha|chelsty|solaria|vps. The repo has hosts/chelsty-infra/, not hosts/chelsty/. Deploying the chelsty node therefore finds no override file and falls through to the compose default NODE_NAME=chelsty. Live proof: Redis holds a homelab:nodes:chelsty key, stale since 2026-06-02, alongside correctly-named vps/piha/solaria keys. hosts/chelsty-infra/runtime/stability-agent/docker-compose.override.yml — the only override carrying MQTT_HOST and a tuned DISK_THRESHOLD_PCT=85 — has never been applied.


2. UNIQUE TO stability-agent

Nine items. One is functional.

# Capability Functional end-to-end? Code path / reason
1 Redis publishinghomelab:nodes:*, homelab:services:*, homelab:events stream YES — and exclusive stability_agent.py:330-372agent-system-redis on piha (100.108.208.3) → services/agent-system/runtime-materializer/materializer.py:122,138,155 reads all three → writes the agent-system WebUI's world dir. Verified live: node keys refreshed 2026-07-30T14:28, service keys 14:29. Not an incident path — it feeds a UI, never the supervisor. node-agent has no Redis code at all.
2 containers_not_running (its own) dead Breaks #1#5 above. Additionally the deployed build emits the pre-fix aggregate form. Never opens an incident; observer.py:770 would handle it if the file were ever read.
3 disk_usage_high dead ×2 Unreadable (#1#4), and the type appears nowhere in observer.process_event's if/elif chain — the observer only knows disk_pressure. Even hand-fed a well-formed file, it would bump last_seen and fall through. Not in any supervisor trigger set.
4 docker_api_error dead ×2 Same: unreadable, and no branch in process_event, no supervisor trigger. Despite being the 2nd-most-emitted type on VPS (496 events).
5 agent_error dead ×2 Same. Self-diagnostics that nothing collects.
6 redis_publish_error dead ×2 Same.
7 mqtt_unreachable (TCP probe) dead ×3 Unreadable; MQTT_HOST is unset on piha/solaria/vps so the probe never even runs there (only the never-applied chelsty-infra override sets it); and the type was removed from CONTAINER_RESTART_TRIGGERS on 2026-07-28 (supervisor.py:47-51) precisely because the observer never creates such an incident.
8 Tailscale presence check inert stability_agent.py:189 — result goes to state/stability-agent.json only. No event emitted. Grep confirms no consumer of that file anywhere in the repo outside its own docs.
9 Zigbee2MQTT presence/running check inert stability_agent.py:297-310 — same status-file-only fate. Also a naive "zigbee2mqtt" in name substring match.

Two structural properties also unique to it, neither a "capability":

  • :ro sock mount — a safety property. It is the only sock-holder on the fleet that is read-only by mount (recon A1). Preserving it means preserving a container whose outputs are dead.
  • Broader container coverage — no restart-policy filter, so it sees one-shot and manually-run containers that node-agent deliberately ignores. Inert, since the events go nowhere.

Bonus: scripts/lib/events.sh — the documented shell event emitter — writes to the same dead date-partitioned JSONL path (scripts/lib/events.sh:18-20). Every shell-emitted event in the system shares stability-agent's fate. CLAUDE.md ("Events are append-only JSON lines at /opt/homelab/events/YYYY-MM-DD/<node>/events.jsonl") documents the dead format as the canonical one.


3. UNIQUE TO node-agent

# Capability Notes
1 Dispatch execution (container_restart) The on-node hands of the HITL loop. Type whitelist of exactly one, self-restart guard, idempotency markers, node-scoping check (:921-991). No equivalent exists in stability-agent — and adding one would require flipping its sock to rw.
2 action_result reporting Closes the executor's loop (:993); without it actions hang in running/.
3 Event shipping to VPS rsync --remove-source-files (:777). This is the mechanism by which any off-VPS node reaches the control plane. stability-agent has no shipping whatsoever.
4 An event format the control plane can read One *.json file per event, evt-<node>-<ts>-<type>-<svc> naming that carries the checkpoint ordering key, payload field. Everything downstream is built around this shape.
5 RAM monitoring (high_memory) Genuinely absent from stability-agent. Matters most on the 4 GiB no-swap VPS.
6 CPU monitoring (high_cpu) Ditto. 121 events on vps, 129 on piha, 48 on lustro in the retained window.
7 Crash-loop detection RestartCount ≥ 3 threshold distinguishes a real flap from deploy churn (:482-516). stability-agent cannot tell them apart.
8 Docker healthcheck-status readinghealthcheck_failed 3,375 events on piha. The single largest actionable signal in the fleet after containers_not_running.
9 service_healthy positive confirmation Keeps services.json populated so the supervisor's drift loop doesn't classify every desired service as missing_service and flood the action queue (:528-549, observer.py:762). Load-bearing.
10 node_health heartbeat Drives observer node liveness (node_offline/node_stale/node_online). The single highest-volume event type fleet-wide.
11 Docker cleanup, per-node-type The whole lte_node/sd_card/ai_node/standard policy (:646-667). SD-card write-cycle protection and the "never -a on ai_node" rule live only here.
12 Control-plane FS rotation (vps) Old actions, deploy logs, checkpointed noise-event sweep (:673). The fix for the 358k-file backlog that paralyzed the supervisor.
13 Control-plane HTTP health probe (vps) The only thing that maps the 4-container control-plane stack to the single control-plane key in services.yaml (:1016).
14 Declarative deploy path In hosts/*/services.yaml, deployed by the normal pipeline.

Node-agent's own gaps, for symmetry: no service.yaml/README/env.example/healthcheck.sh (CLAUDE.md violation); no memory limit declared on VPS despite the VPS rule; rw sock with no allowlist of restartable container names (recon A2); and on solaria the docker half is broken by a group_add gid mismatch — though note that as of today solaria's node-agent container reports Up 28 minutes (healthy), so this is worth re-verifying separately before relying on either state.


4. RUNTIME EVIDENCE

Collected 2026-07-30 ~14:30. VPS event store = /opt/homelab/events/.

Container liveness

Node stability-agent node-agent
vps running, Up 7 weeks (healthy) running, Up 13 days (healthy)
piha running, Up 13 days (healthy) running, Up 7 days (healthy)
solaria running, Up ~1 h (healthy) running, Up 28 min (healthy)

Both are running everywhere. Neither is crashed. This is a delivery problem, not an availability problem.

Events reaching the VPS control plane (node-agent subtree, events/<node>/)

Node Files Top types
piha 33,058 ha_entity_unavailable_long 22,897 · node_health 4,112 · healthcheck_failed 3,375 · containers_not_running 2,488 · high_cpu 129 · ha_* 47 · action_result 2
lustro 11,935 service_healthy 8,893 · node_health 2,940 · high_cpu 48 · liveness 54
vps 4,520 node_health 4,232 · disk_pressure 132 · high_cpu 121 · service_unhealthy 34 · containers_not_running 1
solaria 1,260 node_health 1,211 · liveness 42 · service_healthy 5 · containers_not_running 2
chelsty-infra 0 node offline
Total 50,773 all from node-agent / ha-diag-agent / observer

Events from stability-agent (date-partitioned subtree)

Node Reaches VPS? Local volume Types Last event
vps same fs, but unreadable 2,240 all-time (13 date dirs) containers_not_running 1,744 · docker_api_error 496 2026-07-16 — nothing for 14 days
piha never shipped 2,528 in July alone containers_not_running 2,527 · docker_api_error 1 2026-07-27
solaria never shipped 2,549 all-time / 288 in July containers_not_running 287 · docker_api_error 1 2026-07-23

service field across all 5,056 events sampled: None, 100 %. No exceptions on any node.

Confirmation of break #4: the 13 date directories on VPS contain only a vps/ subdirectory — no piha/, no solaria/, ever. piha's and solaria's ~5,000 lines have never left their nodes.

Does any stability-agent event produce an incident? — No. Zero. On any node.

world/incidents.json on VPS contains exactly one incident:

inc-1784736375-piha-homeassistant
  node=piha  service=homeassistant  status=active
  trigger_type=containers_not_running  occurrence_count=1307

Traced to the node-agent subtree (events/piha/evt-*.json) — it has a service field, which is the precondition observer.py:754 requires to key a service and open an incident. stability-agent contributed nothing to it.

Quantified answer to the question asked: over the full retained window, node-agent's channel produced 50,773 events and 1 incident. stability-agent's channel produced 7,317 events and 0 incidents. Its incident yield is not low — it is structurally zero, and has been since the service was first deployed (2026-05-17).

The Redis channel — live and exclusive

Metric Value
homelab:nodes:* keys 4 (vps, piha, solaria + stale chelsty from 2026-06-02)
homelab:services:* keys 130
homelab:incidents:* keys 1
homelab:events stream 31,458 entries
Freshness vps 14:28:29, piha 14:28:27, solaria 14:27:52, services 14:29:27 — all today
Consumers running on piha agent-system-redis, agent-system-runtime-materializer, agent-system-webui, agent-system-telegram-bot

Every one of those keys is written by stability-agent and by nothing else. The service-key set also shows the deployed build's staleness — entries like homelab:services:vps:dc8be279ea1a_node-agent carry the docker stale-state hash prefix that container_service_name() was written to strip.


5. COST OF EACH OPTION

(a) Retire stability-agent entirely

Monitoring capability lost:

  • Nothing that currently reaches an incident, an action, or an operator alert. All six filesystem event types are already inert (§2).
  • MQTT TCP reachability probe — the only fleet component that probes a broker port. Currently unconfigured on every online node and untriggerable, so the loss is prospective, not actual. Relevant to chelsty (offline-first, Mosquitto is load-bearing) if that site is revived.
  • Tailscale presence and Zigbee2MQTT presence checks — inert today, but they are the only code that looks at either.
  • Container visibility without a restart-policy filter (one-shot/manual containers).
  • Disk monitoring of / as distinct from /opt/homelab — on any node where those are separate mounts, node-agent does not watch /.

What breaks — the real cost:

  • The agent-system WebUI goes blank. homelab:nodes:*, homelab:services:* and the homelab:events stream lose their only writer. runtime-materializer continues polling and materializes empty sets. This is a live, working, user-facing surface — 130 service keys refreshed a minute before this recon was written.
  • Any Telegram-bot or WebUI view built on that materialized world dir degrades with it (not separately verified in this recon).
  • The fleet loses its only :ro-sock container; every remaining sock holder is rw (recon A1).
  • hosts/vps/runtime/stability-agent/ was allotted 96 MiB of the 3,200 MiB VPS budget — reclaimed.

Deploy/config cleanup entailed:

  • Delete services/stability-agent/ (8 files), scripts/deploy/deploy-stability-agent.sh, 4 × hosts/*/runtime/stability-agent/docker-compose.override.yml.
  • Remove entries from hosts/solaria/services.yaml and hosts/vps/services.yaml.
  • docker compose down on vps, piha, solaria (chelsty-infra when reachable).
  • Purge or archive /opt/homelab/events/2026-*/ on all four nodes (~5 MB on piha, mostly May).
  • Update CLAUDE.md (agent-system architecture §1, event-path claim at line 100), docs/chelsty-stability-agent.md, recon A1/A2/B7.
  • Not covered by the cleanup: the Redis publisher must be rehomed first or the UI loss is permanent.

(b) Merge stability-agent's unique checks into node-agent

Which checks would actually move — four, and only one is non-trivial:

Check Surface area
Redis publishing (the load-bearing one) ~60 lines: a RedisClient (or the redis package, already a dependency of agent-system), plus the homelab:nodes:* / homelab:services:* HSET loop and the homelab:events XADD. node-agent already computes every input — disk/mem/cpu in run_once(), container state in check_containers(). It needs a publish hook, REDIS_HOST/PORT/ENABLED env vars in the compose and in 47 host overrides, and a decision on whether an offline-first node may block on a piha-hosted Redis (must be fail-open; stability-agent's version already is). This alone preserves the only functional output.
MQTT TCP probe ~15 lines: socket.create_connection((MQTT_HOST, MQTT_PORT), timeout=5) → emit mqtt_unreachable. Plus MQTT_HOST/MQTT_PORT env. Note: to be useful it also needs an observer branch and a supervisor trigger — the type was deliberately removed from CONTAINER_RESTART_TRIGGERS on 2026-07-28 because no incident path existed. So the honest surface is 15 lines in node-agent plus an observer process_event branch plus a supervisor mapping.
Tailscale presence ~10 lines, two os.path.exists calls. Needs a new event type and observer/supervisor wiring to be worth anything; as a status-file value it is pointless.
Zigbee2MQTT presence ~5 lines, but it is a site-specific check on a node that is offline. Arguably belongs in a chelsty-specific probe, not the fleet sensor.
/ disk check alongside /opt/homelab ~10 lines: a second shutil.disk_usage("/") and a mount label in the payload.
Drop the restart-policy filter Not a merge — a behavior change to check_containers() with a known noise cost; the filter exists on purpose.

Not moving: docker_api_error, agent_error, redis_publish_error — node-agent already logs the equivalents; promoting them to events adds volume with no consumer unless observer branches are added too.

Rough total: ~100120 lines in node_agent.py, env additions across services/node-agent/docker-compose.yml and 47 host overrides, plus tests (Definition of Done: docker build + smoke run + pytest — node-agent has 5 test files to extend). The merged agent keeps its rw sock, so the fleet's last :ro sock holder disappears either way. Requires a coordinated cutover: node-agent must be publishing to Redis before stability-agent stops, or the WebUI gaps.

(c) Keep both

Duplicated cost, as measured:

Dimension Cost
Deploy paths Two mechanisms for the same job. stability-agent's is the shadow path — hardcoded Tailscale IPs, explicitly skipped by deploy-node.sh:93, a node-name whitelist that doesn't match the repo's own host dirs (chelsty vs chelsty-infra), and --force-recreate on every run. Its hosts/*/services.yaml entries are decorative: the pipeline skips them, so desired state claims coverage the pipeline does not provide.
Undeployed fixes The gap is not theoretical. f92e161 (2026-07-29) is in master; the vps image is from 2026-05-17 and has been running 7 weeks. Nobody noticed, because fixing the service tag changes nothing while breaks #1#4 stand. A second deploy path is a second thing to forget.
Sock exposure Two containers per node hold /var/run/docker.sock. stability-agent's is :ro and its output is dead — the fleet pays the attack surface of a sock mount on every node for a channel with zero consumers.
Event volume ~7,300 dead events written to disk across three nodes, ~5 MB, in 1320 date directories per node that no retention sweep touches (node-agent's _cleanup_control_plane_fs only prunes service_healthy/node_health under events/<node>/, never the date subtree). It grows without bound and nothing reads it.
Confusion — the compounding cost CLAUDE.md documents the dead JSONL path as the canonical event format (line 100). scripts/lib/events.sh — the sanctioned shell emitter — writes to it. Recon D15 diagnosed one symptom (service=None) and a fix was written and merged for it, when four upstream breaks made that fix inert. The 2026-07-06 prometheus recon reached the correct conclusion and it was not carried forward. Two sensors emitting the same event type name into two incompatible stores means every future "why didn't this alert fire" investigation starts by disambiguating which agent, which store, which format.
What it costs to keep it honest Nothing about (c) is stable as-is: keeping both means either accepting a permanently dead channel, or fixing breaks #1#4 — which is a rewrite of stability-agent's emitter into node-agent's format, i.e. option (b) performed inside a second codebase.

DECISION NEEDED

The choice is whether the Redis→agent-system-WebUI publisher is worth a second per-node daemon with a docker.sock mount and a parallel deploy path, or whether it moves into node-agent. Everything else about stability-agent is already dead: all six of its filesystem event types are unreadable by the observer and supervisor for four independent structural reasons beyond the one recon D15 named, its events have never opened an incident on any node in the service's lifetime, and the two nodes that are not the VPS have never shipped a single one of them anywhere. Against that, it is the only writer of the 4 node keys, 130 service keys and 31,458-entry stream that the agent-system UI renders — refreshed one minute before this recon was written — and node-agent has no Redis code at all. So (a) retire is the cheapest cleanup but blanks a live UI unless the publisher is rehomed first; (b) merge is ~100120 lines plus env plumbing across 47 host overrides plus a coordinated cutover, and ends with one sensor, one deploy path, one sock, one event format; (c) keep both preserves the UI at zero engineering cost today, but the duplicated cost is not static — it is a permanently dead event channel that the repo's own CLAUDE.md documents as canonical, a shadow deploy path that has already silently withheld a merged fix for 7 weeks, and a second sock mount on every node buying nothing. The key trade-off is therefore not "monitoring coverage" — node-agent strictly dominates on every check that reaches a consumer — but whether to pay a one-time migration of the Redis publisher now, or keep paying the ambiguity of two sensors, two event stores and two deploy paths indefinitely.