czujniki-2026-07-30 (node-agent vs stability-agent) lustro-shipping-2026-07-16 (event=dead prom=up, 1507 mismatchy) prometheus-cutover-2026-07-06 (recon starego toru livenesci) piha-slim-2026-07-02 (audyt odchudzania PIHA) vps-stacki-2026-07-27 (audyt niezarzadzanych stackow na VPS) ODSTEPSTWO OD RECONU — swiadome. Recon typowal te 5 plikow jako SPLIT (audit+decision / audit+incident / audit+phase). Rozstrzygniecie 2 wprowadza typ `audit` z polem as_of i mapuje kazdy z nich na JEDNA sciezke kb/audits/<obszar>-<data>.md. Audyt jest spojna migawka stanu z konkretna data — rozbicie go na "ustalenia" i "rekomendacje" rozerwaloby ten kontekst i wymagaloby redakcji tresci, czego etap 2 zabrania. Zostaja w calosci. Efekt: 29 SPLIT-ow z reconu realizowane jako 24 (10 service+runbook, 14 wielotypowych), 5 zamienionych na caloscowe dokumenty type: audit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30 KiB
| okf | type | visibility | status | updated | as_of | links |
|---|---|---|---|---|---|---|
| 0.1 | audit | private | active | 2026-07-30 | 2026-07-30 |
Recon — node-agent vs stability-agent (2026-07-30)
Read-only recon. Ground truth: docs/architecture/RECON-multiagent-2026-07-27.md
(A1, A2, B7, D15). Source read at master @ 473bf8e; this branch is cut from
master @ 0650eb8. The two intervening merges (0650eb8 ha-mcp, cb8a19d
kb-query tests) touch none of the audited paths — services/node-agent/,
services/stability-agent/, scripts/observer/, services/control-plane/ — so
every finding and line citation below holds at 0650eb8. Runtime evidence
collected 2026-07-30 ~14:30 UTC from vps, piha and solaria (solaria = the host
this recon ran on).
FINDING FIRST
stability-agent's filesystem event output is not merely mis-tagged — it is
structurally unreadable by the entire control plane, and always has been.
Recon D15 identified one break (service=None). There are five, stacked, and
any one of them alone is fatal:
| # | Break | Evidence |
|---|---|---|
| 1 | Wrong path subtree. stability-agent writes /opt/homelab/events/<YYYY-MM-DD>/<node>/events.jsonl. node-agent writes /opt/homelab/events/<node>/evt-*.json. These are sibling subtrees, not the same store. |
stability_agent.py:47-51 vs node_agent.py:261,287 |
| 2 | Wrong file extension. Observer and supervisor both discover events with glob("**/*.json"). events.jsonl does not match *.json. |
observer.py:912, supervisor.py:605 |
| 3 | Wrong file format. Even if matched, the observer does json.load(f) on the whole file — a multi-line JSONL append log is not a JSON document. |
observer.py:938 |
| 4 | Never shipped off-node. node-agent's rsync ships only EVENTS_DIR/<node>/. The date-partitioned subtree is outside that path, so piha's and solaria's stability-agent events never reach the VPS at all. |
node_agent.py:788 |
| 5 | Wrong payload key + service=None (recon D15). stability-agent writes details; the observer reads payload. And the deployed build still emits one aggregate event with no service key. |
stability_agent.py:42 vs observer.py:701 |
Break #4 is confirmed empirically: on VPS, every date-partitioned directory
contains only vps/ — there is no piha/ or solaria/ subdir in any of the
13 date dirs, despite both nodes having written thousands of local lines.
Break #5's repo-side fix exists but is not deployed: commit f92e161
("tag containers_not_running events with compose service") is dated 2026-07-29;
the stability-agent image running on solaria was built 2026-05-17. 100 % of the
5,056 deployed events sampled across three nodes carry service: None. So D15's
claim is still true on current runtime, while being stale on current master —
and it is moot either way, because breaks #1–#4 mean the observer never opens the
file.
Consequence: all six of stability-agent's filesystem event types are inert —
containers_not_running, disk_usage_high, mqtt_unreachable, docker_api_error,
agent_error, redis_publish_error. None reaches an incident. None reaches the
supervisor. This is not a per-type gap; it is the whole channel.
The counterweight: stability-agent has a second, entirely separate output
channel that node-agent does not have at all — Redis publishing to
agent-system-redis on piha — and that channel is live, fresh, and exclusive.
It is the sole populator of the agent-system WebUI's runtime view (4 node keys,
130 service keys, a 31,458-entry event stream, last write 14:29 today). Retiring
stability-agent blanks that UI.
So the two sensors do not overlap the way they appear to. They overlap on measurement and diverge completely on delivery: node-agent feeds the control plane, stability-agent feeds the agent-system UI, and stability-agent's half of the event store is a write-only archive nothing has ever read.
A prior recon reached the same conclusion about the event path on 2026-07-06
(docs/infra/prometheus-cutover-recon-2026-07-06.md:88-91); it has not been acted on.
1. FEATURE MATRIX
| Capability | node-agent | stability-agent |
|---|---|---|
| Source | services/node-agent/src/node_agent.py (1116 ln) |
services/stability-agent/src/stability_agent.py (386 ln) |
| Cycle | 60 s (CHECK_INTERVAL) |
60 s (STABILITY_CHECK_INTERVAL) |
| Docker access | docker-py SDK, from_env() |
hand-rolled UnixHTTPConnection → GET /containers/json?all=1 |
| Sock mode | rw | ro |
| Why that mode | Genuinely needs write: container.restart() (:984), images.prune, containers.prune, api.prune_builds (:619-641) |
Read-only by construction — issues only HTTP GETs; no write code path exists |
| Disk check | shutil.disk_usage(RUNTIME_PATH) → the /opt/homelab partition. Warn 75 %, crit 85 % (hardcoded) |
shutil.disk_usage("/") → root fs, not the runtime partition. Single threshold, DISK_THRESHOLD_PCT (default 90) |
| RAM check | ✅ /proc/meminfo, warn 85 % / crit 95 % |
❌ none |
| CPU check | ✅ two-sample /proc/stat delta, threshold 90 % |
❌ none |
| Container state | list(all=True); skips created; filters to managed only (restart_policy ∈ {unless-stopped, always, on-failure}); reads docker Health.Status and RestartCount |
all containers; skips created; no restart-policy filter (reports one-shot/manual containers too); no health-status read, no RestartCount |
| Crash-loop detection | ✅ restarting + RestartCount ≥ 3 → containers_not_running; below threshold → container_restarting |
❌ restarting is simply "not running" |
| Docker healthcheck status | ✅ running + unhealthy → healthcheck_failed |
❌ not read |
| MQTT probe | ❌ none | ✅ TCP connect to MQTT_HOST:MQTT_PORT, 5 s timeout |
| Tailscale check | ❌ none | ✅ socket / tailscale0 iface presence — status file only, no event |
| Zigbee2MQTT check | ❌ none | ✅ substring match over container list — status file only, no event |
| Docker API error | logged, no event | ✅ docker_api_error event |
| Self-error reporting | logged only | ✅ agent_error event |
| Control-plane HTTP probe | ✅ vps-only, localhost:18180/summary (:1016) |
❌ |
| Docker cleanup / prune | ✅ per-node-type policy (:646) |
❌ none — pure observer |
| Control-plane FS rotation | ✅ vps-only: old actions >7 d, deploy logs >30 d, checkpointed noise events >3 d (:673) |
❌ |
| Dispatch execution | ✅ pulls actions/dispatch/<node>/, executes container_restart (:859-991) |
❌ no action code path at all |
| Event shipping to VPS | ✅ rsync --remove-source-files (:777) |
❌ none — events stay on the node forever |
| Event file layout | events/<node>/evt-<node>-<ts>-<type>-<svc>.json, one file per event, key payload |
events/<date>/<node>/events.jsonl, append-only JSONL, key details |
| Read by observer? | ✅ | ❌ (breaks #1–#5) |
| Redis publishing | ❌ none | ✅ HSET homelab:nodes:<node>, HSET homelab:services:<node>:<container>, XADD homelab:events |
| Heartbeat | state/node-agent.heartbeat (touch) |
state/stability-agent.heartbeat + state/stability-agent.json status summary |
| Deploy path | declarative — in hosts/*/services.yaml (7 hosts) → deploy.sh / deploy-node.sh |
shadow — scripts/deploy/deploy-stability-agent.sh <node> [--ssh] (hardcoded TS IPs) → services/stability-agent/deploy-local.sh → compose up -d --build --force-recreate. Explicitly skipped by deploy-node.sh:93 because it ships a deploy-local.sh |
| In desired state? | yes, all 7 hosts | yes on hosts/solaria + hosts/vps (added in the 2026-07 truth cleanup — recon B7 is stale here), but the entry is inert: the deploy loop skips it |
| Service contract files | ❌ no service.yaml, no README, no env.example, no healthcheck.sh — violates the CLAUDE.md service layout |
✅ all four present |
| Config surface | NODE_NAME, NODE_TYPE, VPS_EVENTS_HOST/USER/PATH, CHECK_INTERVAL, CRASH_LOOP_RESTART_THRESHOLD, RUNTIME_PATH, REPO_ROOT. Thresholds hardcoded |
NODE_NAME, STABILITY_CHECK_INTERVAL, DISK_THRESHOLD_PCT, MQTT_HOST/PORT, REDIS_HOST/PORT/ENABLED |
| Per-node overrides | hosts/<node>/runtime/node-agent/ |
hosts/<node>/runtime/stability-agent/ (4 files: chelsty-infra, piha, solaria, vps) |
| Per-node-type behavior | ✅ real: lte_node no cleanup at all · sd_card dangling images + stopped containers, max 1×/24 h, no builder prune · ai_node + build cache, never -a · standard + control-plane FS rotation |
❌ none. No node-type concept. Only env values differ |
| Memory limit (vps) | not set in hosts/vps/runtime/node-agent/ |
mem_limit: 96m, oom_score_adj: -900 |
Event types, with trigger condition
node-agent — 11 types:
| Event | Severity | Condition |
|---|---|---|
disk_pressure |
high / medium | /opt/homelab usage ≥ 85 % / ≥ 75 % |
high_memory |
high / medium | MemAvailable-derived usage ≥ 95 % / ≥ 85 % |
high_cpu |
medium | CPU ≥ 90 % |
containers_not_running |
high | managed container exited/dead, or restarting with RestartCount ≥ 3 |
container_restarting |
low | restarting with RestartCount < 3 (observational, deliberately unwired) |
healthcheck_failed |
high | running + docker health unhealthy |
container_state_unexpected |
medium | paused, or any unhandled docker state |
service_healthy |
info | running, emitted only on the ¬healthy→healthy transition |
service_unhealthy |
high | vps-only: control-plane HTTP non-200 or unreachable |
node_health |
info | every cycle — carries disk/mem/cpu |
action_result |
info / high | after executing (or refusing) a dispatched action |
stability-agent — 6 types:
| Event | Severity | Condition |
|---|---|---|
containers_not_running |
warning | any container with State ∉ {running, created} |
disk_usage_high |
warning | / usage > DISK_THRESHOLD_PCT (90) |
mqtt_unreachable |
error | TCP connect to MQTT_HOST:MQTT_PORT fails — only fires if MQTT_HOST is set |
docker_api_error |
warning | sock exists but API request returns None |
agent_error |
error | unhandled exception in the main loop |
redis_publish_error |
warning | Redis publish threw |
Config-surface defect found in passing
deploy-local.sh validates the node name against piha|chelsty|solaria|vps. The
repo has hosts/chelsty-infra/, not hosts/chelsty/. Deploying the chelsty
node therefore finds no override file and falls through to the compose default
NODE_NAME=chelsty. Live proof: Redis holds a homelab:nodes:chelsty key,
stale since 2026-06-02, alongside correctly-named vps/piha/solaria keys.
hosts/chelsty-infra/runtime/stability-agent/docker-compose.override.yml — the
only override carrying MQTT_HOST and a tuned DISK_THRESHOLD_PCT=85 — has
never been applied.
2. UNIQUE TO stability-agent
Nine items. One is functional.
| # | Capability | Functional end-to-end? | Code path / reason |
|---|---|---|---|
| 1 | Redis publishing → homelab:nodes:*, homelab:services:*, homelab:events stream |
✅ YES — and exclusive | stability_agent.py:330-372 → agent-system-redis on piha (100.108.208.3) → services/agent-system/runtime-materializer/materializer.py:122,138,155 reads all three → writes the agent-system WebUI's world dir. Verified live: node keys refreshed 2026-07-30T14:28, service keys 14:29. Not an incident path — it feeds a UI, never the supervisor. node-agent has no Redis code at all. |
| 2 | containers_not_running (its own) |
❌ dead | Breaks #1–#5 above. Additionally the deployed build emits the pre-fix aggregate form. Never opens an incident; observer.py:770 would handle it if the file were ever read. |
| 3 | disk_usage_high |
❌ dead ×2 | Unreadable (#1–#4), and the type appears nowhere in observer.process_event's if/elif chain — the observer only knows disk_pressure. Even hand-fed a well-formed file, it would bump last_seen and fall through. Not in any supervisor trigger set. |
| 4 | docker_api_error |
❌ dead ×2 | Same: unreadable, and no branch in process_event, no supervisor trigger. Despite being the 2nd-most-emitted type on VPS (496 events). |
| 5 | agent_error |
❌ dead ×2 | Same. Self-diagnostics that nothing collects. |
| 6 | redis_publish_error |
❌ dead ×2 | Same. |
| 7 | mqtt_unreachable (TCP probe) |
❌ dead ×3 | Unreadable; MQTT_HOST is unset on piha/solaria/vps so the probe never even runs there (only the never-applied chelsty-infra override sets it); and the type was removed from CONTAINER_RESTART_TRIGGERS on 2026-07-28 (supervisor.py:47-51) precisely because the observer never creates such an incident. |
| 8 | Tailscale presence check | ❌ inert | stability_agent.py:189 — result goes to state/stability-agent.json only. No event emitted. Grep confirms no consumer of that file anywhere in the repo outside its own docs. |
| 9 | Zigbee2MQTT presence/running check | ❌ inert | stability_agent.py:297-310 — same status-file-only fate. Also a naive "zigbee2mqtt" in name substring match. |
Two structural properties also unique to it, neither a "capability":
:rosock mount — a safety property. It is the only sock-holder on the fleet that is read-only by mount (recon A1). Preserving it means preserving a container whose outputs are dead.- Broader container coverage — no restart-policy filter, so it sees one-shot and manually-run containers that node-agent deliberately ignores. Inert, since the events go nowhere.
Bonus: scripts/lib/events.sh — the documented shell event emitter — writes to
the same dead date-partitioned JSONL path (scripts/lib/events.sh:18-20). Every
shell-emitted event in the system shares stability-agent's fate. CLAUDE.md
("Events are append-only JSON lines at /opt/homelab/events/YYYY-MM-DD/<node>/events.jsonl")
documents the dead format as the canonical one.
3. UNIQUE TO node-agent
| # | Capability | Notes |
|---|---|---|
| 1 | Dispatch execution (container_restart) |
The on-node hands of the HITL loop. Type whitelist of exactly one, self-restart guard, idempotency markers, node-scoping check (:921-991). No equivalent exists in stability-agent — and adding one would require flipping its sock to rw. |
| 2 | action_result reporting |
Closes the executor's loop (:993); without it actions hang in running/. |
| 3 | Event shipping to VPS | rsync --remove-source-files (:777). This is the mechanism by which any off-VPS node reaches the control plane. stability-agent has no shipping whatsoever. |
| 4 | An event format the control plane can read | One *.json file per event, evt-<node>-<ts>-<type>-<svc> naming that carries the checkpoint ordering key, payload field. Everything downstream is built around this shape. |
| 5 | RAM monitoring (high_memory) |
Genuinely absent from stability-agent. Matters most on the 4 GiB no-swap VPS. |
| 6 | CPU monitoring (high_cpu) |
Ditto. 121 events on vps, 129 on piha, 48 on lustro in the retained window. |
| 7 | Crash-loop detection | RestartCount ≥ 3 threshold distinguishes a real flap from deploy churn (:482-516). stability-agent cannot tell them apart. |
| 8 | Docker healthcheck-status reading → healthcheck_failed |
3,375 events on piha. The single largest actionable signal in the fleet after containers_not_running. |
| 9 | service_healthy positive confirmation |
Keeps services.json populated so the supervisor's drift loop doesn't classify every desired service as missing_service and flood the action queue (:528-549, observer.py:762). Load-bearing. |
| 10 | node_health heartbeat |
Drives observer node liveness (node_offline/node_stale/node_online). The single highest-volume event type fleet-wide. |
| 11 | Docker cleanup, per-node-type | The whole lte_node/sd_card/ai_node/standard policy (:646-667). SD-card write-cycle protection and the "never -a on ai_node" rule live only here. |
| 12 | Control-plane FS rotation (vps) | Old actions, deploy logs, checkpointed noise-event sweep (:673). The fix for the 358k-file backlog that paralyzed the supervisor. |
| 13 | Control-plane HTTP health probe (vps) | The only thing that maps the 4-container control-plane stack to the single control-plane key in services.yaml (:1016). |
| 14 | Declarative deploy path | In hosts/*/services.yaml, deployed by the normal pipeline. |
Node-agent's own gaps, for symmetry: no service.yaml/README/env.example/healthcheck.sh
(CLAUDE.md violation); no memory limit declared on VPS despite the VPS rule; rw sock
with no allowlist of restartable container names (recon A2); and on solaria the
docker half is broken by a group_add gid mismatch — though note that as of today
solaria's node-agent container reports Up 28 minutes (healthy), so this is worth
re-verifying separately before relying on either state.
4. RUNTIME EVIDENCE
Collected 2026-07-30 ~14:30. VPS event store = /opt/homelab/events/.
Container liveness
| Node | stability-agent | node-agent |
|---|---|---|
| vps | running, Up 7 weeks (healthy) | running, Up 13 days (healthy) |
| piha | running, Up 13 days (healthy) | running, Up 7 days (healthy) |
| solaria | running, Up ~1 h (healthy) | running, Up 28 min (healthy) |
Both are running everywhere. Neither is crashed. This is a delivery problem, not an availability problem.
Events reaching the VPS control plane (node-agent subtree, events/<node>/)
| Node | Files | Top types |
|---|---|---|
| piha | 33,058 | ha_entity_unavailable_long 22,897 · node_health 4,112 · healthcheck_failed 3,375 · containers_not_running 2,488 · high_cpu 129 · ha_* 47 · action_result 2 |
| lustro | 11,935 | service_healthy 8,893 · node_health 2,940 · high_cpu 48 · liveness 54 |
| vps | 4,520 | node_health 4,232 · disk_pressure 132 · high_cpu 121 · service_unhealthy 34 · containers_not_running 1 |
| solaria | 1,260 | node_health 1,211 · liveness 42 · service_healthy 5 · containers_not_running 2 |
| chelsty-infra | 0 | node offline |
| Total | 50,773 | all from node-agent / ha-diag-agent / observer |
Events from stability-agent (date-partitioned subtree)
| Node | Reaches VPS? | Local volume | Types | Last event |
|---|---|---|---|---|
| vps | same fs, but unreadable | 2,240 all-time (13 date dirs) | containers_not_running 1,744 · docker_api_error 496 |
2026-07-16 — nothing for 14 days |
| piha | ❌ never shipped | 2,528 in July alone | containers_not_running 2,527 · docker_api_error 1 |
2026-07-27 |
| solaria | ❌ never shipped | 2,549 all-time / 288 in July | containers_not_running 287 · docker_api_error 1 |
2026-07-23 |
service field across all 5,056 events sampled: None, 100 %. No exceptions on
any node.
Confirmation of break #4: the 13 date directories on VPS contain only a vps/
subdirectory — no piha/, no solaria/, ever. piha's and solaria's ~5,000 lines
have never left their nodes.
Does any stability-agent event produce an incident? — No. Zero. On any node.
world/incidents.json on VPS contains exactly one incident:
inc-1784736375-piha-homeassistant
node=piha service=homeassistant status=active
trigger_type=containers_not_running occurrence_count=1307
Traced to the node-agent subtree (events/piha/evt-*.json) — it has a
service field, which is the precondition observer.py:754 requires to key a
service and open an incident. stability-agent contributed nothing to it.
Quantified answer to the question asked: over the full retained window, node-agent's channel produced 50,773 events and 1 incident. stability-agent's channel produced 7,317 events and 0 incidents. Its incident yield is not low — it is structurally zero, and has been since the service was first deployed (2026-05-17).
The Redis channel — live and exclusive
| Metric | Value |
|---|---|
homelab:nodes:* keys |
4 (vps, piha, solaria + stale chelsty from 2026-06-02) |
homelab:services:* keys |
130 |
homelab:incidents:* keys |
1 |
homelab:events stream |
31,458 entries |
| Freshness | vps 14:28:29, piha 14:28:27, solaria 14:27:52, services 14:29:27 — all today |
| Consumers running on piha | agent-system-redis, agent-system-runtime-materializer, agent-system-webui, agent-system-telegram-bot |
Every one of those keys is written by stability-agent and by nothing else. The
service-key set also shows the deployed build's staleness — entries like
homelab:services:vps:dc8be279ea1a_node-agent carry the docker stale-state hash
prefix that container_service_name() was written to strip.
5. COST OF EACH OPTION
(a) Retire stability-agent entirely
Monitoring capability lost:
- Nothing that currently reaches an incident, an action, or an operator alert. All six filesystem event types are already inert (§2).
- MQTT TCP reachability probe — the only fleet component that probes a broker port. Currently unconfigured on every online node and untriggerable, so the loss is prospective, not actual. Relevant to chelsty (offline-first, Mosquitto is load-bearing) if that site is revived.
- Tailscale presence and Zigbee2MQTT presence checks — inert today, but they are the only code that looks at either.
- Container visibility without a restart-policy filter (one-shot/manual containers).
- Disk monitoring of
/as distinct from/opt/homelab— on any node where those are separate mounts, node-agent does not watch/.
What breaks — the real cost:
- The agent-system WebUI goes blank.
homelab:nodes:*,homelab:services:*and thehomelab:eventsstream lose their only writer.runtime-materializercontinues polling and materializes empty sets. This is a live, working, user-facing surface — 130 service keys refreshed a minute before this recon was written. - Any Telegram-bot or WebUI view built on that materialized world dir degrades with it (not separately verified in this recon).
- The fleet loses its only
:ro-sock container; every remaining sock holder is rw (recon A1). hosts/vps/runtime/stability-agent/was allotted 96 MiB of the 3,200 MiB VPS budget — reclaimed.
Deploy/config cleanup entailed:
- Delete
services/stability-agent/(8 files),scripts/deploy/deploy-stability-agent.sh, 4 ×hosts/*/runtime/stability-agent/docker-compose.override.yml. - Remove entries from
hosts/solaria/services.yamlandhosts/vps/services.yaml. docker compose downon vps, piha, solaria (chelsty-infra when reachable).- Purge or archive
/opt/homelab/events/2026-*/on all four nodes (~5 MB on piha, mostly May). - Update CLAUDE.md (agent-system architecture §1, event-path claim at line 100),
docs/chelsty-stability-agent.md, recon A1/A2/B7. - Not covered by the cleanup: the Redis publisher must be rehomed first or the UI loss is permanent.
(b) Merge stability-agent's unique checks into node-agent
Which checks would actually move — four, and only one is non-trivial:
| Check | Surface area |
|---|---|
| Redis publishing (the load-bearing one) | ~60 lines: a RedisClient (or the redis package, already a dependency of agent-system), plus the homelab:nodes:* / homelab:services:* HSET loop and the homelab:events XADD. node-agent already computes every input — disk/mem/cpu in run_once(), container state in check_containers(). It needs a publish hook, REDIS_HOST/PORT/ENABLED env vars in the compose and in 4–7 host overrides, and a decision on whether an offline-first node may block on a piha-hosted Redis (must be fail-open; stability-agent's version already is). This alone preserves the only functional output. |
| MQTT TCP probe | ~15 lines: socket.create_connection((MQTT_HOST, MQTT_PORT), timeout=5) → emit mqtt_unreachable. Plus MQTT_HOST/MQTT_PORT env. Note: to be useful it also needs an observer branch and a supervisor trigger — the type was deliberately removed from CONTAINER_RESTART_TRIGGERS on 2026-07-28 because no incident path existed. So the honest surface is 15 lines in node-agent plus an observer process_event branch plus a supervisor mapping. |
| Tailscale presence | ~10 lines, two os.path.exists calls. Needs a new event type and observer/supervisor wiring to be worth anything; as a status-file value it is pointless. |
| Zigbee2MQTT presence | ~5 lines, but it is a site-specific check on a node that is offline. Arguably belongs in a chelsty-specific probe, not the fleet sensor. |
/ disk check alongside /opt/homelab |
~10 lines: a second shutil.disk_usage("/") and a mount label in the payload. |
| Drop the restart-policy filter | Not a merge — a behavior change to check_containers() with a known noise cost; the filter exists on purpose. |
Not moving: docker_api_error, agent_error, redis_publish_error — node-agent
already logs the equivalents; promoting them to events adds volume with no consumer
unless observer branches are added too.
Rough total: ~100–120 lines in node_agent.py, env additions across
services/node-agent/docker-compose.yml and 4–7 host overrides, plus tests
(Definition of Done: docker build + smoke run + pytest — node-agent has 5 test files
to extend). The merged agent keeps its rw sock, so the fleet's last :ro sock holder
disappears either way. Requires a coordinated cutover: node-agent must be publishing
to Redis before stability-agent stops, or the WebUI gaps.
(c) Keep both
Duplicated cost, as measured:
| Dimension | Cost |
|---|---|
| Deploy paths | Two mechanisms for the same job. stability-agent's is the shadow path — hardcoded Tailscale IPs, explicitly skipped by deploy-node.sh:93, a node-name whitelist that doesn't match the repo's own host dirs (chelsty vs chelsty-infra), and --force-recreate on every run. Its hosts/*/services.yaml entries are decorative: the pipeline skips them, so desired state claims coverage the pipeline does not provide. |
| Undeployed fixes | The gap is not theoretical. f92e161 (2026-07-29) is in master; the vps image is from 2026-05-17 and has been running 7 weeks. Nobody noticed, because fixing the service tag changes nothing while breaks #1–#4 stand. A second deploy path is a second thing to forget. |
| Sock exposure | Two containers per node hold /var/run/docker.sock. stability-agent's is :ro and its output is dead — the fleet pays the attack surface of a sock mount on every node for a channel with zero consumers. |
| Event volume | ~7,300 dead events written to disk across three nodes, ~5 MB, in 13–20 date directories per node that no retention sweep touches (node-agent's _cleanup_control_plane_fs only prunes service_healthy/node_health under events/<node>/, never the date subtree). It grows without bound and nothing reads it. |
| Confusion — the compounding cost | CLAUDE.md documents the dead JSONL path as the canonical event format (line 100). scripts/lib/events.sh — the sanctioned shell emitter — writes to it. Recon D15 diagnosed one symptom (service=None) and a fix was written and merged for it, when four upstream breaks made that fix inert. The 2026-07-06 prometheus recon reached the correct conclusion and it was not carried forward. Two sensors emitting the same event type name into two incompatible stores means every future "why didn't this alert fire" investigation starts by disambiguating which agent, which store, which format. |
| What it costs to keep it honest | Nothing about (c) is stable as-is: keeping both means either accepting a permanently dead channel, or fixing breaks #1–#4 — which is a rewrite of stability-agent's emitter into node-agent's format, i.e. option (b) performed inside a second codebase. |
DECISION NEEDED
The choice is whether the Redis→agent-system-WebUI publisher is worth a second per-node daemon with a docker.sock mount and a parallel deploy path, or whether it moves into node-agent. Everything else about stability-agent is already dead: all six of its filesystem event types are unreadable by the observer and supervisor for four independent structural reasons beyond the one recon D15 named, its events have never opened an incident on any node in the service's lifetime, and the two nodes that are not the VPS have never shipped a single one of them anywhere. Against that, it is the only writer of the 4 node keys, 130 service keys and 31,458-entry stream that the agent-system UI renders — refreshed one minute before this recon was written — and node-agent has no Redis code at all. So (a) retire is the cheapest cleanup but blanks a live UI unless the publisher is rehomed first; (b) merge is ~100–120 lines plus env plumbing across 4–7 host overrides plus a coordinated cutover, and ends with one sensor, one deploy path, one sock, one event format; (c) keep both preserves the UI at zero engineering cost today, but the duplicated cost is not static — it is a permanently dead event channel that the repo's own CLAUDE.md documents as canonical, a shadow deploy path that has already silently withheld a merged fix for 7 weeks, and a second sock mount on every node buying nothing. The key trade-off is therefore not "monitoring coverage" — node-agent strictly dominates on every check that reaches a consumer — but whether to pay a one-time migration of the Redis publisher now, or keep paying the ambiguity of two sensors, two event stores and two deploy paths indefinitely.