Decide-support recon for the two per-node sensors. Finding-first, sections 1-5
per brief, ends with DECISION NEEDED (no recommendation).
Headline: stability-agent's filesystem event channel is structurally unreadable
by the control plane, not merely mis-tagged. Recon D15 named one break; there
are five, stacked, any one fatal:
1. wrong subtree — events/<date>/<node>/events.jsonl vs events/<node>/
2. wrong extension — observer.py:912 / supervisor.py:605 glob **/*.json
3. wrong format — observer json.load()s the whole file; JSONL is not one
4. never shipped — node_agent.py:788 rsyncs only events/<node>/
5. details vs payload key, plus service=None (D15)
D15 is stale on master but still true in production: fix f92e161 (2026-07-29) is
merged, the running solaria image was built 2026-05-17, and 100% of the 5,056
deployed events sampled carry service=None. Moot regardless — breaks 1-4 mean
the observer never opens the file. All six event types are inert, not just
containers_not_running.
Counterweight: stability-agent is the sole writer of the Redis keys the
agent-system WebUI renders (4 node, 130 service, 31,458-entry stream, refreshed
during the recon). node-agent has no Redis code. Retiring it blanks that UI.
Runtime evidence over the retained window:
node-agent 50,773 events -> 1 incident
stability-agent 7,317 events -> 0 incidents (structurally zero since
first deploy 2026-05-17)
Also found: deploy-local.sh validates 'chelsty' but the repo has
hosts/chelsty-infra/, so the only override carrying MQTT_HOST has never been
applied (stale homelab:nodes:chelsty key from 2026-06-02); scripts/lib/events.sh
writes to the same dead path; CLAUDE.md:100 documents the dead format as
canonical; recon B7 is stale — stability-agent IS in hosts/{solaria,vps}
services.yaml, but deploy-node.sh:93 skips it.
Read-only recon: no runtime mutations, no config changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RECON appendix summarizes the whole etap-0 cleanup (topology dormant status,
hosts reconciliation, dead-code removal, legacy decision), carries the
runbook line for the stale chelsty pending action
(rm /opt/homelab/actions/pending/redeploy-chelsty-infra-ha-diag-agent.json —
verified the only chelsty-targeted pending file 2026-07-30), and lists
discrepancies found but out of scope: lustro watchtower, stale solaria/chelsty
world-state keys, piha naming-drift duplicate keys, gokapi still down with a
stuck pending redeploy, outline/joplin/umami still lacking services/ dirs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bus codex/* dead since 2026-06-09 (recon C9). Records what survives into
subsystem B (patterns only: task bus, worker roles, enforced command
allowlists, preview mode), that task/ai-cluster-solaria stays unmerged
as documentation, and that the vps containers get stopped in a separate
supervised session.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One page of binding architecture decisions per the 2026-07-27 recon:
two subsystems (A keep-it-running / B do-the-work), node roles incl.
solaria's by-design duty cycle and dormant chelsty site, ai-cluster
declared legacy (bus dead since 2026-06-09), approvals stay HITL with
Telegram as the etap-2 channel.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
