node-agent calls docker containers.prune() with no filters every
CHECK_INTERVAL (60 s on VPS). The Docker API removes EVERY non-running
container regardless of RestartPolicy or compose labels — this destroyed
ollama@solaria on 2026-07-30 (19 s after an operator `docker stop`).
On VPS the blast radius is worse: humanai-mailer and humanai-landing have
no compose definition in this repo (recreated by hand from `docker inspect`),
so a pruned container there is an irreversible loss of the only config source.
self.node_type is read only by run_safe_cleanup() (node_agent.py:648,654) and
two log lines (250, 1103). Verified additionally for VPS: the control-plane
filesystem rotation and health probe in run_once() are gated on
`node_name == VPS_NODE_NAME`, not node_type — so they keep running. Monitoring,
event shipping and action dispatch are likewise unaffected.
Temporary — remove once R1 (explicit-enumeration prune) is deployed.
Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 M1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Etap 0 truth cleanup (recon B5/B6/F20), all facts re-verified 2026-07-30 via
read-only ssh (docker ps, world/services.json, events store):
- vps: add stability-agent, npm, outline, joplin-server, umami (all running
and healthy in world state). joplin keyed joplin-server to match the
world-state key — an aggregate 'joplin' entry would drift as
missing_service forever. humanai-mailer/humanai-landing documented as
unmanaged: true + monitor: false (hand-run, no compose labels — supervisor
must not queue dead-end redeploys for them). ai-cluster deliberately gets
NO entry: legacy, retiring — comment points at ai-cluster-LEGACY.md.
- piha: comment block enumerating the ~28 known unmanaged containers plus the
host systemd mosquitto; bringing them in is a later stage (recon OQ5).
- saturn: create services.yaml with explicit empty list (dev workstation /
orchestrator, nothing monitored — closes recon F20.4).
- lustro: add node-exporter (hyphenated key = actual container name) and
piper-tts (runs locally on the Pi, no services/ dir); watchtower noted as
deliberately unmanaged.
All YAMLs parse (yaml.safe_load); control-plane suite 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nextcloud zostaje prywatny (mesh/kapala.org); Gokapi to osobny publiczny
serwis do wysylania linkow do plikow na zewnatrz (Firefox Send alt).
owner_node=vps, storage lokalny dysk (nie S3), E2E encryption ON, port
53842 bindowany tylko na TAILSCALE_BIND_IP (hairpin NAT przez npm@VPS,
nigdy 0.0.0.0). Cutover checklist w README (DNS, wildcard *.okit.pl na
npm@VPS, setup wizard) — nie zdeployowane w tym commicie.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Declares the real OS-level hostname for each node so deploy-node.sh
can resolve the correct hosts/ directory without assuming
OS-hostname == logical name. Only VPS differs: its OS hostname is
ubuntu-4gb-hel1-1 while the repo directory is hosts/vps/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- hosts/vps/services.yaml: fleet-prometheus entry (role: fleet-liveness-source,
exposure: tailscale-internal, port 9090, runtime paths, depends_on node_exporter)
- inventory/topology.yaml: add fleet-prometheus to the vps services list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Celowy stan host-level (poza GitOps) udokumentowany w manifescie, zeby przy
odtwarzaniu VPS nie zniknal cicho. Tylko opisowy blok host_state, bez zmian
struktury runtime/deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The _check_control_plane_health() method probes localhost:18180, which
is the control-plane's mapped port. Inside a bridged container, localhost
resolves to the container's own loopback — the probe always fails.
host network mode shares the VPS host's network namespace so that
localhost:18180 correctly reaches the control-plane.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- observer: store trigger_type on incidents for supervisor routing
- supervisor: route containers_not_running/mqtt_unreachable to container_restart instead of redeploy
- supervisor: fix node alias normalization via NODE_ALIAS_MAP
- supervisor: fix pending action dedup (scan by content not filename)
- executor: implement container_restart via SSH docker restart with retry
- control-plane override: configure NODE_ALIAS_MAP for production
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala