homelab-codex-ws/services/node-agent
oskar 0526af1299 fix(node-agent): R1/R2/R3 — stop deleting deliberately stopped containers
R1 (critical): _prune_stopped_containers no longer calls containers.prune().
The unfiltered API call removes EVERY non-running container, ignoring
RestartPolicy and compose labels — that is what deleted ollama@solaria 19 s
after an operator `docker stop`. No filter argument can fix it (`until` filters
on creation time, not stop time). Replaced with explicit enumeration over
containers.list(all=True, filters={"status": "exited"}), skipping anything with
restart policy unless-stopped/always/on-failure or a com.docker.compose.project
label. A stopped managed service is recorded operator intent and now belongs to
the module's NEVER TOUCHED list; only one-off leftovers are removed. Dangling
image and build cache prune are unchanged.

R2 (high): _sd_card_rate_ok → _cleanup_rate_ok, applied to every cleanup-
eligible node type. ai_node and standard pruned every 60 s (1440/day, "0 MB
reclaimed" in practically every cycle) with no rate limit at all; they now share
sd_card's CLEANUP_INTERVAL_SECS (24 h) and mark the cleanup timestamp.

R3 (medium): removals are named in the log — WARNING with the container names
when the list is non-empty, INFO otherwise. The old line reported megabytes
only, which is why ollama's deletion looked identical to every no-op cycle.

Tests: services/node-agent/tests/test_safe_cleanup.py (22 cases) — the ollama
regression (exited + unless-stopped survives, prune() never called), all three
restart policies, compose-label protection, disposable-leftover removal, sweep
continues past a failed remove, WARNING-level naming, and the rate limit for
ai_node/standard/sd_card plus lte_node still doing nothing.

Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 R1-R3.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 14:55:26 +02:00
..
src fix(node-agent): R1/R2/R3 — stop deleting deliberately stopped containers 2026-08-04 14:55:26 +02:00
tests fix(node-agent): R1/R2/R3 — stop deleting deliberately stopped containers 2026-08-04 14:55:26 +02:00
docker-compose.yml fix(node-agent): run as uid 1000 with docker group access 2026-06-03 18:20:31 +02:00
Dockerfile fix(node-agent): run as uid 1000 with docker group access 2026-06-03 18:20:31 +02:00