f6a016f551
node-agent calls docker containers.prune() with no filters every CHECK_INTERVAL (60 s on VPS). The Docker API removes EVERY non-running container regardless of RestartPolicy or compose labels — this destroyed ollama@solaria on 2026-07-30 (19 s after an operator `docker stop`). On VPS the blast radius is worse: humanai-mailer and humanai-landing have no compose definition in this repo (recreated by hand from `docker inspect`), so a pruned container there is an irreversible loss of the only config source. self.node_type is read only by run_safe_cleanup() (node_agent.py:648,654) and two log lines (250, 1103). Verified additionally for VPS: the control-plane filesystem rotation and health probe in run_once() are gated on `node_name == VPS_NODE_NAME`, not node_type — so they keep running. Monitoring, event shipping and action dispatch are likewise unaffected. Temporary — remove once R1 (explicit-enumeration prune) is deployed. Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 M1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
29 lines
1.6 KiB
YAML
29 lines
1.6 KiB
YAML
services:
|
|
node-agent:
|
|
environment:
|
|
- NODE_NAME=vps
|
|
- CHECK_INTERVAL=60
|
|
# TEMPORARY mitigation (M1) for the unfiltered-prune incident
|
|
# (docs/incidents/2026-07-30-ollama-solaria-vanish.md §7). node-agent runs
|
|
# `docker container prune()` with NO filters every CHECK_INTERVAL, and the
|
|
# Docker API removes EVERY non-running container regardless of restart
|
|
# policy or compose labels — this already destroyed ollama@solaria. On VPS
|
|
# the loss is worse: humanai-mailer and humanai-landing have no compose
|
|
# definition in this repo, so a pruned container cannot be recreated.
|
|
# node_type is read ONLY by run_safe_cleanup() (plus two log lines), so
|
|
# lte_node disables cleanup and nothing else — monitoring, event shipping
|
|
# and action dispatch keep working.
|
|
# REMOVE once R1 (explicit-enumeration prune) is deployed to VPS.
|
|
- NODE_TYPE=lte_node
|
|
# host network mode: node-agent on VPS shares the host's network namespace
|
|
# so that localhost:18180 resolves to the control-plane's exposed port.
|
|
# Without this, localhost inside the container is the container's own loopback
|
|
# and the _check_control_plane_health() probe would always fail.
|
|
network_mode: host
|
|
# HARD memory ceiling: node-agent mounts /opt/homelab/events/ (page cache)
|
|
# and may accumulate Python RSS over hours; 640m cap ensures it is killed and
|
|
# auto-restarted by Docker before consuming host memory. oom_score_adj -900
|
|
# prevents the host kernel OOM-killer from picking it as a global victim.
|
|
mem_limit: 640m
|
|
oom_score_adj: -900
|