2026-05-17 15:54:19 +02:00
|
|
|
host: vps
|
|
|
|
|
|
|
|
|
|
services:
|
feat(node-agent): implement health monitor and safe cleanup policy
scripts/monitor/health-monitor.sh (new):
- Standalone bash health monitor: disk/RAM/CPU checks + docker container health
- Per-node-type cleanup policy enforced:
lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops
sd_card (piha, saturn): dangling images + containers, rate-limited once/24h
ai_node (solaria): dangling + containers + build cache, NEVER -a
standard (vps): dangling + containers + build cache + CP filesystem rotation
- VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d,
events >3d AND past observer checkpoint
- Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu,
containers_not_running, healthcheck_failed)
services/node-agent/ (new):
- Python daemon (node_agent.py): same policy as bash script, Docker SDK
for container checks and cleanup, /proc for system metrics
- Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var)
- Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0
- docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only
observer.py:
- Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure
- Handle disk_pressure: record severity on node, clear when healthy
- Handle high_memory / high_cpu: record pressure level for correlation
supervisor.py:
- Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha}
- reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure
- _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node},
checks all active states, risk=guarded (operator approval required)
executor.py:
- Handle disk_cleanup action type via _execute_disk_cleanup()
- Commands come from action payload; safety gate rejects any command touching
/opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf /
hosts/*/services.yaml:
- Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra
- Add node-agent to chelsty-ha (previously missing)
- Add cleanup policy notes to LTE node comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
|
|
|
node-agent:
|
2026-05-17 15:54:19 +02:00
|
|
|
role: node-stability-monitor
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: local-only
|
|
|
|
|
offline_required: true
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
runtime:
|
feat(node-agent): implement health monitor and safe cleanup policy
scripts/monitor/health-monitor.sh (new):
- Standalone bash health monitor: disk/RAM/CPU checks + docker container health
- Per-node-type cleanup policy enforced:
lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops
sd_card (piha, saturn): dangling images + containers, rate-limited once/24h
ai_node (solaria): dangling + containers + build cache, NEVER -a
standard (vps): dangling + containers + build cache + CP filesystem rotation
- VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d,
events >3d AND past observer checkpoint
- Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu,
containers_not_running, healthcheck_failed)
services/node-agent/ (new):
- Python daemon (node_agent.py): same policy as bash script, Docker SDK
for container checks and cleanup, /proc for system metrics
- Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var)
- Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0
- docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only
observer.py:
- Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure
- Handle disk_pressure: record severity on node, clear when healthy
- Handle high_memory / high_cpu: record pressure level for correlation
supervisor.py:
- Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha}
- reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure
- _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node},
checks all active states, risk=guarded (operator approval required)
executor.py:
- Handle disk_cleanup action type via _execute_disk_cleanup()
- Commands come from action payload; safety gate rejects any command touching
/opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf /
hosts/*/services.yaml:
- Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra
- Add node-agent to chelsty-ha (previously missing)
- Add cleanup policy notes to LTE node comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
|
|
|
config_path: /opt/homelab/config/node-agent
|
2026-05-17 15:54:19 +02:00
|
|
|
data_path: /opt/homelab/state
|
|
|
|
|
logs_path: /opt/homelab/events
|
2026-05-20 12:06:23 +02:00
|
|
|
|
|
|
|
|
control-plane:
|
|
|
|
|
role: management-and-orchestration
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: tailscale-internal
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local:
|
feat(node-agent): implement health monitor and safe cleanup policy
scripts/monitor/health-monitor.sh (new):
- Standalone bash health monitor: disk/RAM/CPU checks + docker container health
- Per-node-type cleanup policy enforced:
lte_node (chelsty-infra, chelsty-ha): NO cleanup, no docker ops
sd_card (piha, saturn): dangling images + containers, rate-limited once/24h
ai_node (solaria): dangling + containers + build cache, NEVER -a
standard (vps): dangling + containers + build cache + CP filesystem rotation
- VPS filesystem rotation: completed/failed actions >7d, deploy logs >30d,
events >3d AND past observer checkpoint
- Emits structured JSON events (node_health, disk_pressure, high_memory, high_cpu,
containers_not_running, healthcheck_failed)
services/node-agent/ (new):
- Python daemon (node_agent.py): same policy as bash script, Docker SDK
for container checks and cleanup, /proc for system metrics
- Optional event shipping to VPS via rsync+SSH (VPS_EVENTS_HOST env var)
- Dockerfile: python:3.11-slim + openssh-client + rsync + docker>=6.0
- docker-compose.yml: mounts docker socket, /opt/homelab, repo read-only
observer.py:
- Handle node_health: update node status + disk/mem/cpu metrics, clear disk_pressure
- Handle disk_pressure: record severity on node, clear when healthy
- Handle high_memory / high_cpu: record pressure level for correlation
supervisor.py:
- Add NO_DISK_CLEANUP_NODES = {chelsty-infra, chelsty-ha}
- reconcile() step 3: generate disk_cleanup actions for nodes with high disk pressure
- _generate_disk_cleanup_recommendation(): stable ID disk-cleanup-{node},
checks all active states, risk=guarded (operator approval required)
executor.py:
- Handle disk_cleanup action type via _execute_disk_cleanup()
- Commands come from action payload; safety gate rejects any command touching
/opt/homelab/data/, /opt/homelab/config/, /opt/homelab/state/, or rm -rf /
hosts/*/services.yaml:
- Rename stability-agent -> node-agent on piha, vps, solaria, chelsty-infra
- Add node-agent to chelsty-ha (previously missing)
- Add cleanup policy notes to LTE node comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:15:06 +02:00
|
|
|
- node-agent
|
2026-05-20 12:06:23 +02:00
|
|
|
external:
|
2026-05-21 11:40:55 +02:00
|
|
|
- piha:redis
|
2026-05-20 12:06:23 +02:00
|
|
|
ports:
|
|
|
|
|
- name: http
|
|
|
|
|
container_port: 18180
|
|
|
|
|
protocol: tcp
|
|
|
|
|
runtime:
|
|
|
|
|
config_path: /opt/homelab/config/control-plane
|
|
|
|
|
data_path: /opt/homelab/data/control-plane
|
|
|
|
|
logs_path: /opt/homelab/logs/control-plane
|
2026-05-21 11:40:55 +02:00
|
|
|
|
|
|
|
|
node_exporter:
|
|
|
|
|
role: metrics-exporter
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: local-only
|
|
|
|
|
offline_required: true
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
2026-06-24 17:13:55 +02:00
|
|
|
|
|
|
|
|
fleet-prometheus:
|
|
|
|
|
role: fleet-liveness-source
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: tailscale-internal
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local:
|
|
|
|
|
- node_exporter
|
|
|
|
|
external: []
|
|
|
|
|
ports:
|
|
|
|
|
- name: http
|
|
|
|
|
container_port: 9090
|
|
|
|
|
protocol: tcp
|
|
|
|
|
runtime:
|
|
|
|
|
config_path: /opt/homelab/config/fleet-prometheus
|
|
|
|
|
data_path: /opt/homelab/data/fleet-prometheus
|
|
|
|
|
logs_path: /opt/homelab/logs/fleet-prometheus
|
2026-07-09 17:29:28 +02:00
|
|
|
|
feat(hosts): reconcile desired state with reality on vps, piha, saturn, lustro
Etap 0 truth cleanup (recon B5/B6/F20), all facts re-verified 2026-07-30 via
read-only ssh (docker ps, world/services.json, events store):
- vps: add stability-agent, npm, outline, joplin-server, umami (all running
and healthy in world state). joplin keyed joplin-server to match the
world-state key — an aggregate 'joplin' entry would drift as
missing_service forever. humanai-mailer/humanai-landing documented as
unmanaged: true + monitor: false (hand-run, no compose labels — supervisor
must not queue dead-end redeploys for them). ai-cluster deliberately gets
NO entry: legacy, retiring — comment points at ai-cluster-LEGACY.md.
- piha: comment block enumerating the ~28 known unmanaged containers plus the
host systemd mosquitto; bringing them in is a later stage (recon OQ5).
- saturn: create services.yaml with explicit empty list (dev workstation /
orchestrator, nothing monitored — closes recon F20.4).
- lustro: add node-exporter (hyphenated key = actual container name) and
piper-tts (runs locally on the Pi, no services/ dir); watchtower noted as
deliberately unmanaged.
All YAMLs parse (yaml.safe_load); control-plane suite 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-30 15:36:23 +02:00
|
|
|
stability-agent:
|
|
|
|
|
role: node-watchdog # read-only docker.sock watchdog, emits filesystem events
|
|
|
|
|
# Deploys via its own deploy-local.sh, outside the declarative pipeline
|
|
|
|
|
# (recon B7) — entry added 2026-07 truth cleanup so desired state matches
|
|
|
|
|
# what actually runs here.
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: local-only
|
|
|
|
|
offline_required: true
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
runtime:
|
|
|
|
|
data_path: /opt/homelab/state
|
|
|
|
|
logs_path: /opt/homelab/events
|
|
|
|
|
|
|
|
|
|
npm:
|
|
|
|
|
role: reverse-proxy # public ingress on the VPS (separate from npm-piha)
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: public
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
ports:
|
|
|
|
|
- name: http
|
|
|
|
|
container_port: 80
|
|
|
|
|
protocol: tcp
|
|
|
|
|
- name: https
|
|
|
|
|
container_port: 443
|
|
|
|
|
protocol: tcp
|
|
|
|
|
runtime:
|
|
|
|
|
# Data stays at the pre-GitOps bind mounts (CLAUDE.md data migration rule):
|
|
|
|
|
# /home/dockeruser/docker/npm/{data,letsencrypt}
|
|
|
|
|
config_path: services/npm
|
|
|
|
|
|
|
|
|
|
outline:
|
|
|
|
|
# Runs on vps but has NO services/outline dir on master (recon F20.2) — the
|
|
|
|
|
# containers predate GitOps management. Companion containers
|
|
|
|
|
# outline-postgres-1 / outline-redis-1 are visible in world state but are
|
|
|
|
|
# not separate desired-state entries.
|
|
|
|
|
role: knowledge-base
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: public
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
|
|
|
|
|
joplin-server:
|
|
|
|
|
# Keyed joplin-server (not "joplin") to match the compose service name the
|
|
|
|
|
# world state tracks — an aggregate "joplin" entry would sit in
|
|
|
|
|
# missing_service drift forever. No services/joplin dir on master (recon
|
|
|
|
|
# F20.2). Companion joplin-db is tracked in world state, not desired here.
|
|
|
|
|
role: notes-sync
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: public
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
|
|
|
|
|
umami:
|
|
|
|
|
# Web analytics; compose-labelled but no services/umami dir in the repo —
|
|
|
|
|
# pre-GitOps deployment (recon B5). Companion umami-db not desired here.
|
|
|
|
|
role: web-analytics
|
|
|
|
|
deployment_model: docker-compose
|
|
|
|
|
exposure: public
|
|
|
|
|
offline_required: false
|
|
|
|
|
depends_on:
|
|
|
|
|
local: []
|
|
|
|
|
external: []
|
|
|
|
|
|
|
|
|
|
# humanai-mailer / humanai-landing: hand-run containers with NO compose
|
|
|
|
|
# labels (recon B5/B7). Documented here as unmanaged so the truth map is
|
|
|
|
|
# complete. monitor: false keeps the supervisor from generating (dead-end
|
|
|
|
|
# redeploy) actions for containers the pipeline cannot manage; `unmanaged`
|
|
|
|
|
# itself is documentation only — no code reads it.
|
|
|
|
|
humanai-mailer:
|
|
|
|
|
unmanaged: true
|
|
|
|
|
monitor: false
|
|
|
|
|
role: mailer
|
|
|
|
|
humanai-landing:
|
|
|
|
|
unmanaged: true
|
|
|
|
|
monitor: false
|
|
|
|
|
role: landing-page
|
|
|
|
|
|
|
|
|
|
# ai-cluster (openclaw, codex-worker, planner-worker, service-ops-worker,
|
|
|
|
|
# redis, mosquitto): legacy stack, runs UNMANAGED on vps and is scheduled
|
|
|
|
|
# for retirement — its codex/* bus has been idle since 2026-06-09.
|
|
|
|
|
# Deliberately NO entry here: legacy is not pulled into desired state.
|
fix(kb): przepiecie wszystkich odwolan wewnetrznych po migracji
126 plikow (md, yaml, sh, py) odwolywalo sie do sciezek sprzed migracji.
15 markdown-linkow [..](..) -> policzona sciezka WZGLEDNA wobec pliku
odsylajacego (wczesniej czesc z nich byla repo-root-relative i nie
rozwiazywala sie z katalogu, w ktorym lezala)
200 odwolan tekstowych (backticki, proza, yaml, importy w kodzie)
-> nowa sciezka repo-root-relative, zgodnie z konwencja repo
5 linkow rodzenstwa (gole nazwy plikow, np. "](DEPLOY.md)") — dzialaly
tylko w starym katalogu; przeliczone recznie
Objete m.in.: CLAUDE.md (scripts/onboard/README.md -> kb/runbooks/
node-onboarding-tool.md, docs/backlog.md -> kb/phases/backlog.md),
README.md, .claude/skills/, 20 session logow, kod jobow.
Ostatnie 5 odwolan pochodzi z tresci wciagnietej rebasem z origin/master
(session log 2026-07-31, override node-agenta na SOLARII, dwie pozycje
backlogu) — wskazywaly na docs/incidents/, docs/kb/modules/ i
services/narty27/README.md sprzed migracji.
Dodany wzajemny link miedzy kb/services/control-plane.md (stub kodu)
a kb/subsystems/control-plane.md (opis, deprecated) — dwa dokumenty o tym
samym systemie, latwe do pomylenia.
Weryfikacja na 790 plikach: 0 odwolan do starych sciezek,
0 martwych linkow markdown. Lint OKF: 190/190 plikow ZGODNE.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 15:12:24 +02:00
|
|
|
# See kb/decisions/ai-cluster-legacy.md and
|
|
|
|
|
# kb/subsystems/recon-multiagent.md (C9).
|