Commit graph

13 commits

Author SHA1 Message Date
oskar 1bab3219d6 revert(m1): zdjecie NODE_TYPE=lte_node na SOLARII i VPS po wdrozeniu R1
Warunek zdjecia M1 brzmial "R1 (prune filtrowany po restart policy / labelu
compose) wdrozony na tym nodzie". Zweryfikowane bezposrednio w kodzie dzialajacych
kontenerow, nie po datach deployu:

  SOLARIA  md5(/app/src/node_agent.py) = c9ac64e10b42b3e0ed9e4c168579bfaa,
           identyczny z origin/master.
  VPS      rozni sie od origin/master wylacznie trescia komentarzy (5 linii
           `docs/backlog.md` vs `kb/phases/backlog.md`, skutek migracji sciezek
           w 9128530) — zero roznic funkcjonalnych.

Na obu nodach `_prune_stopped_containers` jest obecny (te same numery linii:
635/700/717/725), a jedyne wystapienia `containers.prune()` to tekst docstringa
i komentarza — brak wykonywalnego niefiltrowanego prune. Sprawdzone dodatkowo,
ze scripts/monitor/health-monitor.sh (ktory nadal ma niefiltrowane
`docker container prune -f` — R1 objelo tylko node_agent.py) nie jest wpiety w
zaden crontab ani timer na SOLARII i VPS, wiec node-agent byl faktycznie jedynym
zrodlem prune i cleanup byl na obu nodach realnie wylaczony.

SOLARIA: przywrocone jawne NODE_TYPE=ai_node — stan sprzed M1 (1cd6401),
zgodnie z konwencja pozostalych hostow, ktore wszystkie ustawiaja NODE_TYPE
jawnie (piha/lustro sd_card, chelsty-infra lte_node). solaria jest w AI_NODES,
wiec default dalby to samo, ale jawny wpis nie zalezy od hostname'u.

VPS: linia usunieta w calosci wraz z komentarzem TEMPORARY — dokladny stan
sprzed 11f3f80, gdzie NODE_TYPE nie bylo ustawione wcale. Potwierdzone, ze
default daje `standard`, nie None: base compose przekazuje `NODE_TYPE=${NODE_TYPE:-}`,
czyli pusty string, ktory jest falsy, wiec _resolve_node_type() schodzi do
rozpoznania po nazwie, a `vps` nie nalezy do LTE_NODES/SD_CARD_NODES/AI_NODES.
Sprawdzone na zlozonym `docker compose config` (NODE_TYPE: "") i uruchomieniem
_resolve_node_type() -> 'standard'. Rotacja filesystemu control-plane jest
bramkowana node_name == VPS_NODE_NAME, nie node_type, wiec dziala niezaleznie.

UWAGA DO DEPLOYU: na obu nodach brak /opt/homelab/state/last-docker-cleanup,
a przy braku markera _cleanup_rate_ok() zwraca True — pierwszy cleanup pojdzie
w pierwszym cyklu po restarcie (<=60 s), nie po 24 h jak na LUSTRO.
W chwili sprawdzenia zero kontenerow `exited` na obu nodach, wiec galaz
kontenerowa nie ma czego usunac; do sprzatniecia sa 4 dangling images na SOLARII
(~553 MB) i 1 na VPS (395 MB) plus build cache. humanai-mailer i humanai-landing
maja restart=unless-stopped, wiec sa chronione pierwsza galezia filtra R1 nawet
gdyby zostaly zatrzymane.

node-agent: 70 passed.

Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md (§7, M1),
docs/sessions/2026-08-06.md (follow-up #5)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 13:47:09 +02:00
oskar 9128530589 fix(kb): 13 pozostalych odwolan do sciezek sprzed migracji
Weryfikacja 0-odwolan z 01db57a liczyla tylko podzbior prefiksow — poza nim
zostalo 13 wskaznikow w plikach niemarkdownowych i w prozie dokumentow:

  docs/kb/modules/05-faza3-plan.md   -> kb/phases/kb-m5-faza3.md (2x systemd)
  docs/kb/modules/05-faza4-plan.md   -> kb/phases/kb-m5-faza4.md (kb-query app.js)
  docs/kb/modules/DECYZJE-*.md       -> kb/decisions/kb-dokumenty-otwarte.md
  docs/backlog.md (npm panel admina) -> kb/decisions/backlog-aktywne.md
  docs/backlog/ (uid/gid floty)      -> kb/decisions/backlog-uid-gid-flota.md
  docs/kb/modules/0X-*.md            -> kb/phases/kb-m*.md
  docs/incidents/2026-07-30-*.md     -> kb/incidents/ (3x node-agent)
  docs/architecture/RECON-multi*.md  -> kb/subsystems/recon-multiagent.md
  jobs/deploy-runner/README.md       -> kb/services/job-deploy-runner.md

Wyjatek zamierzony: `docs/kb/modules/05-faza3-pilot-streszczen.md` w §11 planu
fazy 3 to nazwa artefaktu, ktory nigdy nie powstal — przepiety na docelowa
konwencje (kb/phases/kb-m5-faza3-pilot-streszczen.md), zeby przyszly plik
wyladowal w nowym drzewie, a nie w skasowanym katalogu.

Weryfikacja: skan po 67 sciezkach zmigrowanych w tej galezi (git grep -F na
kazdej) = 0 trafien poza docs/sessions (logi historyczne, celowo nietkniete);
0 martwych linkow markdown na 190 plikach; check_okf.py 190/190 ZGODNE.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:02:12 +02:00
oskar 11f3f808e6 fix(vps): M1 mitigation — NODE_TYPE=lte_node disables unfiltered prune
node-agent calls docker containers.prune() with no filters every
CHECK_INTERVAL (60 s on VPS). The Docker API removes EVERY non-running
container regardless of RestartPolicy or compose labels — this destroyed
ollama@solaria on 2026-07-30 (19 s after an operator `docker stop`).

On VPS the blast radius is worse: humanai-mailer and humanai-landing have
no compose definition in this repo (recreated by hand from `docker inspect`),
so a pruned container there is an irreversible loss of the only config source.

self.node_type is read only by run_safe_cleanup() (node_agent.py:648,654) and
two log lines (250, 1103). Verified additionally for VPS: the control-plane
filesystem rotation and health probe in run_once() are gated on
`node_name == VPS_NODE_NAME`, not node_type — so they keep running. Monitoring,
event shipping and action dispatch are likewise unaffected.

Temporary — remove once R1 (explicit-enumeration prune) is deployed.
Refs docs/incidents/2026-07-30-ollama-solaria-vanish.md §7 M1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:20:02 +02:00
oskar 9e7ed3e077 feat(observer): persist SHADOW_LIVENESS_MISMATCH to mounted file — survives container recreate (cutover evidence)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 15:30:44 +02:00
Oskar Kapala 6e80b92348 chore(control-plane): enable Prometheus shadow-read liveness — start etap 2 parallel-run 2026-07-09 18:29:16 +02:00
Oskar Kapala e5ecefe2b7 feat(observer): shadow-read Prometheus up{} liveness with mismatch logging (cutover etap 1, no switching)
Observer now optionally (PROM_SHADOW_URL) queries Prometheus up{} once per
cycle and LOGS SHADOW_LIVENESS_MISMATCH when its event-driven liveness
disagrees. Parallel-run only: compute_liveness and _emit_node_transition are
untouched; authoritative liveness stays 100% event-driven. Fail-open on any
Prometheus error (down/timeout/bad JSON -> {}). 9 new tests, incl. proof that
shadow-read does not change node_info liveness/status.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 15:38:20 +02:00
Oskar Kapala f64cec645e vps: mem_limit + oom_score_adj na serwisach in-repo; deploy-local stosuje override (stop OOM) 2026-06-01 14:23:58 +02:00
Oskar Kapala 267742c7d7 vps/node-agent: add network_mode: host for control-plane health probe
The _check_control_plane_health() method probes localhost:18180, which
is the control-plane's mapped port. Inside a bridged container, localhost
resolves to the container's own loopback — the probe always fails.

host network mode shares the VPS host's network namespace so that
localhost:18180 correctly reaches the control-plane.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 14:52:32 +02:00
Oskar Kapala c5c080b3e3 feat(vps): add node-agent runtime override with NODE_NAME=vps
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 13:18:19 +02:00
Oskar Kapala 7742bda245 feat(control-plane): add container_restart remediation
- observer: store trigger_type on incidents for supervisor routing
- supervisor: route containers_not_running/mqtt_unreachable to container_restart instead of redeploy
- supervisor: fix node alias normalization via NODE_ALIAS_MAP
- supervisor: fix pending action dedup (scan by content not filename)
- executor: implement container_restart via SSH docker restart with retry
- control-plane override: configure NODE_ALIAS_MAP for production

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 12:50:46 +02:00
oskar 8f305ba3df Merge VPS control plane deployment and observer runtime 2026-05-17 17:30:04 +02:00
oskar c9ddfa9ac1 Roll out stability agent to homelab nodes 2026-05-17 15:54:19 +02:00
Oskar Kapala 2029457f57 Implement VPS control-plane deployment profile 2026-05-12 20:19:05 +02:00