- targety node_exporter floty (piha/solaria/lustro) wdrożone (7d4014e)
- fix deploy.sh vps↔control-plane potwierdzony w boju (3b71707)
- PENDING: health-verify targetów w /api/v1/targets nie potwierdzony
- lekcja: nie commitować na master równolegle gdy CC pracuje na wątku (potrójny rebase)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy.sh vps iterates hosts/vps/services.yaml in deploy-node.sh, which
includes control-plane. But control-plane also has a dedicated deploy path
(deploy.sh control-plane → deploy-control-plane.sh → deploy-local.sh) that
runs compose from services/control-plane/. The node loop runs compose from
${REPO_PATH}, so on Compose versions that derive the project name from cwd
the two paths own the containers under different project names. The loop's
`up -d --remove-orphans` then Recreates and tears down the running brain
(observer/supervisor/executor/ui), aborting the loop under set -e. This
wiped the VPS control-plane on 2026-06-25.
Generic guard: skip any service that ships its own services/<svc>/deploy-local.sh.
control-plane stays in services.yaml so the gate (pytest+build) still covers it;
only the destructive loop deploy is skipped. Protects future services with a
dedicated deploy path too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Usuwa Dockerfile i docker-compose.yml; job odpalany lokalnie na PIHA (pip install -e)
- _parse_attachments: manifest MIME → entities[]{type,filename,content_type,size,sha256};
bajty zostają w .eml, wyciąganie/OCR = faza 2
- Batch inserty co 500 wpisów (executemany + ON CONFLICT DO NOTHING); idempotentny
na skipped przez _eml_ref (mirrors archive._UNSAFE); pełna wznawialność
- --limit N: ucina pętlę po N wiadomościach do testów na próbce
- epoch_fallback: licznik + WARNING gdy Date nieparsowalne/brak
- Nowe stats: msgs_with_attachments, total_attachments, total_attachment_bytes
- DSN w docstringu: localhost:5433/kb i piha:5433/kb; usunięto solaria:5433
- 9 nowych testów (24 razem), wszystkie zielone
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One-shot job `jobs/gmail-bulk-import/` wczytuje plik .mbox z Google Takeout
i importuje każdą wiadomość do archiwum .eml + opcjonalnie do koperty w DB.
Idempotentny (FileExistsError → skip; ON CONFLICT DO NOTHING w DB).
15 testów jednostkowych (bez DB, bez zewnętrznych serwisów) — wszystkie zielone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The test_run_once_* cases were flaky/order-dependent. Root cause: observer.observer
derives OBSERVER_STATE_FILE from STATE_DIR at import time. The helper patched
STATE_DIR but never OBSERVER_STATE_FILE, so run_once()/_save_checkpoint() wrote the
checkpoint to the real /opt/homelab/state/observer_checkpoint.json. Those node_checkpoints
(tmp paths tagged with a pytest run number) leaked across tests and across pytest runs;
run_once's `file_path > checkpoint` string compare then skipped/kept events based on
run-number ordering. The helper also never restored the module globals it overwrote.
Replace both ad-hoc helpers with an autouse monkeypatch fixture that redirects every
observer path — including OBSERVER_STATE_FILE — into the per-test tmp_path and reverts
them afterward. Tests no longer touch real disk and are deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without --env-file, docker compose resolved variables from the repo root
(cwd), not from services/<service>/.env where the file actually lives.
This caused ${TAILSCALE_BIND_IP} to expand to empty string, binding
fleet-prometheus on 0.0.0.0:9090 instead of the Tailscale-only IP —
a security hole on the public VPS.
Guard mirrors the existing override-file pattern: only add --env-file
when the file exists, so services without .env continue to work as
before. Flag is injected into COMPOSE_CMD (before the `up` subcommand)
so docker compose sees it as a global option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the brittle `hosts/$(hostname|lower)` assumption with a scan
of hosts/*/host.yaml for a matching os_hostname field. This fixes VPS
where the OS hostname (ubuntu-4gb-hel1-1) never matched the repo
directory (hosts/vps/), causing a silent "No services found" false-green.
Fallback to lowercase-hostname dir preserved for nodes that haven't yet
received the os_hostname field; exits 1 with a clear message if neither
match nor fallback directory exists.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Declares the real OS-level hostname for each node so deploy-node.sh
can resolve the correct hosts/ directory without assuming
OS-hostname == logical name. Only VPS differs: its OS hostname is
ubuntu-4gb-hel1-1 while the repo directory is hosts/vps/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously published "9090:9090" → Docker bound to 0.0.0.0 (all host interfaces,
including the public Hetzner IP), leaving tailscale-internal enforced only by the
VPS firewall. Now bind explicitly to the VPS Tailscale interface for
defense-in-depth: the port does not exist on the public IP at all.
- ports -> "${TAILSCALE_BIND_IP}:9090:9090"
- env.example: add TAILSCALE_BIND_IP (VPS Tailscale IP, verify via `tailscale ip -4`)
- README: deploy section — .env is mandatory; a missing .env makes Compose
silently bind 0.0.0.0 (warns, does not fail), so use --env-file and verify host_ip
Smoke: config with --env-file and with co-located .env both resolve
host_ip=100.95.58.48; with .env absent Compose warns and falls back to 0.0.0.0
(documented). .env is gitignored (global *.env rule).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- hosts/vps/services.yaml: fleet-prometheus entry (role: fleet-liveness-source,
exposure: tailscale-internal, port 9090, runtime paths, depends_on node_exporter)
- inventory/topology.yaml: add fleet-prometheus to the vps services list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clean scaffold of a Prometheus instance dedicated to fleet liveness, deliberately
separate from the home `prom` on PIHA. Scrapes only itself + the VPS-local
node_exporter for now.
- image pinned to prom/prometheus:v3.5.0 (LTS) — no :latest anti-pattern
- TSDB retention 15d / 2GB on the tight VPS; --web.enable-lifecycle for reloads
- mem_limit 512m, oom_score_adj +200 (sacrificial OOM victim before control-plane)
- tailscale-internal exposure via plain published 9090, mirroring control-plane
- node_exporter scraped via host.docker.internal:9100 (it runs network_mode: host)
- secret-free prometheus.yml; alerting/rule_files left as commented placeholders
- in-container healthcheck via busybox wget (present in the image; curl is not)
Smoke: docker compose config OK; promtool check config SUCCESS; up -d ->
/-/healthy + /-/ready OK; both targets (prometheus, fleet-node) report up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Celowy stan host-level (poza GitOps) udokumentowany w manifescie, zeby przy
odtwarzaniu VPS nie zniknal cicho. Tylko opisowy blok host_state, bez zmian
struktury runtime/deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swap 2-4GB na VPS -> Zamkniete (2026-06-22). Nowa sekcja planu monitoringu
floty (Prometheus pull up{}, osobny instance, bez Alertmanagera) z krokami.
Zastepuje szkic blackbox+Alertmanager z 2026-06-17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nodes that crash or lose connectivity without emitting node_offline
stay online in world state indefinitely. _prune_stale_world() now
flips any online node to offline if its last_seen is older than
NODE_OFFLINE_TTL_SECS (default 300 s = 5× the 60 s heartbeat interval).
Nodes with last_seen=None (never reported) and already-offline nodes
are left unchanged. Five new tests cover all branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses review findings — all reverting to the live config that worked:
- config.yml mounts at /app/vikunja/config.yml (live path), not /etc/vikunja.
- Remove the vikunja container healthcheck: the image ships no wget/curl, so
an in-container HTTP check is always unhealthy. Health stays on db
(pg_isready) + host-side healthcheck.sh (curl). Live had no app healthcheck.
- Secrets injected exclusively via env_file (.env) on BOTH services; dropped
the ${VAR:?} parse-time interpolation that depended on a .env in cwd. db now
also has env_file. Non-secret env stays inline.
- Rename .env.example -> env.example to match repo convention (forgejo).
Verified: `docker compose -f services/vikunja/docker-compose.yml config` passes;
services/vikunja/.env is gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wraps the existing manually-run Vikunja instance on PIHA in the standard
service layout. No deploy — definition only, for review.
- services/vikunja/: docker-compose.yml (vikunja + postgres:16-alpine),
service.yaml, README, healthcheck, config.yml (OIDC, secret-free),
.env.example. Real .env stays gitignored.
- Pins EXISTING named volumes (vikunja_vikunja_db, vikunja_vikunja_files)
so DB + uploaded files survive cutover.
- extra_hosts forgejo.okit.pl=192.168.31.5 (npm on PIHA) so OIDC discovery
resolves over LAN instead of flaky public DNS.
- OIDC client secret injected via env (VIKUNJA_AUTH_OPENID_PROVIDERS_
FORGEJO_CLIENTSECRET); config.yml keeps trailing-slash authurl/redirecturl.
- Registers vikunja in hosts/piha/services.yaml + inventory/topology.yaml,
plus hosts/piha/runtime/vikunja override (advisory mem_limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container runs as uid 1000 (homelab), HOME=/home/homelab. ssh without -i
looks for $HOME/.ssh — mounting to /root/.ssh was never visible to the
process user and caused silent Permission denied on event shipping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recovery from bad merge of task/observer-poison-quarantine (c255a02)
which carried false deletes from a stale branch base. Re-applies only
the genuine observer changes on top of correct master state.
When an event file fails to parse (malformed JSON, truncated, corrupted),
the observer previously kept retrying on every cycle while the node's
checkpoint stayed pinned — all subsequent good events for that node lost.
Now: first parse failure -> atomic os.replace to STATE_DIR/observer_failed_events/<node>/
with collision handling. Checkpoint advances, downstream events flow.
Move failures are logged but don't crash the loop.
Complementary to the atomic_write_json fix on state files; this addresses
the same race-pattern on event files instead.
Regression test asserts: bad event quarantined to failed_events dir,
removed from hot path, subsequent good event processed (node online),
checkpoint moves to good event.
Port 8087 is no longer mapped to the host. Operator verify commands
that used curl http://localhost:8087/health now use docker exec with
Python's urllib (the image is python:3.11-slim, no curl binary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port 8087 conflicted with zigbee2mqtt on piha (8087:8080 mapping active
for 7+ days), preventing ha-diag-agent from starting.
Grep across the full repo confirms no external consumer (no nginx/npm
proxy, no Prometheus scrape, no control-plane reference) uses this port.
The Docker healthcheck runs inside the container network namespace and
does not require a host-side mapping. Internal FastAPI binding on 8087
is unchanged.
Removed: ports section from docker-compose.yml and service.yaml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- main.py: rename event= to ha_event= in _log.warning() — structlog treats
'event' as a reserved positional arg; the old name caused TypeError when
any check returned unhealthy results (events were still emitted, but the
check was logged as check_error instead of check_unhealthy)
- tests/test_ha_client.py: replace aioresponses with unittest.mock — aioresponses
0.7.8 is incompatible with aiohttp >=3.12 (missing stream_writer kwarg)
- pyproject.toml: remove aioresponses from dev dependencies
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
node-agent runs as uid 1000 (homelab) since the base compose sets
user "1000:1000"; ssh in _ship_events_to_vps() has no -i flag and looks
for keys in $HOME/.ssh = /home/homelab/.ssh. The old mount target
/root/.ssh was never consulted, so rsync to VPS failed with
'Permission denied'. uid match (pi=1000 on RPi OS) keeps OpenSSH strict
ownership checks happy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oskar Kapala
