One-shot job `jobs/gmail-bulk-import/` wczytuje plik .mbox z Google Takeout
i importuje każdą wiadomość do archiwum .eml + opcjonalnie do koperty w DB.
Idempotentny (FileExistsError → skip; ON CONFLICT DO NOTHING w DB).
15 testów jednostkowych (bez DB, bez zewnętrznych serwisów) — wszystkie zielone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The test_run_once_* cases were flaky/order-dependent. Root cause: observer.observer
derives OBSERVER_STATE_FILE from STATE_DIR at import time. The helper patched
STATE_DIR but never OBSERVER_STATE_FILE, so run_once()/_save_checkpoint() wrote the
checkpoint to the real /opt/homelab/state/observer_checkpoint.json. Those node_checkpoints
(tmp paths tagged with a pytest run number) leaked across tests and across pytest runs;
run_once's `file_path > checkpoint` string compare then skipped/kept events based on
run-number ordering. The helper also never restored the module globals it overwrote.
Replace both ad-hoc helpers with an autouse monkeypatch fixture that redirects every
observer path — including OBSERVER_STATE_FILE — into the per-test tmp_path and reverts
them afterward. Tests no longer touch real disk and are deterministic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without --env-file, docker compose resolved variables from the repo root
(cwd), not from services/<service>/.env where the file actually lives.
This caused ${TAILSCALE_BIND_IP} to expand to empty string, binding
fleet-prometheus on 0.0.0.0:9090 instead of the Tailscale-only IP —
a security hole on the public VPS.
Guard mirrors the existing override-file pattern: only add --env-file
when the file exists, so services without .env continue to work as
before. Flag is injected into COMPOSE_CMD (before the `up` subcommand)
so docker compose sees it as a global option.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the brittle `hosts/$(hostname|lower)` assumption with a scan
of hosts/*/host.yaml for a matching os_hostname field. This fixes VPS
where the OS hostname (ubuntu-4gb-hel1-1) never matched the repo
directory (hosts/vps/), causing a silent "No services found" false-green.
Fallback to lowercase-hostname dir preserved for nodes that haven't yet
received the os_hostname field; exits 1 with a clear message if neither
match nor fallback directory exists.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Declares the real OS-level hostname for each node so deploy-node.sh
can resolve the correct hosts/ directory without assuming
OS-hostname == logical name. Only VPS differs: its OS hostname is
ubuntu-4gb-hel1-1 while the repo directory is hosts/vps/.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously published "9090:9090" → Docker bound to 0.0.0.0 (all host interfaces,
including the public Hetzner IP), leaving tailscale-internal enforced only by the
VPS firewall. Now bind explicitly to the VPS Tailscale interface for
defense-in-depth: the port does not exist on the public IP at all.
- ports -> "${TAILSCALE_BIND_IP}:9090:9090"
- env.example: add TAILSCALE_BIND_IP (VPS Tailscale IP, verify via `tailscale ip -4`)
- README: deploy section — .env is mandatory; a missing .env makes Compose
silently bind 0.0.0.0 (warns, does not fail), so use --env-file and verify host_ip
Smoke: config with --env-file and with co-located .env both resolve
host_ip=100.95.58.48; with .env absent Compose warns and falls back to 0.0.0.0
(documented). .env is gitignored (global *.env rule).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- hosts/vps/services.yaml: fleet-prometheus entry (role: fleet-liveness-source,
exposure: tailscale-internal, port 9090, runtime paths, depends_on node_exporter)
- inventory/topology.yaml: add fleet-prometheus to the vps services list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clean scaffold of a Prometheus instance dedicated to fleet liveness, deliberately
separate from the home `prom` on PIHA. Scrapes only itself + the VPS-local
node_exporter for now.
- image pinned to prom/prometheus:v3.5.0 (LTS) — no :latest anti-pattern
- TSDB retention 15d / 2GB on the tight VPS; --web.enable-lifecycle for reloads
- mem_limit 512m, oom_score_adj +200 (sacrificial OOM victim before control-plane)
- tailscale-internal exposure via plain published 9090, mirroring control-plane
- node_exporter scraped via host.docker.internal:9100 (it runs network_mode: host)
- secret-free prometheus.yml; alerting/rule_files left as commented placeholders
- in-container healthcheck via busybox wget (present in the image; curl is not)
Smoke: docker compose config OK; promtool check config SUCCESS; up -d ->
/-/healthy + /-/ready OK; both targets (prometheus, fleet-node) report up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Celowy stan host-level (poza GitOps) udokumentowany w manifescie, zeby przy
odtwarzaniu VPS nie zniknal cicho. Tylko opisowy blok host_state, bez zmian
struktury runtime/deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swap 2-4GB na VPS -> Zamkniete (2026-06-22). Nowa sekcja planu monitoringu
floty (Prometheus pull up{}, osobny instance, bez Alertmanagera) z krokami.
Zastepuje szkic blackbox+Alertmanager z 2026-06-17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the "dead node shown NOMINAL" silent outage: node status was set only by
events and never expired, so a node that crashed/lost connectivity stayed
"online" forever (chelsty-infra was online for 16d, piha ~6d). The only thing
that flipped status to offline was a node_offline event, which an unreachable
node can never emit.
Now node status is derived from freshness (now - last_seen), recomputed every
observer cycle (incl. cycles with no new events):
- always-on: fresh <=180s, stale 180-600s, dead >600s (3x the 60s heartbeat)
- remote/LTE (chelsty-*): fresh <=900s, stale 900-3600s, dead >3600s
Thresholds + tier logic live in ONE shared helper, services/control-plane/src/
liveness.py, imported by the observer and both operator UIs (bind-mounted into
the agent-system webui image). No 3x copy.
Transitions are not silent: the observer emits node_stale / node_offline /
node_online (recovery) events tagged source=observer (skipped on re-ingest so
they never reset last_seen), routed by the supervisor to alert_only actions.
Read-time safety net: both UIs recompute liveness from last_seen at request
time, so a stalled observer still surfaces dead nodes. Services inherit their
node's liveness (cascade, variant B) without mutating services.json.
Replaces the earlier binary NODE_OFFLINE_TTL_SECS flip.
Tests: liveness unit tests, observer 3-state + transitions/recovery/baseline +
self-event skip, operator_ui read-time net + cascade, supervisor node-event
routing. 89 passed. docker compose config valid for both stacks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nodes that crash or lose connectivity without emitting node_offline
stay online in world state indefinitely. _prune_stale_world() now
flips any online node to offline if its last_seen is older than
NODE_OFFLINE_TTL_SECS (default 300 s = 5× the 60 s heartbeat interval).
Nodes with last_seen=None (never reported) and already-offline nodes
are left unchanged. Five new tests cover all branches.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses review findings — all reverting to the live config that worked:
- config.yml mounts at /app/vikunja/config.yml (live path), not /etc/vikunja.
- Remove the vikunja container healthcheck: the image ships no wget/curl, so
an in-container HTTP check is always unhealthy. Health stays on db
(pg_isready) + host-side healthcheck.sh (curl). Live had no app healthcheck.
- Secrets injected exclusively via env_file (.env) on BOTH services; dropped
the ${VAR:?} parse-time interpolation that depended on a .env in cwd. db now
also has env_file. Non-secret env stays inline.
- Rename .env.example -> env.example to match repo convention (forgejo).
Verified: `docker compose -f services/vikunja/docker-compose.yml config` passes;
services/vikunja/.env is gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wraps the existing manually-run Vikunja instance on PIHA in the standard
service layout. No deploy — definition only, for review.
- services/vikunja/: docker-compose.yml (vikunja + postgres:16-alpine),
service.yaml, README, healthcheck, config.yml (OIDC, secret-free),
.env.example. Real .env stays gitignored.
- Pins EXISTING named volumes (vikunja_vikunja_db, vikunja_vikunja_files)
so DB + uploaded files survive cutover.
- extra_hosts forgejo.okit.pl=192.168.31.5 (npm on PIHA) so OIDC discovery
resolves over LAN instead of flaky public DNS.
- OIDC client secret injected via env (VIKUNJA_AUTH_OPENID_PROVIDERS_
FORGEJO_CLIENTSECRET); config.yml keeps trailing-slash authurl/redirecturl.
- Registers vikunja in hosts/piha/services.yaml + inventory/topology.yaml,
plus hosts/piha/runtime/vikunja override (advisory mem_limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container runs as uid 1000 (homelab), HOME=/home/homelab. ssh without -i
looks for $HOME/.ssh — mounting to /root/.ssh was never visible to the
process user and caused silent Permission denied on event shipping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Recovery from bad merge of task/observer-poison-quarantine (c255a02)
which carried false deletes from a stale branch base. Re-applies only
the genuine observer changes on top of correct master state.
When an event file fails to parse (malformed JSON, truncated, corrupted),
the observer previously kept retrying on every cycle while the node's
checkpoint stayed pinned — all subsequent good events for that node lost.
Now: first parse failure -> atomic os.replace to STATE_DIR/observer_failed_events/<node>/
with collision handling. Checkpoint advances, downstream events flow.
Move failures are logged but don't crash the loop.
Complementary to the atomic_write_json fix on state files; this addresses
the same race-pattern on event files instead.
Regression test asserts: bad event quarantined to failed_events dir,
removed from hot path, subsequent good event processed (node online),
checkpoint moves to good event.
Port 8087 is no longer mapped to the host. Operator verify commands
that used curl http://localhost:8087/health now use docker exec with
Python's urllib (the image is python:3.11-slim, no curl binary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port 8087 conflicted with zigbee2mqtt on piha (8087:8080 mapping active
for 7+ days), preventing ha-diag-agent from starting.
Grep across the full repo confirms no external consumer (no nginx/npm
proxy, no Prometheus scrape, no control-plane reference) uses this port.
The Docker healthcheck runs inside the container network namespace and
does not require a host-side mapping. Internal FastAPI binding on 8087
is unchanged.
Removed: ports section from docker-compose.yml and service.yaml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- main.py: rename event= to ha_event= in _log.warning() — structlog treats
'event' as a reserved positional arg; the old name caused TypeError when
any check returned unhealthy results (events were still emitted, but the
check was logged as check_error instead of check_unhealthy)
- tests/test_ha_client.py: replace aioresponses with unittest.mock — aioresponses
0.7.8 is incompatible with aiohttp >=3.12 (missing stream_writer kwarg)
- pyproject.toml: remove aioresponses from dev dependencies
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
node-agent runs as uid 1000 (homelab) since the base compose sets
user "1000:1000"; ssh in _ship_events_to_vps() has no -i flag and looks
for keys in $HOME/.ssh = /home/homelab/.ssh. The old mount target
/root/.ssh was never consulted, so rsync to VPS failed with
'Permission denied'. uid match (pi=1000 on RPi OS) keeps OpenSSH strict
ownership checks happy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rsync fails with "No such file or directory" when intermediate dirs
don't exist. /opt/homelab/deploy/ is not created by 20-base.sh.
Add rrun mkdir -p before rsync_dir; pi owns /opt/homelab so no sudo.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Push-based deploy step for LUSTRO (git_control=false): rsync
services/node-agent/ and the host override to /opt/homelab/deploy/node-agent/
on the remote, then docker compose up --build via SSH.
Guard by effect: skip push+build+up if node-agent container already running
(docker ps filter, not command -v). Verify: container running + events appear
in /opt/homelab/events/lustro/ within 90 s (confirms agent write path).
Override (hosts/lustro/runtime/node-agent/docker-compose.override.yml):
- group_add: ["991"] (docker GID on LUSTRO; 999 from base concatenated — harmless)
- mem_limit: 256m (MagicMirror ~1.9 GiB; agent must be bounded)
- /home/pi/.ssh:/root/.ssh:ro (not /home/oskar/.ssh — pi user)
- /opt/homelab/deploy/node-agent:/repo:ro (no repo checkout on push-based node)
- NODE_NAME=lustro, NODE_TYPE=sd_card, VPS_EVENTS_HOST=100.95.58.48
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stary guard porównywał literał konfigu (SIZE=) zamiast sprawdzać efekt.
Ręcznie postawiony zram był pomijany (dpkg -l vs command -v) i config
był nadpisywany niepotrzebnie.
- Guard by effect: sudo swapon --show | grep /dev/zram + dphys nieaktywny
→ cała sekcja skip bez wchodzenia w substages
- Detekcja pakietu przez dpkg -l zram-tools (nie command -v zramswap — PATH)
- Config: PERCENT=50 (skaluje z RAM) zamiast SIZE=; printf '%s\n' | sudo tee
- Wszystkie weryfikacje zram przez sudo swapon --show (nie zramctl)
- Usuń parsowanie hardware.swap.mb (nieużywane po przejściu na PERCENT)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three idempotent stages with guards (probe-before-mutate), rrun() for all
remote mutations, rprobe() for unconditional state queries. Reads
hardware.swap.mb from node.yaml (default 2048 MB). Adds swap.mb: 2048
to hosts/lustro/node.yaml so the value is declarative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ECC-format skill for the node onboarding workflow. Covers full step
sequence, operational rules, node.yaml key fields, gotchas from LUSTRO
session, and Definition of Done. Marked as living doc — SCAFFOLD sections
to be promoted to PROVEN as steps land on real nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scripts/onboard/README.md (new):
- Tool purpose and --node/--step/--from/--dry-run usage
- Full node.yaml field schema with annotations (ssh_user uid-1000
gotcha, first_contact IP vs .local, deploy_autonomy/git_control gates)
- Step status table (00-access DONE, 00-preflight SCAFFOLD, 10-50 TODO)
- lib/ architecture: run() dry-run convention, yaml_get fallback caveats
- Gotchas/Learnings table from session
CLAUDE.md:
- Node Onboarding section: onboard.sh commands, pointer to README
- Multi-agent worktree mode: add explicit DISCIPLINE RULE — feature
work must happen in agent.sh worktrees, not the main checkout;
references the 2026-06-08 session that violated this
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Records the onboarding session for LUSTRO (RPi4, KEN site):
node facts from preflight, key decisions (user pi/uid-1000, IP
over mDNS, zram target), 00-access status, tool bugs fixed
(dry-run propagation, yaml_get greedy-colon + inline comment,
ssh known-hosts in verify), open items for next session
(worktree hygiene first, bootstrap-runtime, node-agent, register,
verify, mm-watch).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Oskar Kapala