homelab-codex-ws/services
oskar d5139c99ca fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node)
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.

Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.

- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
  existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
  checkpoints (and the older last_processed_file format) to int epochs;
  unparseable → 0 (reprocess all — safe, process_event is idempotent on
  last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
  but-newer processed, unparseable name falls back to mtime (not wedged),
  ts-not-path ordering, both checkpoint-format migrations, helper units.

Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 15:55:38 +02:00
..
agent-system feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00
brain-watchdog feat(brain-watchdog): poll Prometheus /api/v1/alerts as second alert source (Telegram) 2026-06-30 18:51:34 +02:00
control-plane fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node) 2026-07-14 15:55:38 +02:00
fleet-prometheus feat(fleet-prometheus): add liveness alert rules for always-on nodes (vps, piha) 2026-06-30 16:42:52 +02:00
forgejo fix(inventory): correct owner_node — forgejo→piha, mosquitto→vps (per 2026-07-02 verify) 2026-07-02 16:48:12 +02:00
gokapi feat(gokapi): config publicznego file-share na VPS (share.okit.pl, E2E, Tailscale-bind za npm@VPS, disk-protection) — do deployu 2026-07-09 17:29:28 +02:00
ha-diag-agent docs(ha-diag-agent): replace curl verify commands with docker exec 2026-06-11 19:46:33 +02:00
kb-postgres feat(kb-postgres): migration 002_chunks.sql — document_chunk table 2026-07-13 21:13:13 +02:00
llm-gateway feat(llm-gateway): wciagniecie shadow-serwisu z PIHA do GitOps 2026-07-03 15:08:33 +02:00
mosquitto fix(inventory): correct owner_node — forgejo→piha, mosquitto→vps (per 2026-07-02 verify) 2026-07-02 16:48:12 +02:00
nextcloud feat(kb): configi Paperless/Nextcloud wg 9 decyzji — NC na PIHA, domeny kapala, Redis requirepass, backup SOLARIA, NC pin 34 2026-07-09 16:17:27 +02:00
node-agent fix(node-agent): rsync --no-perms/--no-owner/--no-group — same aerbot-owned dir EPERM as dir-times (exit 23 after omit-dir-times fix) 2026-07-13 21:15:41 +02:00
node_exporter Fix pending actions: node_exporter, zigbee2mqtt, chelsty-ha monitoring 2026-05-27 15:10:48 +02:00
npm Add node capability model 2026-05-11 20:46:50 +02:00
ollama Add node capability model 2026-05-11 20:46:50 +02:00
paperless fix(paperless-worker): celery command bypassed manage.py + missing shared scratch dir 2026-07-12 20:51:04 +02:00
paperless-worker fix(paperless-worker): celery command bypassed manage.py + missing shared scratch dir 2026-07-12 20:51:04 +02:00
planner-agent fix+debug(planner-agent): use base_url (not api_base) for litellm.acompletion, add print [TEMP] 2026-05-28 13:07:58 +02:00
stability-agent fix(stability-agent): run as uid 1000 with docker group access 2026-06-03 18:20:54 +02:00
vikunja fix(vikunja): PUBLICURL na vikunja.kapala.org 2026-07-02 17:28:56 +02:00
zigbee2mqtt docs: compress CLAUDE.md + fix zigbee2mqtt coordinator docs 2026-05-29 14:17:23 +02:00
.gitkeep Add infrastructure standards and deployment conventions 2026-05-07 21:16:03 +02:00