homelab-codex-ws/scripts
oskar d5139c99ca fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node)
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.

Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.

- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
  existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
  checkpoints (and the older last_processed_file format) to int epochs;
  unparseable → 0 (reprocess all — safe, process_event is idempotent on
  last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
  but-newer processed, unparseable name falls back to mtime (not wedged),
  ts-not-path ordering, both checkpoint-format migrations, helper units.

Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 15:55:38 +02:00
..
bootstrap Implement VPS control-plane deployment profile 2026-05-12 20:19:05 +02:00
deploy fix(deploy): exclude control-plane from node deploy loop — prevents brain teardown via project-name divergence 2026-06-26 15:16:42 +02:00
dev fix(dev): agent.sh worktree_count/paths grep exit-1 on empty set 2026-06-03 18:04:38 +02:00
lib Implement filesystem-first runtime event system 2026-05-12 13:38:25 +02:00
monitor feat(node-agent): implement health monitor and safe cleanup policy 2026-05-27 13:15:06 +02:00
npm feat(scripts): npm_api.py — CLI do zarzadzania npm PIHA+VPS przez REST API 2026-07-10 14:57:42 +02:00
observer fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node) 2026-07-14 15:55:38 +02:00
onboard feat(onboard): implement 40-register + 50-verify, remove dead scaffold 2026-06-09 20:36:00 +02:00
bootstrap.sh Initial homelab workspace structure 2026-05-07 20:17:27 +02:00