homelab-codex-ws/services/control-plane/tests
oskar d5139c99ca fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node)
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.

Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.

- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
  existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
  checkpoints (and the older last_processed_file format) to int epochs;
  unparseable → 0 (reprocess all — safe, process_event is idempotent on
  last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
  but-newer processed, unparseable name falls back to mtime (not wedged),
  ts-not-path ordering, both checkpoint-format migrations, helper units.

Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 15:55:38 +02:00
..
__init__.py feat(control-plane): route ha-diag-agent events through supervisor 2026-05-29 15:59:23 +02:00
test_incident_lifecycle.py fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node) 2026-07-14 15:55:38 +02:00
test_liveness.py feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00
test_observer_shadow_liveness.py feat(observer): shadow-read Prometheus up{} liveness with mismatch logging (cutover etap 1, no switching) 2026-07-09 15:38:20 +02:00
test_operator_ui_liveness.py feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00
test_state_reliability.py test(control-plane): atomic write and resilient loader coverage 2026-06-03 12:27:05 +02:00
test_supervisor_ha.py feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00
test_supervisor_node_events.py feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00