homelab-codex-ws/docs
oskar d5139c99ca fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node)
Per-node checkpoint now stores the last-processed event TIMESTAMP (int epoch)
instead of a file path compared lexically. A file is "new" iff its timestamp
(parsed from evt-<node>-<unixts>-<type>-<svc>.json, mtime fallback) exceeds the
node's checkpoint; processing is ordered by timestamp, not path.

Root cause (PIHA dead ~34d, 2026-07-12): a stray evt-unknown-<ts>-… file landed
in events/piha/, lexically greater than every evt-piha-… name. The lexical
checkpoint pinned there, so every genuinely newer piha event sorted "before" it
and was skipped forever. Event backlog grew to 7344 files, last_seen frozen,
shadow-read logged false SHADOW_LIVENESS_MISMATCH event=dead prom=up.

- _event_ts_from_path: filename epoch, mtime fallback; NEVER returns 0 for an
  existing file (0 == "older than checkpoint" == the poison).
- _checkpoint_ts_from_value: graceful migration of pre-fix path-string
  checkpoints (and the older last_processed_file format) to int epochs;
  unparseable → 0 (reprocess all — safe, process_event is idempotent on
  last_seen/world_state; bias to reprocess, never to skip).
- Preserved: quarantine of bad events, observer-source re-ingest guard.
- Regression tests (test_incident_lifecycle.py section 9): lexically-smaller-
  but-newer processed, unparseable name falls back to mtime (not wedged),
  ts-not-path ordering, both checkpoint-format migrations, helper units.

Separate bug filed in backlog (not fixed here): ha-diag-agent emits node=
"unknown" events (config.py node_name default) into another node's dir when
NODE_NAME reaches the compose volume path but not the app env — the source of
the poison file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 15:55:38 +02:00
..
infra docs(infra): migracja okit.pl 42.pl->Cloudflare — faza 0 done+zweryfikowana, split-horizon Pi-hole, plan faz 1-2 2026-07-07 20:46:57 +02:00
kb docs(kb): plan fazy 2 modułu 5 — koperta dokumentów (recon + plan v2) 2026-07-13 21:11:30 +02:00
sessions docs(sesja): 2026-07-12 Deploy 2 OCR-worker DZIALA (split-host NFS) + backlog: nowe serwisy KB poza monitoringiem 2026-07-12 21:15:38 +02:00
access.md Document current homelab state 2026-04-15 17:37:25 +02:00
agents.md docs: uzupelnij dokumentacje pod katem agentow AI 2026-05-20 12:06:23 +02:00
backlog.md fix(observer): checkpoint by timestamp not lexical path — lexically-smaller-but-newer events were silently skipped forever (poisoned node) 2026-07-14 15:55:38 +02:00
capabilities.md docs: uzupelnij dokumentacje pod katem agentow AI 2026-05-20 12:06:23 +02:00
chelsty-runtime.md docs: session summary 2026-05-27 + update observer/control-plane/chelsty docs 2026-05-27 16:18:31 +02:00
chelsty-stability-agent.md docs(chelsty): update docs and topology for site/node split 2026-05-20 14:23:57 +02:00
core-stack.md Document current homelab state 2026-04-15 17:37:25 +02:00
deployment.md docs(deployment): add critical warning — deploy.sh vps destroys control-plane (project-name divergence, 2026-06-25 incident) 2026-06-25 16:30:53 +02:00
event-system.md Implement filesystem-first runtime event system 2026-05-12 13:38:25 +02:00
hardware.md Document current homelab state 2026-04-15 17:37:25 +02:00
hetzner-vps.md Document VPS connectivity diagnostics 2026-04-15 20:38:09 +02:00
joplin-server.md Document VPS connectivity diagnostics 2026-04-15 20:38:09 +02:00
lifecycle.md Harden deployment runtime framework 2026-05-11 21:20:13 +02:00
networking.md Document current homelab state 2026-04-15 17:37:25 +02:00
node-onboarding.md Implement reproducible node onboarding 2026-05-12 13:18:00 +02:00
observer-runtime.md feat(observer): 3-state node liveness (fresh/stale/dead) + transitions + read-time net 2026-06-17 20:07:25 +02:00
questions.md Document Hetzner VPS handoff 2026-04-15 17:46:42 +02:00
service-model.md Add node capability model 2026-05-11 20:46:50 +02:00
services.md Document Hetzner VPS handoff 2026-04-15 17:46:42 +02:00
stability-agent-rollout.md Fix stability agent fleet deploy scripts 2026-05-17 21:09:06 +02:00
standards.md docs: uzupelnij dokumentacje pod katem agentow AI 2026-05-20 12:06:23 +02:00
topology.md Add CHELSTY home automation inventory model 2026-05-11 20:54:54 +02:00
vps-control-plane.md docs: session summary 2026-05-27 + update observer/control-plane/chelsty docs 2026-05-27 16:18:31 +02:00