healthcheck_failed incidents fell through to redeploy, which is broken as
wired (executor calls deploy-node.sh with arguments it ignores, at a path
that does not exist in the container) — so 3376 healthcheck_failed events
dead-ended with no working remediation (recon D14/D15). A container restart
plausibly heals a failing healthcheck and rides the executor path that
actually works; redeploy returns to the map once etap 2 fixes the executor.
service_unhealthy / deployment_failed / missing_service stay on redeploy —
theoretical until etap 2, kept so drift remains visible in pending actions
(noted in comments). CLAUDE.md routing table updated to match; stale
mqtt_unreachable example in the observer's trigger_type comment refreshed.
Tests: trigger-type recognition and the end-to-end observer→supervisor
reconcile test parametrized over both container_restart triggers, with an
assertion that no redeploy action is also generated. Full control-plane
suite: 147 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
node-agent (and stability-agent) emit containers_not_running for exited/dead
and crash-looping containers, but the observer's event->status/incident map
only handled service_recovered/service_healthy/service_unhealthy/
healthcheck_failed. containers_not_running fell through: status stayed at its
last "healthy" value and no incident was opened, so the supervisor saw no drift
and generated no action — a dead container produced ZERO operator alerts
(matches the "action queue empty despite failures" symptom).
- containers_not_running now sets status=unhealthy and opens an incident whose
trigger_type ("containers_not_running") is already in the supervisor's
CONTAINER_RESTART_TRIGGERS, so remediation (container_restart) fires with no
supervisor change. Recovery is unchanged: service_healthy resolves the
incident via the existing svc_key->incident_id link.
- container_restarting / container_state_unexpected (added in 4746ebe) are kept
intentionally observational — no incident, status not flipped to unhealthy
(would cause a false redeploy for a transient blip) — but leave a
last_observation trace so they don't vanish. A real crash-loop still escalates
via node-agent re-emitting containers_not_running.
Tests: services/control-plane/tests/test_observer_container_events.py — status
+ incident + trigger_type, end-to-end reconcile -> container_restart, recovery
auto-resolve, observational no-incident/trace, idempotent (no incident
multiplication). Full control-plane suite: 117 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
