Compare commits

..

2 commits

Author SHA1 Message Date
oskar ff8412b565 chore(vps): remove gokapi (operator decision 2026-08-26)
gokapi was VPS desired state (hosts/vps/services.yaml) with no matching
runtime on the node — a real deployment gap left open at the end of the
2026-08-26 recon session ("redeploy-vps-gokapi pozostawiony — realna
luka wdrożeniowa", docs/sessions/2026-08-26.md). Operator decision this
session: drop it instead of deploying it. Verified zero footprint on
VPS: no data, no container, no image, no /opt/homelab/config/gokapi.

Removed the desired-state entry from hosts/vps/services.yaml and the
services/gokapi/ compose stack. No hosts/vps/runtime/gokapi override
existed to remove.

Grepped the repo for dangling references: jobs/deploy-runner/tests and
services/control-plane/tests use "gokapi" only as an arbitrary example
service name in synthetic tmp_path fixtures (not reading the real
services/gokapi/ directory) — unaffected, left as-is. Fixed one stale
mention in services/control-plane/env.example's example-services
comment. kb/ and docs/sessions/ mentions (service doc, cutover
runbook, an open backlog item, prior session logs) are historical/
narrative record, not code or active config — left untouched, out of
this task's scope; flagged as a follow-up below.

Full control-plane (183), node-agent (70), and deploy-runner (44) test
suites pass unchanged.

Follow-up (not done here — kb/ editing is out of scope for this
worktree task): kb/decisions/backlog-aktywne.md still has an open
"gokapi: deploy-node VPS rzuca błąd — brakujący .env" entry that is now
moot and should be closed/removed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WDKj5LRY8vdQMx57dfNnu
2026-08-26 21:09:46 +02:00
oskar 17ac070b46 fix(control-plane): unique container_restart action_id, no more history overwrite
_generate_recommendation() built container_restart ids as the bare
container-restart-<node>-<service>. Two DIFFERENT incidents for the
same node+service (e.g. a generic containers_not_running restart,
later followed — after recovery and recurrence — by an unrelated
restart for the same service) produced the identical id. Once the
first action reached cancelled/completed/failed, the second action's
own transition into that same directory silently overwrote the first
one's history file. This is exactly what happened 2026-08-26 to a
shadow-mode HA-websocket restart colliding with an unrelated 08-06
entry (docs/sessions/2026-08-26.md) — worked around by hand-renaming
the file that session.

Fix: suffix the id with the triggering incident's started_at —
container-restart-<node>-<service>-<unixts> — NOT time.time() at
generation call time. reconcile() calls _generate_recommendation() on
every loop iteration while the drift persists, and the pending/
approved/running existence check immediately below is what makes that
idempotent; it only works if repeated calls for the SAME ongoing
incident produce the SAME id. started_at is fixed for an incident's
whole life (observer._handle_incident only bumps
last_occurrence/occurrence_count on repeat occurrences — see
COMMIT-1-adjacent code) and changes only when a genuinely new incident
opens for that service, which is exactly "same id while ongoing,
different id on recurrence". Falls back to time.time() if the
incident record is missing/malformed so a restart is still generated.

Scope: only the generic CONTAINER_RESTART_TRIGGERS path
(_generate_recommendation). Left unchanged, deliberately:
  - redeploy-<node>-<service> ids — no observed collision, out of
    scope for this fix (flagged as a latent follow-up below).
  - The HA-specific container-restart-<node>-homeassistant id used by
    _generate_ha_container_restart / _generate_ha_shadow_alert /
    _cancel_ha_container_restart: these three functions rely on an
    exact-match lookup of that fixed id (cooldown check via
    _ha_action_recently_completed, and the cancel path finding the
    specific pending file to move) — adding a suffix there would
    break both without a broader refactor to prefix-glob lookups.
  - alert-ha-*/alert-node-* ids: _ha_action_recently_completed also
    exact-matches these for cooldown dedup; a suffix would defeat
    cooldown entirely (every occurrence would look "new").

node-agent idempotency gate confirmed unaffected: _already_processed()
in node_agent.py does a full-string action_id match against
processed-actions/<id>.done, guarding against RE-processing the exact
same dispatched action file (e.g. a duplicate rsync delivery) — not
against a new action_id for a new occurrence of the same service. A
suffixed id is legitimately a new action to node-agent, which is the
correct behavior (a genuine new incident should actually restart the
container again).

Tests: new test_supervisor_action_id_uniqueness.py covers (1) repeated
_generate_recommendation() calls for the same ongoing incident produce
the same id and do not duplicate the pending file, (2) a new incident
after the old one completed gets a different id and does not overwrite
the old completed record, (3) fallback to time.time() when the
incident record is missing, (4) redeploy ids stay bare. Updated
test_observer_container_events.py's end-to-end assertion to match by
prefix instead of exact filename. Full control-plane suite: 183
passed; node-agent suite: 70 passed (unchanged, confirming the
idempotency gate needed no code change).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WDKj5LRY8vdQMx57dfNnu
2026-08-26 21:08:04 +02:00
2 changed files with 15 additions and 47 deletions

View file

@ -469,27 +469,15 @@ class Supervisor:
# life of one incident (observer._handle_incident only bumps
# last_occurrence/occurrence_count on repeat occurrences) and changes
# only when a new incident is opened for that service — exactly the
# cases we want "same id" and "different id" for, respectively.
#
# No incident record → bare id (the pre-fix format, no suffix at
# all), NOT time.time(). A missing/unlinked incident_id is not just
# a malformed-data edge case: observer._prune_stale_world Case 3
# (commit 71a7af5) clears service.incident_id after 24h of event
# silence even while the underlying drift is still ongoing, so this
# path is hit by a live, still-restarting service. time.time() would
# mint a new action_id — and a new pending file — on every single
# reconcile() tick, defeating the dedup check below entirely. The
# bare id has no incident to distinguish "same" from "different"
# occurrences by, but it is at least stable across calls, which is
# what idempotency here actually requires.
# cases we want "same id" and "different id" for, respectively. Falls
# back to time.time() if the incident record is missing/malformed so
# a restart action is still generated (never block remediation on
# this being unavailable).
if trigger_type in CONTAINER_RESTART_TRIGGERS:
incident_id = self.actual_state["services"].get(drift["svc_key"], {}).get("incident_id")
incident = self.actual_state["incidents"].get(incident_id, {}) if incident_id else {}
started_ts = int(_parse_ts(incident.get("started_at")))
if started_ts:
action_id = f"container-restart-{node}-{service}-{started_ts}"
else:
action_id = f"container-restart-{node}-{service}"
suffix_ts = int(_parse_ts(incident.get("started_at"))) or int(time.time())
action_id = f"container-restart-{node}-{service}-{suffix_ts}"
else:
# redeploy IDs stay bare (node-service) — out of scope for this
# fix (see commit message: no observed collision here yet), and

View file

@ -24,6 +24,7 @@ from __future__ import annotations
import json
import sys
import time
from pathlib import Path
import pytest
@ -127,15 +128,10 @@ def test_new_incident_after_old_completed_gets_different_action_id(sup, tmp_path
assert json.loads((completed_dir / "container-restart-piha-paperless-1000.json").read_text())["status"] == "completed"
def test_fallback_to_bare_id_when_incident_record_missing(sup, tmp_path):
def test_fallback_to_now_when_incident_record_missing(sup, tmp_path):
"""Malformed/missing incident data (incident_id set on the service but no
matching record in incidents.json) must still produce a usable
(non-crashing) action_id, not block remediation. The fallback must be the
bare pre-fix id NOT a time.time() suffix since this path is hit
naturally (not just on malformed data): observer._prune_stale_world
Case 3 (commit 71a7af5) clears service.incident_id after 24h of event
silence while the drift is still ongoing, so a time.time() suffix would
mint a new id on every reconcile() tick forever."""
(non-crashing) action_id, not block remediation."""
svc_key = "piha/paperless"
sup.actual_state["services"][svc_key] = {
"node": "piha", "service": "paperless", "status": "unhealthy",
@ -144,32 +140,16 @@ def test_fallback_to_bare_id_when_incident_record_missing(sup, tmp_path):
# Deliberately no matching entry in sup.actual_state["incidents"].
drift = _drift("piha", "paperless")
before = int(time.time())
sup._generate_recommendation(drift)
after = int(time.time())
pending = _pending(tmp_path)
assert len(pending) == 1
assert pending[0].name == "container-restart-piha-paperless.json"
def test_fallback_bare_id_stable_across_repeated_calls(sup, tmp_path):
"""Same missing-incident-record scenario, but simulating reconcile()
calling _generate_recommendation() on every loop iteration while the
drift persists: must not spam a new pending action each time, exactly
like the has-an-incident-record case above."""
svc_key = "piha/paperless"
sup.actual_state["services"][svc_key] = {
"node": "piha", "service": "paperless", "status": "unhealthy",
"incident_id": "inc-missing",
}
drift = _drift("piha", "paperless")
sup._generate_recommendation(drift)
sup._generate_recommendation(drift)
sup._generate_recommendation(drift)
pending = _pending(tmp_path)
assert len(pending) == 1, f"expected exactly one pending action, got {[p.name for p in pending]}"
assert pending[0].name == "container-restart-piha-paperless.json"
name = pending[0].stem
assert name.startswith("container-restart-piha-paperless-")
suffix_ts = int(name.rsplit("-", 1)[-1])
assert before <= suffix_ts <= after
def test_redeploy_action_id_stays_bare(sup, tmp_path):