feat(vps): migracja npm/outline/joplin/ai-cluster do GitOps (manifesty; cutover NIE wykonany)

.claude/skills/deploy/SKILL.md

@@ -1,43 +0,0 @@
----
-name: deploy
-description: Deploy, redeploy, or ship homelab services to a target node. Trigger on any request containing deploy / redeploy / wdróż / zredeployuj / ship for targets control-plane, vps, piha, solaria, or chelsty-infra.
----
-
-Always invoke `scripts/deploy/deploy.sh <target> [--dry-run] [--no-gate]` as the **sole entry point**.
-Never call `deploy-control-plane.sh`, `deploy-node.sh`, or `deploy-local.sh` directly.
-
-## Targets
-
-| Target | What it deploys |
-|---|---|
-| `control-plane` | observer, supervisor, executor, operator-ui on VPS |
-| `vps` | all VPS GitOps services (node-agent, npm, outline, joplin, ai-cluster, …) |
-| `piha` | PIHA services (ha-diag-agent, node-agent, redis, …) |
-| `solaria` | SOLARIA compute services |
-| `chelsty-infra` | CHELSTY LTE edge node (30 s SSH timeout) |
-
-## Invocation
-
-```bash
-scripts/deploy/deploy.sh <target>            # full pipeline
-scripts/deploy/deploy.sh <target> --dry-run  # preflight + gate only
-scripts/deploy/deploy.sh <target> --no-gate  # emergency: bypass tests
-```
-
-## Exit Code Handling
-
-| Code | Meaning | Required action |
-|---|---|---|
-| 0 | Success | Report: target, commit hash, gate status, verify status, elapsed time |
-| 1 | Preflight failed | Fix the upstream issue (push commits, wake node, switch to master). Never bypass. |
-| 2 | Gate failed | Show exactly which test/build failed. Do **not** deploy. Fix the failure first. |
-| 3 | Execute failed | Show full deploy output. Ask user whether to investigate or rollback. |
-| 4 | Verify failed | Show docker ps output. Discuss rollback with the user. |
-| 5 | Sudo handoff | Print the exact manual command from stderr **verbatim** and stop. User must run it. |
-
-## Rules
-
-- Never pass `--no-gate` unless the user explicitly requests emergency/bypass mode.
-- Never deploy uncommitted or unpushed code — preflight enforces this; do not help circumvent it.
-- Canonical branch is `master` — preflight enforces this.
-- For exit 5: reproduce the handoff command exactly as printed to stderr, then stop.

.claude/skills/save-session/SKILL.md

@@ -1,65 +0,0 @@
----
-name: save-session
-description: Save and record the current work session to docs/sessions/. Trigger ONLY on explicit "save session", "zapisz sesję", or "wrap up" — never invoke proactively between tasks.
----
-
-**Trigger condition**: user explicitly says "save session", "zapisz sesję", "wrap up", or equivalent.
-Never invoke proactively. Never invoke mid-task.
-
-## 1. Determine Session Boundary
-
-1. Read the latest entry file in `docs/sessions/` — use its last `## Session HH:MM` heading timestamp as the start boundary.
-2. Fallback if no previous entry exists: 24 hours ago.
-
-## 2. Collect Facts (deterministic only — no invention)
-
-Run exactly:
-```bash
-# All commits since boundary
-git --no-pager log --oneline <boundary>..HEAD
-
-# Changed file summary
-git --no-pager diff --stat <boundary>..HEAD
-```
-
-From the visible conversation transcript: deploys run and their outcomes, test results seen.
-
-## 3. Write the Session Entry
-
-**APPEND** to `docs/sessions/YYYY-MM-DD.md` (create the file if it doesn't exist for today).
-Never overwrite existing content.
-
-```markdown
-## Session HH:MM
-
-### Commits
-<output of git log --oneline>
-
-### Files changed
-<output of git diff --stat>
-
-### Deploys
-<list from transcript, or "None recorded">
-
-### Narrative
-> _user-provided summary_
-```
-
-The `> _user-provided summary_` placeholder is **mandatory**. Never fill it in. The user supplies the narrative separately if desired.
-
-## 4. What NOT to Touch
-
-- `backlog.md` — only on explicit "update backlog" instruction
-- `CLAUDE.md` — only on explicit "update CLAUDE.md" instruction
-- Any other file not listed above
-
-## 5. Commit
-
-Stage and commit **only** the session file:
-
-```bash
-git add docs/sessions/YYYY-MM-DD.md
-git commit -m "docs: session YYYY-MM-DD HH:MM"
-```
-
-No other files. No `git add -A`.

.claude/skills/worktree-aware/SKILL.md

@@ -1,81 +0,0 @@
----
-name: worktree-aware
-description: >
-  Use when working in a git worktree checkout for a parallel agent task.
-  The presence of an .agent-task file in the current working directory indicates
-  a task worktree (NOT the main checkout). Encodes branch hygiene: commit only
-  to the assigned task branch, NEVER push origin master, NEVER touch the main
-  checkout at ~/homelab-codex-ws, NEVER manage worktrees yourself. On task
-  completion, report the branch name verbatim and stop — the human merges via
-  scripts/dev/agent.sh.
----
-
-## When this applies
-
-- `.agent-task` present in your `cwd` → you are in a task worktree. Apply all rules below.
-- `.agent-task` absent → you are in the main checkout. Do NOT treat yourself as a task agent.
-  In the main checkout these rules do not apply.
-
-## Reading the marker
-
-`.agent-task` is a YAML file. Your assigned branch is the value of the `branch:` key, e.g.:
-
-```yaml
-task: my-feature
-branch: task/my-feature
-parent_commit: abc1234
-created_utc: 2026-06-03T10:00:00Z
-worktree_path: /home/oskar/homelab-codex-ws-my-feature
-```
-
-Always read this file first before taking any action.
-
-## Rules
-
-1. **Commit only to your branch.**
-   Before any `git commit`, run `git status` and confirm it says `On branch task/<name>`.
-   If it does not, stop immediately and report the discrepancy.
-
-2. **Push only to your branch.**
-   The only permitted push is `git push origin task/<name>`.
-   NEVER `git push origin master` or any other branch.
-
-3. **Do not touch the main checkout.**
-   `~/homelab-codex-ws/` is the main checkout — deploy-only, owned by the human.
-   Do not read from, write to, or execute commands inside it.
-
-4. **Stay scoped.**
-   Only change files directly related to your assigned task.
-   If you notice other problems, report them in your final summary as separate follow-up proposals.
-   Do not fix them in this worktree.
-
-5. **Never `git add -A`.**
-   Always stage specific files by name: `git add path/to/file`.
-
-6. **Do not manage worktrees.**
-   Never run `git worktree add/remove` or invoke `scripts/dev/agent.sh`.
-   Worktree lifecycle is the human's responsibility.
-
-7. **Final report before stopping.**
-   When the task is done, provide a structured report containing:
-   - Files changed (path and one-line summary of change)
-   - Tests run and results
-   - All commit hashes on the task branch
-   - **Branch name verbatim** (copy-paste ready)
-   - Follow-up items as bulleted proposals for separate tasks
-
-## Definition of Done
-
-- All commits are on `task/<name>` (verify with `git log --oneline master..task/<name>`)
-- Test suite passes
-- Branch pushed: `git push origin task/<name>`
-- Full report delivered in conversation
-
-## What you do NOT do
-
-- Merge branches
-- Create or push tags
-- Run deploys or healthchecks against production nodes
-- Delete branches or worktrees
-- Modify files in other worktrees
-- Push to `origin master` under any circumstances

CLAUDE.md

@@ -180,15 +180,3 @@ Before any new or changed service is considered ready:
 - Services: kebab-case (`stability-agent`, `zigbee2mqtt`)
 - Container names must match service names
 - Always `restart: unless-stopped` unless `service.yaml` says otherwise
-
-## Multi-agent worktree mode
-
-`~/homelab-codex-ws` (main checkout) is **deploy-only** and belongs to the human operator.
-Parallel agent tasks run in isolated git worktrees created by `scripts/dev/agent.sh new <name>`.
-
-If `.agent-task` exists in your current working directory, you are in a task worktree.
-**You must immediately read `.agent-task` and load `.claude/skills/worktree-aware/SKILL.md`
-before taking any action.** That skill defines all branch-hygiene rules for task worktrees.
-
-Worktree lifecycle commands: `agent.sh new | list | merge | clean`.
-Agents never invoke these — only the human does.

hosts/vps/runtime/ai-cluster/docker-compose.override.yml

@@ -0,0 +1,33 @@
+# AI cluster memory limits — HARD caps, containers are OOM-killed and auto-restarted
+# by Docker rather than consuming host memory. ai-cluster is the primary OOM suspect
+# (unbounded Python workers, no limits since deployment).
+#
+# Architectural note: compute workloads here should migrate to SOLARIA (GPU node).
+# Until migration: contain the blast radius with per-container limits.
+#
+# Pre-cutover: service-ops-worker still mounts compose/env from old paths.
+# After cutover and git pull, these overrides are removed and base compose paths are used.
+
+services:
+  codex-worker:
+    mem_limit: 64m
+
+  openclaw:
+    mem_limit: 128m
+
+  planner-worker:
+    mem_limit: 64m
+
+  service-ops-worker:
+    mem_limit: 64m
+    # Pre-cutover: override bind mounts to keep pointing at old dockeruser paths
+    volumes:
+      - /home/dockeruser/docker/ai-cluster/docker-compose.yml:/app/docker-compose.yml:ro
+      - /home/dockeruser/docker/ai-cluster/.env:/app/.env:ro
+      - /var/run/docker.sock:/var/run/docker.sock:rw
+
+  redis:
+    mem_limit: 32m
+
+  mosquitto:
+    mem_limit: 32m

hosts/vps/runtime/joplin/docker-compose.override.yml

@@ -0,0 +1,6 @@
+services:
+  app:
+    mem_limit: 224m
+
+  db:
+    mem_limit: 128m

hosts/vps/runtime/node_exporter/docker-compose.override.yml

@@ -0,0 +1,3 @@
+services:
+  node_exporter:
+    mem_limit: 32m

hosts/vps/runtime/npm/docker-compose.override.yml

@@ -0,0 +1,6 @@
+services:
+  npm:
+    mem_limit: 160m
+    # Public ingress — elevated OOM protection so TLS termination + proxy host
+    # config survive memory pressure. Host OOM-killer will not target this container.
+    oom_score_adj: -800

hosts/vps/runtime/outline/docker-compose.override.yml

@@ -0,0 +1,9 @@
+services:
+  outline:
+    mem_limit: 512m
+
+  postgres:
+    mem_limit: 96m
+
+  redis:
+    mem_limit: 32m

hosts/vps/services.yaml

@@ -41,3 +41,81 @@ services:
     depends_on:
       local: []
       external: []
+
+  npm:
+    role: reverse-proxy-ingress
+    deployment_model: docker-compose
+    exposure: public
+    offline_required: false
+    depends_on:
+      local: []
+      external: []
+    ports:
+      - name: http
+        container_port: 80
+        protocol: tcp
+      - name: https
+        container_port: 443
+        protocol: tcp
+      - name: admin
+        container_port: 81
+        protocol: tcp
+    runtime:
+      data_path: /home/dockeruser/docker/npm/data
+      config_path: /opt/homelab/config/npm
+
+  outline:
+    role: team-wiki
+    deployment_model: docker-compose
+    exposure: public
+    offline_required: false
+    depends_on:
+      local:
+        - npm
+      external: []
+    ports:
+      - name: http
+        container_port: 3000
+        protocol: tcp
+    runtime:
+      config_path: /opt/homelab/config/outline
+
+  joplin:
+    role: note-sync-server
+    deployment_model: docker-compose
+    exposure: tailscale-internal
+    offline_required: false
+    depends_on:
+      local:
+        - npm
+      external: []
+    ports:
+      - name: http
+        container_port: 22300
+        bind: 127.0.0.1
+        protocol: tcp
+    runtime:
+      config_path: /opt/homelab/config/joplin
+
+  ai-cluster:
+    role: ai-worker-cluster
+    deployment_model: docker-compose
+    exposure: tailscale-internal
+    offline_required: false
+    depends_on:
+      local: []
+      external:
+        - piha:gateway
+    ports:
+      - name: openclaw-api
+        container_port: 8000
+        protocol: tcp
+      - name: mqtt
+        container_port: 1883
+        protocol: tcp
+        bind: tailscale
+    runtime:
+      config_path: /opt/homelab/config/ai-cluster
+    notes:
+      - "Local images must be built on VPS — not pulled from registry"
+      - "Compute workloads belong on SOLARIA; migrate when possible"

scripts/deploy/deploy.sh

@@ -1,321 +1,270 @@
 #!/usr/bin/env bash
-# scripts/deploy/deploy.sh — Saturn-side deploy dispatcher
+# deploy.sh - Staged deployment framework for homelab nodes.
-# Usage: deploy.sh <target> [--dry-run] [--no-gate]
-#   target ∈ {control-plane, vps, piha, solaria, chelsty-infra}
-# Exit codes: 0=ok  1=preflight  2=gate  3=execute  4=verify  5=handoff(sudo)
 
-set -uo pipefail
+set -o pipefail
 
-REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+# --- Configuration ---
-SSH_USER="${SSH_USER:-oskar}"
+export RUNTIME_PATH="/opt/homelab"
-START_TIME=$(date +%s)
+export STATE_DIR="${RUNTIME_PATH}/state/deploy"
-TARGET=""
+export LOG_DIR="${RUNTIME_PATH}/logs/deploy"
-DRY_RUN=false
+export REPO_PATH="${HOME}/homelab-codex-ws"
-NO_GATE=false
+export TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+export LOG_FILE="${LOG_DIR}/deploy_${TIMESTAMP}.log"
 
-usage() {
+# --- Initialization ---
-    cat >&2 <<'EOF'
+mkdir -p "$STATE_DIR" "$LOG_DIR"
-Usage: deploy.sh <target> [--dry-run] [--no-gate]
 
-Targets:
+# Redirection for logging
-  control-plane   observer/supervisor/executor/operator-ui on VPS
+exec > >(tee -a "$LOG_FILE") 2>&1
-  vps             all VPS GitOps services
-  piha            PIHA services
-  solaria         SOLARIA compute services
-  chelsty-infra   CHELSTY edge node (LTE, longer SSH timeout)
 
-Flags:
+# --- Load Libraries ---
-  --dry-run   run preflight + gate only; stop before deploy
+LIB_PATH="${REPO_PATH}/scripts/lib"
-  --no-gate   skip pytest + docker build (emergency only; logged as WARNING)
+source "${LIB_PATH}/log.sh"
+source "${LIB_PATH}/state.sh"
+source "${LIB_PATH}/inventory.sh"
+source "${LIB_PATH}/compose.sh"
+source "${LIB_PATH}/diagnostics.sh"
 
-Exit codes: 0=ok  1=preflight  2=gate  3=execute  4=verify  5=handoff(sudo)
+# --- CLI Parsing ---
-EOF
+TARGET_HOST=$(hostname)
-    exit 1
+TARGET_SERVICE=""
-}
+RESUME=false
+REQUESTED_STAGE=""
 
 while [[ $# -gt 0 ]]; do
     case $1 in
-        control-plane|vps|piha|solaria|chelsty-infra)
+        --host)
-            TARGET="$1"; shift ;;
+            TARGET_HOST="$2"
-        --dry-run)
+            shift 2
-            DRY_RUN=true; shift ;;
+            ;;
-        --no-gate)
+        --service)
-            NO_GATE=true; shift ;;
+            TARGET_SERVICE="$2"
-        -h|--help)
+            shift 2
-            usage ;;
+            ;;
+        --resume)
+            RESUME=true
+            shift
+            ;;
+        --stage)
+            REQUESTED_STAGE="$2"
+            shift 2
+            ;;
         *)
-            echo "Unknown argument: $1" >&2
+            if [[ "$1" =~ ^(prepare|validate|deploy|verify|diagnose|complete)$ ]]; then
-            usage ;;
+                REQUESTED_STAGE="$1"
+            fi
+            shift
+            ;;
     esac
 done
 
-[[ -z "$TARGET" ]] && { echo "Error: target is required." >&2; usage; }
+# --- Stages ---
 
-case "$TARGET" in
+stage_prepare() {
-    control-plane) SSH_HOST="vps" ;;
+    local host=$1
-    *)             SSH_HOST="$TARGET" ;;
+    if is_stage_complete "prepare" && [[ "$RESUME" == "true" ]]; then
-esac
+        log "INFO" "Skipping PREPARE (already complete)"
-
+        return 0
-case "$TARGET" in
-    chelsty-*) SSH_TIMEOUT=30 ;;
-    *)         SSH_TIMEOUT=5 ;;
-esac
-
-# ── PREFLIGHT ────────────────────────────────────────────────────────────────
-
-preflight() {
-    echo "=== PREFLIGHT ==="
-
-    local branch
-    branch=$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)
-    if [[ "$branch" != "master" ]]; then
-        echo "ERROR: On branch '${branch}', not master. Switch to master and push first." >&2
-        exit 1
     fi
-    echo "[ok] branch: master"
 
-    if ! git -C "$REPO_ROOT" diff --quiet; then
+    log "INFO" "Stage: PREPARE ($host)"
-        echo "ERROR: Unstaged changes in working tree. Commit or stash before deploying." >&2
+    set_stage "prepare"
-        exit 1
+    
-    fi
+    emit_event "deployment_started" "info" "deploy.sh" "all" "${TIMESTAMP}" "{\"stage\": \"prepare\"}"
-    if ! git -C "$REPO_ROOT" diff --cached --quiet; then
-        echo "ERROR: Staged but uncommitted changes. Commit before deploying." >&2
-        exit 1
-    fi
-    echo "[ok] working tree clean"
 
-    git -C "$REPO_ROOT" fetch origin master --quiet
+    cd "$REPO_PATH" || exit 1
-    local unpushed
+    log "INFO" "Pulling latest changes..."
-    unpushed=$(git -C "$REPO_ROOT" log origin/master..HEAD --oneline)
+    if ! git pull; then
-    if [[ -n "$unpushed" ]]; then
+        log "WARN" "Git pull failed, proceeding with local state (offline mode or network flap)"
-        echo "ERROR: Unpushed commits on master:" >&2
-        echo "$unpushed" >&2
-        echo "Push first:  git push origin master" >&2
-        exit 1
     fi
-    echo "[ok] no unpushed commits"
 
-    echo "Checking SSH: ${SSH_USER}@${SSH_HOST} (ConnectTimeout=${SSH_TIMEOUT}s)..."
+    # Ensure runtime directories exist
-    if ! ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
+    mkdir -p "${RUNTIME_PATH}/config" "${RUNTIME_PATH}/data" "${RUNTIME_PATH}/state" "${RUNTIME_PATH}/logs"
-            "${SSH_USER}@${SSH_HOST}" true 2>/dev/null; then
+
-        echo "ERROR: Cannot reach ${SSH_HOST} via SSH (timeout ${SSH_TIMEOUT}s)." >&2
+    struct_log "prepare" "$host" "all" "success" "repo_updated"
-        exit 1
+    mark_stage_complete "prepare"
-    fi
-    echo "[ok] ${SSH_HOST} reachable"
 }
 
-# ── GATE ─────────────────────────────────────────────────────────────────────
+stage_validate() {
-
+    local host=$1
-gate() {
+    if is_stage_complete "validate" && [[ "$RESUME" == "true" ]]; then
-    if [[ "$NO_GATE" == "true" ]]; then
+        log "INFO" "Skipping VALIDATE (already complete)"
-        echo "=== GATE: SKIPPED ==="
-        echo "WARNING: --no-gate active — pytest + docker build bypassed (emergency mode)." >&2
         return 0
     fi
 
-    echo "=== GATE ==="
+    log "INFO" "Stage: VALIDATE ($host)"
+    set_stage "validate"
 
-    local services=()
+    for service in "${SERVICES[@]}"; do
-
+        log "INFO" "Validating $service..."
-    if [[ "$TARGET" == "control-plane" ]]; then
+        if [[ ! -d "${REPO_PATH}/services/$service" ]]; then
-        services=("control-plane")
+            log "ERROR" "Service definition not found: $service"
-    else
+            struct_log "validate" "$host" "$service" "fail" "not_found"
-        local svc_yaml="${REPO_ROOT}/hosts/${TARGET}/services.yaml"
+            return 1
-        if [[ ! -f "$svc_yaml" ]]; then
-            echo "ERROR: ${svc_yaml} not found." >&2
-            exit 2
-        fi
-        local svc_list
-        svc_list=$(python3 -c "
-import yaml
-with open('${svc_yaml}') as f:
-    data = yaml.safe_load(f)
-svcs = data.get('services', {})
-if isinstance(svcs, dict):
-    print('\n'.join(svcs.keys()))
-elif isinstance(svcs, list):
-    print('\n'.join(svcs))
-")
-        while IFS= read -r svc; do
-            [[ -z "$svc" ]] && continue
-            if [[ -f "${REPO_ROOT}/services/${svc}/Dockerfile" ]]; then
-                services+=("$svc")
-            fi
-        done <<< "$svc_list"
-    fi
-
-    if [[ ${#services[@]} -eq 0 ]]; then
-        echo "[info] No services with local Dockerfile found for ${TARGET} — gate trivially passes."
-        return 0
-    fi
-
-    echo "Services under gate: ${services[*]}"
-    local gate_failed=false
-
-    for svc in "${services[@]}"; do
-        local svc_dir="${REPO_ROOT}/services/${svc}"
-
-        if [[ -d "${svc_dir}/tests" ]]; then
-            echo "--- pytest: ${svc} ---"
-            if ! python3 -m pytest "${svc_dir}/tests" -q; then
-                echo "GATE FAIL: pytest failed for ${svc}" >&2
-                gate_failed=true
-            fi
-        fi
-
-        echo "--- docker build: ${svc} ---"
-        if ! docker build --quiet "${svc_dir}" >/dev/null; then
-            echo "GATE FAIL: docker build failed for ${svc}" >&2
-            gate_failed=true
         fi
     done
 
-    if [[ "$gate_failed" == "true" ]]; then
+    struct_log "validate" "$host" "all" "success" "validated"
-        exit 2
+    mark_stage_complete "validate"
-    fi
-    echo "[ok] gate passed"
 }
 
-# ── EXECUTE ──────────────────────────────────────────────────────────────────
+stage_deploy() {
-
+    local host=$1
-execute() {
+    if is_stage_complete "deploy" && [[ "$RESUME" == "true" ]]; then
-    echo "=== EXECUTE ==="
+        log "INFO" "Skipping DEPLOY (already complete)"
-
+        return 0
-    local cmd_output
-    local cmd_exit=0
-
-    if [[ "$TARGET" == "control-plane" ]]; then
-        echo "Running deploy-control-plane.sh --ssh..."
-        cmd_output=$("${REPO_ROOT}/scripts/deploy/deploy-control-plane.sh" --ssh 2>&1) \
-            || cmd_exit=$?
-    else
-        echo "SSHing to ${SSH_HOST}: git pull + deploy-node.sh..."
-        cmd_output=$(ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
-            "${SSH_USER}@${SSH_HOST}" \
-            'cd ~/homelab-codex-ws && git pull && ./scripts/deploy/deploy-node.sh' 2>&1) \
-            || cmd_exit=$?
     fi
 
-    echo "$cmd_output"
+    log "INFO" "Stage: DEPLOY ($host)"
+    set_stage "deploy"
 
-    if echo "$cmd_output" | grep -qF "[sudo] password"; then
+    local last_s=$(get_last_service)
-        echo "" >&2
+    local skip=false
-        echo "ERROR (exit 5): Deploy hit an interactive sudo prompt." >&2
+    if [[ "$RESUME" == "true" && -n "$last_s" ]]; then
-        echo "Run manually:" >&2
+        skip=true
-        if [[ "$TARGET" == "control-plane" ]]; then
-            echo "  ssh -t ${SSH_USER}@${SSH_HOST} 'cd ~/homelab-codex-ws && git pull origin master && cd services/control-plane && bash deploy-local.sh'" >&2
-        else
-            echo "  ssh -t ${SSH_USER}@${SSH_HOST} 'cd ~/homelab-codex-ws && git pull && ./scripts/deploy/deploy-node.sh'" >&2
-        fi
-        exit 5
     fi
 
-    if [[ $cmd_exit -ne 0 ]]; then
+    for service in "${SERVICES[@]}"; do
-        echo "ERROR: Deploy command exited ${cmd_exit}." >&2
+        if [[ "$skip" == "true" ]]; then
-        exit 3
+            if [[ "$service" == "$last_s" ]]; then
-    fi
+                skip=false
-
+                log "INFO" "Resuming from $service..."
-    echo "[ok] execute completed"
+            else
-}
+                log "INFO" "Skipping $service (already processed)"
-
+                continue
-# ── VERIFY ───────────────────────────────────────────────────────────────────
-
-verify() {
-    echo "=== VERIFY ==="
-
-    local ps_output
-    local ps_exit=0
-    ps_output=$(ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
-        "${SSH_USER}@${SSH_HOST}" \
-        'docker ps --format "{{.Names}}\t{{.Status}}"' 2>&1) \
-        || ps_exit=$?
-
-    if [[ $ps_exit -ne 0 ]]; then
-        echo "ERROR: docker ps failed on ${SSH_HOST}:" >&2
-        echo "$ps_output" >&2
-        exit 4
-    fi
-
-    echo "$ps_output"
-
-    local failed=false
-
-    local not_up
-    not_up=$(echo "$ps_output" | grep -v '^$' | grep -v $'\tUp' || true)
-    if [[ -n "$not_up" ]]; then
-        echo "ERROR: Containers not in Up state:" >&2
-        echo "$not_up" >&2
-        failed=true
-    fi
-
-    local unhealthy
-    unhealthy=$(echo "$ps_output" | grep '(unhealthy)' || true)
-    if [[ -n "$unhealthy" ]]; then
-        echo "ERROR: Unhealthy containers:" >&2
-        echo "$unhealthy" >&2
-        failed=true
-    fi
-
-    if [[ "$TARGET" == "control-plane" ]]; then
-        for cp_svc in supervisor observer executor operator-ui; do
-            if ! echo "$ps_output" | grep -q "$cp_svc"; then
-                echo "ERROR: control-plane component absent from docker ps: ${cp_svc}" >&2
-                failed=true
             fi
-        done
+        fi
-    fi
 
-    if [[ "$failed" == "true" ]]; then
+        log "INFO" "Deploying $service..."
-        echo "" >&2
+        set_last_service "$service"
-        echo "Full docker ps output above." >&2
-        exit 4
-    fi
 
-    echo "[ok] all containers healthy"
+        if ! run_compose_up "$service"; then
+            struct_log "deploy" "$host" "$service" "fail" "docker_compose_failed"
+            collect_diagnostics "$host" "$service"
+            return 1
+        fi
+
+        struct_log "deploy" "$host" "$service" "success" "deployed"
+    done
+    
+    set_last_service ""
+    mark_stage_complete "deploy"
 }
 
-# ── REPORT ───────────────────────────────────────────────────────────────────
+stage_verify() {
-
+    local host=$1
-report() {
+    if is_stage_complete "verify" && [[ "$RESUME" == "true" ]]; then
-    local mode="${1:-deploy}"
+        log "INFO" "Skipping VERIFY (already complete)"
-    local end_time
+        return 0
-    end_time=$(date +%s)
-    local elapsed
-    elapsed=$(( end_time - START_TIME ))
-    local commit_hash
-    commit_hash=$(git -C "$REPO_ROOT" rev-parse --short HEAD)
-    local gate_s verify_s
-
-    if [[ "$NO_GATE" == "true" ]]; then
-        gate_s="skip"
-    else
-        gate_s="ok"
     fi
 
-    if [[ "$mode" == "dry-run" ]]; then
+    log "INFO" "Stage: VERIFY ($host)"
-        verify_s="skip(dry-run)"
+    set_stage "verify"
-    else
-        verify_s="green"
-    fi
 
-    echo ""
+    for service in "${SERVICES[@]}"; do
-    if [[ "$mode" == "dry-run" ]]; then
+        log "INFO" "Verifying $service..."
-        echo "DRY RUN OK | target=${TARGET} | commit=${commit_hash} | gate=${gate_s} | verify=${verify_s} | ${elapsed}s"
+        local health_script="${REPO_PATH}/services/${service}/healthcheck.sh"
-    else
+        if [[ -f "$health_script" ]]; then
-        echo "DEPLOY OK  | target=${TARGET} | commit=${commit_hash} | gate=${gate_s} | verify=${verify_s} | ${elapsed}s"
+            if ! bash "$health_script"; then
-    fi
+                log "ERROR" "Healthcheck failed for $service"
+                struct_log "verify" "$host" "$service" "fail" "healthcheck_failed"
+                collect_diagnostics "$host" "$service"
+                return 1
+            fi
+        else
+            # Generic check if container is running
+            if ! docker ps --filter "name=$service" --filter "status=running" | grep -q "$service"; then
+                log "ERROR" "Container $service is not running"
+                struct_log "verify" "$host" "$service" "fail" "container_not_running"
+                collect_diagnostics "$host" "$service"
+                return 1
+            fi
+        fi
+        struct_log "verify" "$host" "$service" "success" "verified"
+    done
+    mark_stage_complete "verify"
 }
 
-# ── MAIN ─────────────────────────────────────────────────────────────────────
+stage_complete() {
+    local host=$1
+    log "INFO" "Stage: COMPLETE ($host)"
+    set_stage "complete"
+    struct_log "complete" "$host" "all" "success" "deployment_finished"
+    clear_deployment_state
+}
 
-preflight
+# --- Execution Logic ---
-gate
 
-if [[ "$DRY_RUN" == "true" ]]; then
+run_deployment() {
-    report dry-run
+    local start_stage=$1
-    exit 0
+
+    # Sequential execution from start_stage
+    case "$start_stage" in
+        prepare)
+            stage_prepare "$TARGET_HOST" || return 1
+            ;&
+        validate)
+            stage_validate "$TARGET_HOST" || return 1
+            ;&
+        deploy)
+            stage_deploy "$TARGET_HOST" || return 1
+            ;&
+        verify)
+            stage_verify "$TARGET_HOST" || return 1
+            ;&
+        complete)
+            stage_complete "$TARGET_HOST" || return 1
+            ;;
+        *)
+            log "ERROR" "Invalid stage: $start_stage"
+            return 1
+            ;;
+    esac
+}
+
+# --- Main ---
+
+log "INFO" "--- Homelab Deployment Started (Host: $TARGET_HOST, Service: ${TARGET_SERVICE:-all}) ---"
+
+if ! load_inventory "$TARGET_HOST" "$TARGET_SERVICE"; then
+    log "ERROR" "Failed to load inventory"
+    exit 1
 fi
 
-execute
+EXIT_STATUS=0
-verify
+if [[ "$RESUME" == "true" ]]; then
-report
+    CURRENT=$(get_stage)
+    log "INFO" "Resuming from state: $CURRENT"
+    case "$CURRENT" in
+        prepare|validate|deploy|verify)
+            run_deployment "$CURRENT" || EXIT_STATUS=1
+            ;;
+        complete|none)
+            log "INFO" "No interrupted deployment found. Starting from scratch..."
+            run_deployment "prepare" || EXIT_STATUS=1
+            ;;
+        *)
+            log "INFO" "Unknown state. Starting from prepare..."
+            run_deployment "prepare" || EXIT_STATUS=1
+            ;;
+    esac
+elif [[ -n "$REQUESTED_STAGE" ]]; then
+    if [[ "$REQUESTED_STAGE" == "diagnose" ]]; then
+        collect_diagnostics "$TARGET_HOST" "$TARGET_SERVICE"
+    else
+        run_deployment "$REQUESTED_STAGE" || EXIT_STATUS=1
+    fi
+else
+    # New deployment - clear previous state
+    clear_deployment_state
+    run_deployment "prepare" || EXIT_STATUS=1
+fi
+
+if [[ $EXIT_STATUS -eq 0 ]]; then
+    print_summary "$TARGET_HOST" "SUCCESS"
+    log "INFO" "--- Homelab Deployment Finished Successfully ---"
+else
+    print_summary "$TARGET_HOST" "FAILED"
+    log "ERROR" "--- Homelab Deployment Failed ---"
+    exit 1
+fi

scripts/dev/agent.sh

@@ -1,361 +0,0 @@
-#!/usr/bin/env bash
-# Multi-agent worktree manager.
-# EXIT: 0 ok, 1 preflight, 2 operation failed.
-set -euo pipefail
-
-trap 'echo "agent.sh: failed at line $LINENO (exit $?)" >&2' ERR
-
-RESERVED_NAMES=(master main HEAD list merge clean new)
-MAX_WORKTREES=4
-
-die()    { echo "ERROR: $*" >&2; exit "${2:-2}"; }
-prefail(){ echo "PREFLIGHT: $*" >&2; exit 1; }
-
-# ── helpers ──────────────────────────────────────────────────────────────────
-
-is_main_checkout() {
-  local git_dir common_dir
-  git_dir=$(git rev-parse --git-dir 2>/dev/null) || return 1
-  common_dir=$(git rev-parse --git-common-dir 2>/dev/null) || return 1
-  [ "$git_dir" = "$common_dir" ]
-}
-
-require_main_checkout() {
-  is_main_checkout || prefail "must run from the main checkout, not a worktree"
-}
-
-require_master_branch() {
-  local branch
-  branch=$(git rev-parse --abbrev-ref HEAD)
-  [ "$branch" = "master" ] || prefail "must be on master (currently on '$branch')"
-}
-
-require_clean_tree() {
-  local dirty
-  dirty=$(git status --porcelain)
-  [ -z "$dirty" ] || prefail "working tree is not clean — stash or commit first"
-}
-
-worktree_paths() {
-  # list worktree paths (excluding main); || true prevents grep exit-1 when empty
-  local main_path
-  main_path=$(git rev-parse --show-toplevel)
-  git worktree list --porcelain \
-    | awk '/^worktree /{p=$2} /^$/{print p}' \
-    | grep -v "^${main_path}$" \
-    || true
-}
-
-worktree_count() {
-  worktree_paths | wc -l
-}
-
-branch_exists_local()  { git show-ref --verify --quiet "refs/heads/$1"; }
-branch_exists_remote() { git ls-remote --exit-code origin "$1" >/dev/null 2>&1; }
-
-utc_now() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }
-
-age_str() {
-  local created_utc="$1"
-  local now_ts created_ts diff_s
-  now_ts=$(date -u +%s)
-  # strip Z, replace T with space for `date -d`
-  created_ts=$(date -u -d "${created_utc//T/ }" +%s 2>/dev/null) || { echo "?"; return; }
-  diff_s=$(( now_ts - created_ts ))
-  if   (( diff_s < 60 ));   then echo "${diff_s}s"
-  elif (( diff_s < 3600 )); then echo "$(( diff_s/60 ))m"
-  elif (( diff_s < 86400 )); then echo "$(( diff_s/3600 ))h"
-  else echo "$(( diff_s/86400 ))d"
-  fi
-}
-
-validate_name() {
-  local name="$1"
-  if ! [[ "$name" =~ ^[a-z][a-z0-9-]*$ ]]; then
-    prefail "name '$name' must match ^[a-z][a-z0-9-]*$"
-  fi
-  for r in "${RESERVED_NAMES[@]}"; do
-    if [ "$name" = "$r" ]; then
-      prefail "'$name' is a reserved word"
-    fi
-  done
-}
-
-# ── subcommands ───────────────────────────────────────────────────────────────
-
-cmd_new() {
-  local name="${1:-}"
-  [ -n "$name" ] || { usage; exit 1; }
-
-  validate_name "$name"
-  require_main_checkout
-  require_master_branch
-  require_clean_tree
-
-  # worktree limit
-  local count
-  count=$(worktree_count)
-  if (( count >= MAX_WORKTREES )); then
-    echo "ERROR: already at maximum of $MAX_WORKTREES active worktrees:" >&2
-    cmd_list
-    exit 1
-  fi
-
-  # branch collision
-  if branch_exists_local "task/$name"; then
-    prefail "branch task/$name already exists locally"
-  fi
-  git fetch origin master --quiet
-  if branch_exists_remote "refs/heads/task/$name"; then
-    prefail "branch task/$name already exists on origin"
-  fi
-
-  # directory collision
-  local main_path wt_path
-  main_path=$(git rev-parse --show-toplevel)
-  wt_path="$(dirname "$main_path")/homelab-codex-ws-${name}"
-  [ ! -e "$wt_path" ] || prefail "directory $wt_path already exists"
-
-  # create worktree
-  git worktree add -b "task/$name" "$wt_path" origin/master \
-    || die "git worktree add failed"
-
-  # write marker
-  local parent_commit
-  parent_commit=$(git rev-parse origin/master)
-  cat > "$wt_path/.agent-task" <<EOF
-task: $name
-branch: task/$name
-parent_commit: $parent_commit
-created_utc: $(utc_now)
-worktree_path: $wt_path
-EOF
-
-  echo ""
-  echo "Worktree created: $wt_path"
-  echo "Branch:           task/$name"
-  echo ""
-  echo "── Start Claude Code in this worktree ──────────────────────────────────────"
-  echo "cd ~/homelab-codex-ws-${name} && claude --dangerously-skip-permissions \"Jesteś w worktree task '${name}' (branch task/${name}). NAJPIERW przeczytaj .agent-task i .claude/skills/worktree-aware/SKILL.md, dopiero potem zacznij pracę. Commituj wyłącznie na swoją gałąź; nie pushuj origin master.\""
-  echo "─────────────────────────────────────────────────────────────────────────────"
-}
-
-cmd_list() {
-  local main_path
-  main_path=$(git rev-parse --show-toplevel)
-
-  # fetch to get up-to-date ahead/behind
-  git fetch origin master --quiet 2>/dev/null || true
-
-  local paths
-  paths=$(worktree_paths)
-
-  if [ -z "$paths" ]; then
-    echo "(no active task worktrees)"
-    return
-  fi
-
-  printf "%-20s %-25s %-10s %-8s %-8s %-7s %s\n" \
-    "NAME" "BRANCH" "CREATED" "AGE" "STATUS" "A/B" "PARENT"
-
-  while IFS= read -r wt_path; do
-    [ -z "$wt_path" ] && continue
-
-    local marker="$wt_path/.agent-task"
-    local task_name branch parent_commit created_utc
-    if [ -f "$marker" ]; then
-      task_name=$(  grep '^task:'          "$marker" | awk '{print $2}')
-      branch=$(     grep '^branch:'        "$marker" | awk '{print $2}')
-      parent_commit=$(grep '^parent_commit:' "$marker" | awk '{print $2}')
-      created_utc=$(grep '^created_utc:'   "$marker" | awk '{print $2}')
-    else
-      task_name="(no marker)"
-      branch=$(git -C "$wt_path" rev-parse --abbrev-ref HEAD 2>/dev/null || echo "?")
-      parent_commit="?"
-      created_utc=""
-    fi
-
-    local status="clean"
-    local dirty
-    dirty=$(git -C "$wt_path" status --porcelain 2>/dev/null || echo "?")
-    [ -n "$dirty" ] && status="dirty"
-
-    local ahead behind ab
-    ahead=$(git -C "$wt_path" rev-list --count "origin/master..${branch}" 2>/dev/null || echo "?")
-    behind=$(git -C "$wt_path" rev-list --count "${branch}..origin/master" 2>/dev/null || echo "?")
-    ab="+${ahead}/-${behind}"
-
-    local age=""
-    [ -n "$created_utc" ] && age=$(age_str "$created_utc")
-
-    local short_parent="${parent_commit:0:7}"
-    local short_created="${created_utc:0:10}"
-
-    printf "%-20s %-25s %-10s %-8s %-8s %-7s %s\n" \
-      "$task_name" "$branch" "$short_created" "$age" "$status" "$ab" "$short_parent"
-  done <<< "$paths"
-}
-
-cmd_merge() {
-  local name="${1:-}"
-  [ -n "$name" ] || { usage; exit 1; }
-
-  require_main_checkout
-  require_master_branch
-  require_clean_tree
-
-  git fetch origin --quiet
-
-  branch_exists_local "task/$name" || die "branch task/$name not found locally" 1
-
-  local main_path wt_path
-  main_path=$(git rev-parse --show-toplevel)
-  wt_path="$(dirname "$main_path")/homelab-codex-ws-${name}"
-
-  # attempt ff-only merge
-  local merge_failed=0
-  git merge --ff-only "task/$name" || merge_failed=1
-
-  if (( merge_failed )); then
-    # abort any partial merge state
-    git merge --abort 2>/dev/null || true
-    echo ""
-    echo "ERROR: task/$name cannot be fast-forwarded into master." >&2
-    echo "       The branch has likely diverged from master." >&2
-    echo "" >&2
-    echo "Diagnose with:" >&2
-    echo "  git log master..task/$name        # commits only on task branch" >&2
-    echo "  git log task/$name..master        # commits master has that task doesn't" >&2
-    echo "" >&2
-    echo "Then decide: rebase task/$name onto master, or merge manually." >&2
-    echo "Worktree and branch are preserved — no changes made." >&2
-    exit 2
-  fi
-
-  echo "Merged task/$name into master (fast-forward)."
-
-  git push origin master || die "git push origin master failed"
-  echo "Pushed master to origin."
-
-  if [ -d "$wt_path" ]; then
-    git worktree remove "$wt_path" || die "git worktree remove $wt_path failed"
-    echo "Removed worktree: $wt_path"
-  else
-    echo "(worktree directory $wt_path not found — skipping worktree remove)"
-  fi
-
-  git branch -d "task/$name" || die "git branch -d task/$name failed"
-  echo "Deleted local branch task/$name."
-
-  git push origin --delete "task/$name" 2>/dev/null \
-    && echo "Deleted remote branch task/$name." \
-    || echo "(remote branch task/$name not found — nothing to delete)"
-
-  echo ""
-  echo "Done. task/$name merged and cleaned up."
-}
-
-cmd_clean() {
-  local main_path
-  main_path=$(git rev-parse --show-toplevel)
-  git fetch origin --quiet 2>/dev/null || true
-
-  local to_remove=()
-
-  # orphaned registered worktrees: branch deleted or fully merged into master
-  local paths
-  paths=$(worktree_paths)
-  while IFS= read -r wt_path; do
-    [ -z "$wt_path" ] && continue
-    local branch
-    branch=$(git -C "$wt_path" rev-parse --abbrev-ref HEAD 2>/dev/null || echo "")
-    [ -z "$branch" ] && { to_remove+=("worktree:$wt_path (unreadable branch)"); continue; }
-
-    # branch gone locally?
-    if ! branch_exists_local "$branch"; then
-      to_remove+=("worktree:$wt_path (branch $branch no longer exists)")
-      continue
-    fi
-
-    # branch fully merged into master?
-    local ahead
-    ahead=$(git rev-list --count "origin/master..${branch}" 2>/dev/null || echo "1")
-    if [ "$ahead" = "0" ]; then
-      to_remove+=("worktree:$wt_path (branch $branch fully merged into origin/master)")
-    fi
-  done <<< "$paths"
-
-  # dangling directories: ../homelab-codex-ws-* not registered
-  local registered_paths
-  registered_paths=$(git worktree list --porcelain | awk '/^worktree /{print $2}')
-  local parent_dir
-  parent_dir=$(dirname "$main_path")
-  while IFS= read -r candidate; do
-    [ -d "$candidate" ] || continue
-    if ! echo "$registered_paths" | grep -qF "$candidate"; then
-      to_remove+=("dangling:$candidate")
-    fi
-  done < <(find "$parent_dir" -maxdepth 1 -name "homelab-codex-ws-*" -type d 2>/dev/null)
-
-  if [ ${#to_remove[@]} -eq 0 ]; then
-    echo "Nothing to clean."
-    return 0
-  fi
-
-  echo "Found ${#to_remove[@]} item(s) to clean:"
-  for entry in "${to_remove[@]}"; do
-    echo "  $entry"
-  done
-  echo ""
-
-  local overall_rc=0
-  for entry in "${to_remove[@]}"; do
-    local kind="${entry%%:*}"
-    local path="${entry#*:}"
-    # strip trailing annotation in parens
-    local raw_path
-    raw_path="${path%% (*}"
-
-    local confirm
-    read -r -p "Remove $kind '$raw_path'? [y/N] " confirm
-    if [[ "$confirm" =~ ^[Yy]$ ]]; then
-      if [ "$kind" = "worktree" ]; then
-        git worktree remove --force "$raw_path" 2>/dev/null \
-          || { echo "  WARNING: git worktree remove failed, trying rm -rf"; rm -rf "$raw_path" || true; }
-      else
-        rm -rf "$raw_path"
-      fi
-      echo "  Removed."
-    else
-      echo "  Skipped."
-    fi
-  done
-
-  return $overall_rc
-}
-
-usage() {
-  cat <<'EOF'
-Usage: agent.sh <subcommand> [args]
-
-  agent.sh new <name>    Create a new task worktree (branch task/<name>)
-  agent.sh list          List active task worktrees with status
-  agent.sh merge <name>  Fast-forward merge task/<name> into master and clean up
-  agent.sh clean         Remove orphaned or dangling worktrees (interactive)
-
-EXIT: 0 ok, 1 preflight, 2 operation failed.
-EOF
-}
-
-# ── dispatch ──────────────────────────────────────────────────────────────────
-
-SUBCOMMAND="${1:-}"
-shift || true
-
-case "$SUBCOMMAND" in
-  new)   cmd_new   "$@" ;;
-  list)  cmd_list  "$@" ;;
-  merge) cmd_merge "$@" ;;
-  clean) cmd_clean "$@" ;;
-  *)     usage; exit 1  ;;
-esac

scripts/observer/observer.py

@@ -7,34 +7,6 @@ import yaml
 from datetime import datetime, timezone
 from pathlib import Path
 
-
-def _atomic_write_json(path: Path, data) -> None:
-    """Write JSON atomically: write to a sibling .tmp, fsync, then os.replace."""
-    tmp = path.with_suffix(".tmp")
-    with open(tmp, "w") as f:
-        json.dump(data, f, indent=2)
-        f.flush()
-        os.fsync(f.fileno())
-    os.replace(tmp, path)
-
-
-def _parse_ts(ts) -> float:
-    """Return a Unix timestamp float from ts, which may be int/float or an ISO-8601 string.
-
-    Events from node-agent use int(time.time()); events from stability-agent / events.py
-    use ISO format ('2026-06-03T10:30:00Z').  Both appear in incident fields such as
-    last_occurrence and resolved_at, so any arithmetic on them must go through here.
-    Returns 0.0 on None or unparseable input so callers can use plain comparisons.
-    """
-    if ts is None:
-        return 0.0
-    if isinstance(ts, (int, float)):
-        return float(ts)
-    try:
-        return datetime.fromisoformat(str(ts).replace("Z", "+00:00")).timestamp()
-    except Exception:
-        return 0.0
-
 # Constants and Paths
 RUNTIME_PATH = os.getenv("RUNTIME_PATH", "/opt/homelab")
 EVENTS_DIR = Path(RUNTIME_PATH) / "events"
@@ -152,7 +124,8 @@ class Observer:
 
     def _save_checkpoint(self):
         try:
-            _atomic_write_json(OBSERVER_STATE_FILE, {"node_checkpoints": self.node_checkpoints})
+            with open(OBSERVER_STATE_FILE, "w") as f:
+                json.dump({"node_checkpoints": self.node_checkpoints}, f, indent=2)
         except Exception as e:
             logger.error(f"Failed to save checkpoint: {e}")
 
@@ -200,68 +173,12 @@ class Observer:
             logger.info(f"Pruning ghost (hash-prefixed) service key from world state: {k}")
             del self.world_state["services"][k]
 
-        now = time.time()
-
-        try:
-            # Collect incident_ids currently referenced by any service entry.
-            linked_ids: set = {
-                svc.get("incident_id")
-                for svc in self.world_state["services"].values()
-                if svc.get("incident_id")
-            }
-
-            # Case 1 — service is healthy but still points at an active incident.
-            # process_event already calls _resolve_incident on service_healthy events,
-            # but if the observer restarted with on-disk state where the link was
-            # intact (inconsistency from a pre-atomic-write crash), it may not get
-            # resolved until the next service_healthy event is processed.  Resolve
-            # immediately — a healthy service cannot have an ongoing incident.
-            for svc_key, svc in self.world_state["services"].items():
-                if svc.get("status") != "healthy":
-                    continue
-                inc_id = svc.get("incident_id")
-                if not inc_id:
-                    continue
-                inc = self.world_state["incidents"].get(inc_id, {})
-                if inc.get("status") == "active":
-                    logger.info(
-                        f"Auto-resolving incident {inc_id} for {svc_key}: "
-                        f"service is healthy"
-                    )
-                    inc["status"] = "resolved"
-                    inc["resolved_at"] = now
-                    svc["incident_id"] = None
-                    linked_ids.discard(inc_id)
-
-            # Case 2 — orphaned active incident: no service entry links to it and
-            # last_occurrence is older than 5 minutes (guard against creation races).
-            # These are the stale records left behind when on-disk state was
-            # inconsistent: the service entry had incident_id cleared but incidents.json
-            # still had the record as "active".
-            for inc_id, inc in self.world_state["incidents"].items():
-                if inc.get("status") != "active":
-                    continue
-                if inc_id in linked_ids:
-                    continue
-                age = now - _parse_ts(inc.get("last_occurrence"))
-                if age > 300:  # 5-minute guard
-                    logger.info(
-                        f"Auto-resolving orphaned incident {inc_id} "
-                        f"(service={inc.get('service')}, node={inc.get('node')}): "
-                        f"no service references it, age={int(age)}s"
-                    )
-                    inc["status"] = "resolved"
-                    inc["resolved_at"] = now
-
-        except Exception as exc:
-            logger.error(f"Error during incident auto-resolve in _prune_stale_world: {exc}")
-
         # Remove resolved incidents older than 7 days.
-        # Use _parse_ts so ISO-string resolved_at values are handled correctly.
+        now = time.time()
         stale_incidents = [
             k for k, v in self.world_state["incidents"].items()
             if v.get("status") == "resolved"
-            and now - _parse_ts(v.get("resolved_at")) > 7 * 86400
+            and (now - (v.get("resolved_at") or now)) > 7 * 86400
         ]
         for k in stale_incidents:
             del self.world_state["incidents"][k]
@@ -285,12 +202,13 @@ class Observer:
             "services.json": self.world_state["services"],
             "deployments.json": self.world_state["deployments"],
             "incidents.json": self.world_state["incidents"],
-            "recommendations.json": [],
+            "recommendations.json": [], # Placeholder to satisfy requirements
             "runtime-summary.json": self.world_state["summary"]
         }
         for filename, data in files.items():
             try:
-                _atomic_write_json(WORLD_DIR / filename, data)
+                with open(WORLD_DIR / filename, "w") as f:
+                    json.dump(data, f, indent=2)
             except Exception as e:
                 logger.error(f"Failed to save {filename}: {e}")
 

services/ai-cluster/docker-compose.yml

@@ -0,0 +1,110 @@
+services:
+  codex-worker:
+    image: ai-cluster-codex-worker
+    restart: unless-stopped
+    environment:
+      - AGENT_ID=vps-dev-1
+      - ROLE=dev
+      - MQTT_HOST=mosquitto
+      - MQTT_PORT=1883
+      - MQTT_USERNAME=${MQTT_USERNAME:-codex}
+      - MQTT_PASSWORD=${MQTT_PASSWORD}
+      - GATEWAY_BASE_URL=${GATEWAY_BASE_URL:-http://piha:8080}
+      - REQUEST_TIMEOUT_SECONDS=30
+    command: ["python", "worker.py"]
+    networks:
+      - internal
+
+  openclaw:
+    image: ai-cluster-openclaw
+    restart: unless-stopped
+    environment:
+      - MQTT_HOST=mosquitto
+      - MQTT_PORT=1883
+      - MQTT_USERNAME=${MQTT_USERNAME:-codex}
+      - MQTT_PASSWORD=${MQTT_PASSWORD}
+    command: ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+    ports:
+      - "8000:8000"
+    networks:
+      - internal
+      - npm_default
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 15s
+
+  planner-worker:
+    image: ai-cluster-planner-worker
+    restart: unless-stopped
+    environment:
+      - AGENT_ID=vps-planner-1
+      - ROLE=planner
+      - MQTT_HOST=mosquitto
+      - MQTT_PORT=1883
+      - MQTT_USERNAME=${MQTT_USERNAME:-codex}
+      - MQTT_PASSWORD=${MQTT_PASSWORD}
+    command: ["python", "planner_worker.py"]
+    networks:
+      - internal
+
+  service-ops-worker:
+    image: ai-cluster-service-ops-worker
+    restart: unless-stopped
+    environment:
+      - AGENT_ID=vps-service-ops-1
+      - ROLE=service-ops
+      - MQTT_HOST=mosquitto
+      - MQTT_PORT=1883
+      - MQTT_USERNAME=${MQTT_USERNAME:-codex}
+      - MQTT_PASSWORD=${MQTT_PASSWORD}
+      - COMPOSE_PROJECT_NAME=ai-cluster
+    command: ["python", "service_ops_worker.py"]
+    volumes:
+      # Post-migration: compose definition and env are in the repo/runtime paths.
+      # Pre-cutover these are overridden to old paths via docker-compose.override.yml.
+      - /home/oskar/homelab-codex-ws/services/ai-cluster/docker-compose.yml:/app/docker-compose.yml:ro
+      - /opt/homelab/config/ai-cluster/.env:/app/.env:ro
+      - /var/run/docker.sock:/var/run/docker.sock:rw
+    networks:
+      - internal
+
+  redis:
+    image: redis:7-alpine
+    restart: unless-stopped
+    command: ["redis-server"]
+    volumes:
+      - redis_data:/data
+    networks:
+      - internal
+
+  mosquitto:
+    image: eclipse-mosquitto:2
+    container_name: mosquitto
+    restart: unless-stopped
+    command: ["/usr/sbin/mosquitto", "-c", "/mosquitto/config/mosquitto.conf"]
+    ports:
+      # Tailscale IP binding — matches running container
+      - "100.95.58.48:1883:1883"
+    volumes:
+      # Config: kept at old path until mosquitto config migration is complete
+      - /home/dockeruser/docker/ai-cluster/mosquitto:/mosquitto/config:ro
+      - mosquitto_data:/mosquitto/data
+      - mosquitto_log:/mosquitto/log
+    networks:
+      - internal
+
+volumes:
+  redis_data:
+  mosquitto_data:
+  mosquitto_log:
+
+networks:
+  internal:
+    driver: bridge
+    name: ai-cluster_ai-cluster
+  npm_default:
+    external: true
+    name: npm_default

services/ai-cluster/env.example

@@ -0,0 +1,14 @@
+# AI Cluster — /opt/homelab/config/ai-cluster/.env
+# Read by all worker containers and mounted into service-ops-worker as /app/.env
+
+# MQTT broker credentials
+MQTT_HOST=mosquitto
+MQTT_PORT=1883
+MQTT_USERNAME=codex
+MQTT_PASSWORD=
+
+# API gateway (piha)
+GATEWAY_BASE_URL=http://piha:8080
+
+# Compose project name (required for service-ops-worker docker-compose operations)
+COMPOSE_PROJECT_NAME=ai-cluster

services/ai-cluster/healthcheck.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Healthcheck for AI cluster (checks openclaw API gateway is responding)
+
+if ! docker ps --filter "name=ai-cluster-openclaw-1" --filter "status=running" | grep -q "openclaw"; then
+    echo "[FAIL] openclaw container is not running"
+    exit 1
+fi
+
+if ! curl -sf http://localhost:8000/health > /dev/null; then
+    echo "[FAIL] openclaw HTTP health endpoint not responding"
+    exit 1
+fi
+
+echo "[OK] ai-cluster is healthy"
+exit 0

services/ai-cluster/service.yaml

@@ -0,0 +1,37 @@
+service:
+  name: ai-cluster
+  owner_node: vps
+  exposure: tailscale-internal
+  dependencies:
+    - mosquitto
+    - redis
+  ports:
+    - container: 8000
+      host: 8000
+      protocol: tcp
+      service: openclaw
+    - container: 1883
+      host: 1883
+      protocol: tcp
+      bind: 100.95.58.48   # Tailscale only
+      service: mosquitto
+  healthcheck:
+    type: http
+    endpoint: http://localhost:8000/health
+    interval: 30s
+    timeout: 10s
+    retries: 3
+  restart_policy: unless-stopped
+  persistence:
+    paths:
+      - volume:mosquitto_config_bind    # /home/dockeruser/docker/ai-cluster/mosquitto (bind, not volume)
+  runtime:
+    env_file: /opt/homelab/config/ai-cluster/.env
+    env_vars:
+      - MQTT_PASSWORD
+      - MQTT_USERNAME
+      - GATEWAY_BASE_URL
+  notes:
+    - "Local images (ai-cluster-*) must be built on VPS before deployment"
+    - "service-ops-worker mounts docker.sock and the compose file — needs post-migration path update"
+    - "Recommendation: move ai-cluster compute workloads to SOLARIA (GPU/compute node)"

services/control-plane/Dockerfile

@@ -20,5 +20,4 @@ ENV RUNTIME_PATH=/opt/homelab
 ENV PYTHONUNBUFFERED=1
 
 # Default command (will be overridden in docker-compose)
-USER homelab
 CMD ["python", "src/operator_ui.py"]

services/control-plane/deploy-local.sh

@@ -39,24 +39,10 @@ for dir in "${DIRS[@]}"; do
     fi
 done
 
-# 3. chown/chmod for UID 1000 — self-healing: only calls sudo when actually needed
+# 3. chown/chmod for UID 1000
-echo "Checking /opt/homelab ownership..."
+echo "Setting permissions for UID 1000 on /opt/homelab..."
-_chown_needed=$(find /opt/homelab \( ! -uid 1000 -o ! -gid 1000 \) -print -quit 2>/dev/null)
+sudo chown -R 1000:1000 /opt/homelab
-if [[ -n "$_chown_needed" ]]; then
+sudo chmod -R 775 /opt/homelab 2>/dev/null || true
-    echo "Found files not owned by 1000:1000 (e.g. $_chown_needed) — fixing..."
-    sudo chown -R 1000:1000 /opt/homelab
-else
-    echo "Ownership already correct, skipping chown"
-fi
-
-echo "Checking /opt/homelab directory permissions..."
-_chmod_needed=$(find /opt/homelab -type d ! -perm -775 -print -quit 2>/dev/null)
-if [[ -n "$_chmod_needed" ]]; then
-    echo "Found directories with wrong permissions (e.g. $_chmod_needed) — fixing..."
-    sudo chmod -R 775 /opt/homelab 2>/dev/null || true
-else
-    echo "Permissions already correct, skipping chmod"
-fi
 
 # 4. Run docker compose up -d --build --force-recreate
 echo "--- Starting Control Plane Services ---"

services/control-plane/docker-compose.yml

@@ -56,9 +56,6 @@ services:
   executor:
     build: .
     container_name: control-plane-executor
-    user: "1000:1000"
-    group_add:
-      - "999"
     command: python src/executor.py
     volumes:
       - /opt/homelab:/opt/homelab

services/control-plane/src/executor.py

@@ -5,16 +5,6 @@ import logging
 import subprocess
 from pathlib import Path
 
-
-def _atomic_write_json(path: Path, data) -> None:
-    """Write JSON atomically: write to a sibling .tmp, fsync, then os.replace."""
-    tmp = path.with_suffix(".tmp")
-    with open(tmp, "w") as f:
-        json.dump(data, f, indent=2)
-        f.flush()
-        os.fsync(f.fileno())
-    os.replace(tmp, path)
-
 # Constants and Paths
 RUNTIME_PATH = os.getenv("RUNTIME_PATH", "/opt/homelab")
 ACTIONS_DIR = Path(RUNTIME_PATH) / "actions"
@@ -67,7 +57,8 @@ class Executor:
                 data = json.load(f)
             data["status"] = "running"
             data["started_at"] = time.time()
-            _atomic_write_json(running_path, data)
+            with open(running_path, "w") as f:
+                json.dump(data, f, indent=2)
             action_file.unlink()
         except Exception as e:
             logger.error(f"Failed to move {action_id} to running: {e}")
@@ -130,7 +121,8 @@ class Executor:
             data["finished_at"] = time.time()
             if not success:
                 data["error"] = error_msg
-            _atomic_write_json(target_path, data)
+            with open(target_path, "w") as f:
+                json.dump(data, f, indent=2)
             running_path.unlink()
             logger.info(f"Action {action_id} {target_status}")
         except Exception as e:

services/control-plane/src/operator_ui.py

@@ -147,18 +147,12 @@ def current_deployments():
 
 
 def current_incidents():
-    """Return active incidents as a list sorted most-recent-first.
+    """Return incidents as a list sorted most-recent-first."""
-
-    Only incidents with status='active' are returned; resolved and cancelled
-    records are excluded so the dashboard reflects the current operational state.
-    """
     raw = read_json_file(WORLD_DIR / "incidents.json", default={})
     if isinstance(raw, list):
-        return [i for i in raw if i.get("status") == "active"]
+        return raw
     result = []
     for inc in raw.values():
-        if inc.get("status") != "active":
-            continue
         # Synthesise a human-readable message if not stored (observer doesn't set one).
         if "message" not in inc:
             inc = dict(inc)

services/control-plane/src/supervisor.py

@@ -5,16 +5,6 @@ import logging
 import yaml
 from pathlib import Path
 
-
-def _atomic_write_json(path: Path, data) -> None:
-    """Write JSON atomically: write to a sibling .tmp, fsync, then os.replace."""
-    tmp = path.with_suffix(".tmp")
-    with open(tmp, "w") as f:
-        json.dump(data, f, indent=2)
-        f.flush()
-        os.fsync(f.fileno())
-    os.replace(tmp, path)
-
 # Constants and Paths
 RUNTIME_PATH = os.getenv("RUNTIME_PATH", "/opt/homelab")
 WORLD_DIR = Path(RUNTIME_PATH) / "world"
@@ -185,11 +175,7 @@ class Supervisor:
                         logger.error(f"Failed to load {svc_file}: {e}")
         self.desired_state["services"] = services
 
-    def _load_actual_state(self) -> bool:
+    def _load_actual_state(self):
-        """Load world state from disk.  Returns False if any file is unreadable
-        (empty / mid-write truncation), in which case actual_state is NOT updated
-        so the caller can skip this reconcile cycle rather than treating missing
-        data as a real drift signal."""
         files = {
             "services": WORLD_DIR / "services.json",
             "nodes": WORLD_DIR / "nodes.json",
@@ -202,11 +188,8 @@ class Supervisor:
                     with open(path, "r") as f:
                         raw[key] = json.load(f)
                 except Exception as e:
-                    logger.warning(
+                    logger.error(f"Failed to load {key} actual state: {e}")
-                        f"World state {path.name} unreadable (truncated write?): {e} "
+                    raw[key] = {}
-                        f"— skipping reconcile cycle, keeping last known state"
-                    )
-                    return False
             else:
                 raw[key] = {}
 
@@ -236,7 +219,6 @@ class Supervisor:
         self.actual_state["services"] = normalized_services
         self.actual_state["nodes"] = raw.get("nodes", {})
         self.actual_state["incidents"] = normalized_incidents
-        return True
 
     # ------------------------------------------------------------------
     # Incident helpers
@@ -270,8 +252,7 @@ class Supervisor:
             logger.error(f"Failed to touch heartbeat file: {e}")
 
         self._load_desired_state()
-        if not self._load_actual_state():
+        self._load_actual_state()
-            return  # world state unreadable this cycle — skip to avoid false drift
 
         drifts = []
 
@@ -394,7 +375,8 @@ class Supervisor:
 
         action_path = ACTIONS_DIR / "pending" / f"{action_id}.json"
         try:
-            _atomic_write_json(action_path, action)
+            with open(action_path, "w") as f:
+                json.dump(action, f, indent=2)
             logger.info(
                 f"Generated recommendation: {action_id} "
                 f"(type={action['type']}, risk={action['risk_level']})"
@@ -446,7 +428,8 @@ class Supervisor:
 
         action_path = ACTIONS_DIR / "pending" / f"{action_id}.json"
         try:
-            _atomic_write_json(action_path, action)
+            with open(action_path, "w") as f:
+                json.dump(action, f, indent=2)
             logger.info(
                 f"Generated disk cleanup recommendation: {action_id} "
                 f"(node={node}, risk=guarded)"
@@ -511,7 +494,8 @@ class Supervisor:
                     action["status"] = "cancelled"
                     action["cancelled_reason"] = cancel_reason
                     action["cancelled_at"] = time.time()
-                    _atomic_write_json(dest, action)
+                    with open(dest, "w") as f:
+                        json.dump(action, f, indent=2)
                     action_file.unlink()
                     logger.info(
                         f"Auto-cancelled {action_file.name}: "
@@ -741,7 +725,8 @@ class Supervisor:
             action["status"] = "cancelled"
             action["cancelled_reason"] = "ha_websocket_recovered"
             action["cancelled_at"] = time.time()
-            _atomic_write_json(dest, action)
+            with open(dest, "w") as f:
+                json.dump(action, f, indent=2)
             pending_path.unlink()
             logger.info(f"Cancelled {action_id}: ha_websocket_recovered on {node}")
         except Exception as e:
@@ -751,7 +736,8 @@ class Supervisor:
         action_id = action["action_id"]
         action_path = ACTIONS_DIR / "pending" / f"{action_id}.json"
         try:
-            _atomic_write_json(action_path, action)
+            with open(action_path, "w") as f:
+                json.dump(action, f, indent=2)
             logger.info(
                 f"Generated HA action: {action_id} "
                 f"(type={action['type']}, risk={action['risk_level']})"

services/control-plane/tests/test_incident_lifecycle.py

@@ -1,333 +0,0 @@
-"""Tests for incident lifecycle: auto-resolve, orphan detection, timestamp parsing."""
-from __future__ import annotations
-
-import json
-import sys
-import time
-from pathlib import Path
-
-import pytest
-
-# Observer lives outside the control-plane package; add scripts/ to path.
-sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / "scripts"))
-from observer.observer import Observer, _parse_ts, _atomic_write_json
-
-
-# ---------------------------------------------------------------------------
-# Helpers
-# ---------------------------------------------------------------------------
-
-def _make_observer(tmp_path: Path) -> Observer:
-    """Return an Observer with all runtime paths redirected to tmp_path."""
-    import observer.observer as obs_mod
-
-    world = tmp_path / "world"
-    state = tmp_path / "state"
-    events = tmp_path / "events"
-    logs = tmp_path / "logs"
-    repo = tmp_path / "repo"
-
-    for d in (world, state, events, logs, repo / "inventory", repo / "hosts"):
-        d.mkdir(parents=True, exist_ok=True)
-
-    # Minimal topology so inventory isn't empty (avoids prune-guard early-return)
-    (repo / "inventory" / "topology.yaml").write_text(
-        "nodes:\n  vps:\n    roles: [control-plane]\n    connectivity: {}\n"
-    )
-
-    original_world = obs_mod.WORLD_DIR
-    original_state = obs_mod.STATE_DIR
-    original_events = obs_mod.EVENTS_DIR
-    original_logs = obs_mod.LOGS_DIR
-    original_inventory = obs_mod.INVENTORY_TOPOLOGY
-    original_repo = obs_mod.REPO_ROOT
-
-    obs_mod.WORLD_DIR = world
-    obs_mod.STATE_DIR = state
-    obs_mod.EVENTS_DIR = events
-    obs_mod.LOGS_DIR = logs
-    obs_mod.INVENTORY_TOPOLOGY = repo / "inventory" / "topology.yaml"
-    obs_mod.REPO_ROOT = repo
-
-    obs = Observer()
-
-    # Restore module-level constants (monkeypatching at module level is sufficient
-    # for the Observer instance which captures paths at construction time via globals)
-    obs_mod.WORLD_DIR = original_world
-    obs_mod.STATE_DIR = original_state
-    obs_mod.EVENTS_DIR = original_events
-    obs_mod.LOGS_DIR = original_logs
-    obs_mod.INVENTORY_TOPOLOGY = original_inventory
-    obs_mod.REPO_ROOT = original_repo
-
-    return obs
-
-
-def _make_observer_simple(tmp_path: Path):
-    """Return an Observer instance and patch its world_state in-place."""
-    import observer.observer as obs_mod
-
-    world = tmp_path / "world"
-    state = tmp_path / "state"
-    events = tmp_path / "events"
-    logs = tmp_path / "logs"
-    repo = tmp_path / "repo"
-
-    for d in (world, state, events, logs, repo / "inventory", repo / "hosts"):
-        d.mkdir(parents=True, exist_ok=True)
-
-    (repo / "inventory" / "topology.yaml").write_text(
-        "nodes:\n  vps:\n    roles: [control-plane]\n    connectivity: {}\n"
-    )
-
-    # Patch before construction
-    obs_mod.WORLD_DIR = world
-    obs_mod.STATE_DIR = state
-    obs_mod.EVENTS_DIR = events
-    obs_mod.LOGS_DIR = logs
-    obs_mod.INVENTORY_TOPOLOGY = repo / "inventory" / "topology.yaml"
-    obs_mod.REPO_ROOT = repo
-
-    obs = Observer()
-    return obs
-
-
-# ---------------------------------------------------------------------------
-# 1. _parse_ts — timestamp normalisation
-# ---------------------------------------------------------------------------
-
-def test_parse_ts_int():
-    ts = int(time.time()) - 3600
-    assert abs(_parse_ts(ts) - ts) < 1
-
-
-def test_parse_ts_float():
-    ts = time.time() - 100.5
-    assert abs(_parse_ts(ts) - ts) < 0.01
-
-
-def test_parse_ts_iso_string():
-    # ISO format as emitted by events.py / stability-agent
-    from datetime import datetime, timezone
-    iso = "2026-06-01T00:03:22Z"
-    expected = datetime(2026, 6, 1, 0, 3, 22, tzinfo=timezone.utc).timestamp()
-    result = _parse_ts(iso)
-    assert result > 0
-    assert isinstance(result, float)
-    assert abs(result - expected) < 1
-
-
-def test_parse_ts_none_returns_zero():
-    assert _parse_ts(None) == 0.0
-
-
-def test_parse_ts_garbage_returns_zero():
-    assert _parse_ts("not-a-date") == 0.0
-
-
-def test_parse_ts_zero_int():
-    assert _parse_ts(0) == 0.0
-
-
-# ---------------------------------------------------------------------------
-# 2. Lifecycle: service_healthy event resolves linked incident
-# ---------------------------------------------------------------------------
-
-def test_service_healthy_resolves_active_incident(tmp_path):
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-111-vps-outline"
-    obs.world_state["services"]["vps/outline"] = {
-        "node": "vps", "service": "outline",
-        "status": "unhealthy", "last_check": None,
-        "incident_id": inc_id,
-    }
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "node": "vps", "service": "outline",
-        "status": "active", "trigger_type": "service_unhealthy",
-        "started_at": int(time.time()) - 600,
-        "last_occurrence": int(time.time()) - 600,
-        "occurrence_count": 1, "events": [],
-    }
-
-    obs.process_event({
-        "type": "service_healthy",
-        "node": "vps",
-        "service": "outline",
-        "severity": "info",
-        "timestamp": int(time.time()),
-        "payload": {},
-    })
-
-    assert obs.world_state["services"]["vps/outline"]["status"] == "healthy"
-    assert obs.world_state["services"]["vps/outline"]["incident_id"] is None
-    assert obs.world_state["incidents"][inc_id]["status"] == "resolved"
-
-
-def test_service_healthy_does_not_resolve_other_incidents(tmp_path):
-    """service_healthy for service A must not touch incident for service B."""
-    obs = _make_observer_simple(tmp_path)
-    inc_b = "inc-222-vps-supervisor"
-    obs.world_state["services"]["vps/supervisor"] = {
-        "node": "vps", "service": "supervisor",
-        "status": "unhealthy", "last_check": None,
-        "incident_id": inc_b,
-    }
-    obs.world_state["incidents"][inc_b] = {
-        "id": inc_b, "status": "active",
-        "last_occurrence": int(time.time()) - 300,
-    }
-
-    obs.process_event({
-        "type": "service_healthy",
-        "node": "vps",
-        "service": "outline",   # different service
-        "severity": "info",
-        "timestamp": int(time.time()),
-        "payload": {},
-    })
-
-    assert obs.world_state["incidents"][inc_b]["status"] == "active"
-
-
-# ---------------------------------------------------------------------------
-# 3. _prune_stale_world: healthy-service-linked incident → immediate resolve
-# ---------------------------------------------------------------------------
-
-def test_prune_resolves_healthy_linked_incident(tmp_path):
-    """If a service is healthy but still points at an active incident, resolve it."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-333-vps-outline"
-    obs.world_state["services"]["vps/outline"] = {
-        "node": "vps", "service": "outline",
-        "status": "healthy",          # <-- healthy but incident_id still set
-        "last_check": None,
-        "incident_id": inc_id,
-    }
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active",
-        "started_at": int(time.time()) - 7200,
-        "last_occurrence": int(time.time()) - 7200,
-    }
-
-    obs._prune_stale_world()
-
-    assert obs.world_state["services"]["vps/outline"]["incident_id"] is None
-    assert obs.world_state["incidents"][inc_id]["status"] == "resolved"
-
-
-def test_prune_resolves_healthy_linked_incident_iso_timestamp(tmp_path):
-    """Healthy-linked incident with ISO-string last_occurrence must still resolve."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-444-vps-outline"
-    obs.world_state["services"]["vps/outline"] = {
-        "node": "vps", "service": "outline",
-        "status": "healthy", "last_check": None, "incident_id": inc_id,
-    }
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active",
-        "last_occurrence": "2026-06-01T00:03:22Z",  # ISO string from events.py
-    }
-
-    obs._prune_stale_world()   # must not raise TypeError
-
-    assert obs.world_state["incidents"][inc_id]["status"] == "resolved"
-
-
-# ---------------------------------------------------------------------------
-# 4. _prune_stale_world: orphaned incident (no service link) → resolve after 5 min
-# ---------------------------------------------------------------------------
-
-def test_prune_resolves_orphaned_incident_old_enough(tmp_path):
-    """Orphaned active incident older than 5 min must be auto-resolved."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-555-vps-supervisor"
-    # No service entry links to this incident
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active", "node": "vps", "service": "supervisor",
-        "last_occurrence": int(time.time()) - 400,   # 6.7 min ago
-    }
-
-    obs._prune_stale_world()
-
-    assert obs.world_state["incidents"][inc_id]["status"] == "resolved"
-
-
-def test_prune_does_not_resolve_orphaned_incident_too_recent(tmp_path):
-    """Orphaned incident younger than 5 min must stay active (guard against race)."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-666-vps-supervisor"
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active",
-        "last_occurrence": int(time.time()) - 60,   # 1 min ago — within guard
-    }
-
-    obs._prune_stale_world()
-
-    assert obs.world_state["incidents"][inc_id]["status"] == "active"
-
-
-def test_prune_resolves_orphaned_incident_iso_timestamp(tmp_path):
-    """Orphaned incident with ISO-string last_occurrence must resolve correctly."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-777-vps-outline"
-    # ISO timestamp well in the past (2026-06-01)
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active",
-        "last_occurrence": "2026-06-01T00:03:22Z",
-    }
-
-    obs._prune_stale_world()   # must not raise TypeError
-
-    assert obs.world_state["incidents"][inc_id]["status"] == "resolved"
-
-
-def test_prune_does_not_touch_linked_incident(tmp_path):
-    """An active incident still linked from a non-healthy service must stay active."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-888-vps-outline"
-    obs.world_state["services"]["vps/outline"] = {
-        "node": "vps", "service": "outline",
-        "status": "unhealthy",   # <-- still unhealthy
-        "last_check": None,
-        "incident_id": inc_id,
-    }
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "active",
-        "last_occurrence": int(time.time()) - 3600,
-    }
-
-    obs._prune_stale_world()
-
-    assert obs.world_state["incidents"][inc_id]["status"] == "active"
-
-
-# ---------------------------------------------------------------------------
-# 5. 7-day stale incident prune with ISO resolved_at
-# ---------------------------------------------------------------------------
-
-def test_prune_removes_old_resolved_incident_iso_resolved_at(tmp_path):
-    """Resolved incidents with ISO-string resolved_at older than 7 days must be pruned."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-old-resolved"
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "resolved",
-        "resolved_at": "2026-05-01T00:00:00Z",  # >7 days before 2026-06-03
-    }
-
-    obs._prune_stale_world()
-
-    assert inc_id not in obs.world_state["incidents"]
-
-
-def test_prune_keeps_recently_resolved_incident(tmp_path):
-    """Resolved incidents within 7 days must be kept."""
-    obs = _make_observer_simple(tmp_path)
-    inc_id = "inc-recent-resolved"
-    obs.world_state["incidents"][inc_id] = {
-        "id": inc_id, "status": "resolved",
-        "resolved_at": time.time() - 86400,  # 1 day ago
-    }
-
-    obs._prune_stale_world()
-
-    assert inc_id in obs.world_state["incidents"]

services/control-plane/tests/test_state_reliability.py

@@ -1,199 +0,0 @@
-"""Tests for atomic writes and resilient world-state loading in the supervisor."""
-from __future__ import annotations
-
-import json
-import sys
-import time
-from pathlib import Path
-
-import pytest
-
-sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
-import supervisor as supervisor_module
-from supervisor import Supervisor, _atomic_write_json
-
-
-# ---------------------------------------------------------------------------
-# Helpers (reused from test_supervisor_ha)
-# ---------------------------------------------------------------------------
-
-def _setup_supervisor(tmp_path: Path, monkeypatch) -> Supervisor:
-    actions = tmp_path / "actions"
-    events = tmp_path / "events"
-    world = tmp_path / "world"
-    repo = tmp_path / "repo"
-
-    for d in (actions, events, world, repo / "hosts"):
-        d.mkdir(parents=True, exist_ok=True)
-
-    monkeypatch.setattr(supervisor_module, "ACTIONS_DIR", actions)
-    monkeypatch.setattr(supervisor_module, "EVENTS_DIR", events)
-    monkeypatch.setattr(supervisor_module, "WORLD_DIR", world)
-    monkeypatch.setattr(supervisor_module, "REPO_ROOT", repo)
-
-    sup = Supervisor()
-    sup.desired_state = {"services": {}}
-    sup.actual_state = {"services": {}, "nodes": {}, "incidents": {}}
-    return sup
-
-
-# ---------------------------------------------------------------------------
-# 1. atomic_write_json correctness
-# ---------------------------------------------------------------------------
-
-def test_atomic_write_json_produces_valid_json(tmp_path):
-    path = tmp_path / "out.json"
-    data = {"services": {"vps/outline": {"status": "healthy"}}, "count": 42}
-    _atomic_write_json(path, data)
-
-    assert path.exists(), "output file must exist after atomic write"
-    loaded = json.loads(path.read_text())
-    assert loaded == data
-
-
-def test_atomic_write_json_no_tmp_left_behind(tmp_path):
-    path = tmp_path / "world.json"
-    _atomic_write_json(path, {"ok": True})
-
-    tmp = path.with_suffix(".tmp")
-    assert not tmp.exists(), ".tmp must be cleaned up by os.replace"
-
-
-def test_atomic_write_json_overwrites_existing(tmp_path):
-    path = tmp_path / "state.json"
-    path.write_text('{"old": true}')
-    _atomic_write_json(path, {"new": True})
-    assert json.loads(path.read_text()) == {"new": True}
-
-
-def test_atomic_write_json_nested_structure(tmp_path):
-    path = tmp_path / "complex.json"
-    data = {
-        "nodes": {"vps": {"status": "online", "disk_usage_pct": 42}},
-        "incidents": {},
-        "list": [1, 2, 3],
-    }
-    _atomic_write_json(path, data)
-    assert json.loads(path.read_text()) == data
-
-
-# ---------------------------------------------------------------------------
-# 2. Resilient loader: empty / truncated file → skip cycle, no drift
-# ---------------------------------------------------------------------------
-
-def _populate_desired(sup: Supervisor, svc_key: str = "vps/outline"):
-    node, service = svc_key.split("/", 1)
-    sup.desired_state["services"][svc_key] = {
-        "node": node,
-        "service": service,
-        "desired": "running",
-    }
-
-
-def test_empty_services_json_skips_reconcile(tmp_path, monkeypatch):
-    """Empty services.json (truncated write) must not generate any redeploy action."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-    _populate_desired(sup)
-
-    # Write empty services.json — simulates a mid-write truncation
-    (tmp_path / "world" / "services.json").write_text("")
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-
-    sup.reconcile()
-
-    pending = list((tmp_path / "actions" / "pending").glob("*.json"))
-    assert pending == [], f"No actions should be generated on empty state file, got: {[p.name for p in pending]}"
-
-
-def test_truncated_services_json_skips_reconcile(tmp_path, monkeypatch):
-    """Partially-written (truncated mid-write) JSON must not generate any action."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-    _populate_desired(sup)
-
-    (tmp_path / "world" / "services.json").write_text('{"vps/outline": {"status": "hea')
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-
-    sup.reconcile()
-
-    pending = list((tmp_path / "actions" / "pending").glob("*.json"))
-    assert pending == [], f"No actions expected on truncated state, got: {[p.name for p in pending]}"
-
-
-def test_empty_incidents_json_skips_reconcile(tmp_path, monkeypatch):
-    """Empty incidents.json (any world-state file failing) skips full cycle."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-    _populate_desired(sup)
-
-    (tmp_path / "world" / "services.json").write_text("{}")
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("")
-
-    sup.reconcile()
-
-    pending = list((tmp_path / "actions" / "pending").glob("*.json"))
-    assert pending == [], f"No actions expected when any state file is unreadable, got: {[p.name for p in pending]}"
-
-
-def test_load_actual_state_returns_false_on_empty_file(tmp_path, monkeypatch):
-    """_load_actual_state must return False (not raise) when a file is empty."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-
-    (tmp_path / "world" / "services.json").write_text("")
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-
-    result = sup._load_actual_state()
-    assert result is False
-
-
-def test_load_actual_state_returns_true_on_valid_files(tmp_path, monkeypatch):
-    """_load_actual_state returns True and populates actual_state on valid files."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-
-    services = {"vps/outline": {"node": "vps", "service": "outline", "status": "healthy"}}
-    (tmp_path / "world" / "services.json").write_text(json.dumps(services))
-    (tmp_path / "world" / "nodes.json").write_text('{"vps": {"status": "online"}}')
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-
-    result = sup._load_actual_state()
-    assert result is True
-    assert "vps/outline" in sup.actual_state["services"]
-
-
-def test_parse_failure_preserves_last_known_good_state(tmp_path, monkeypatch):
-    """When a file becomes unreadable, actual_state retains the previous good values."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-
-    # First successful load
-    services = {"vps/outline": {"node": "vps", "service": "outline", "status": "healthy"}}
-    (tmp_path / "world" / "services.json").write_text(json.dumps(services))
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-    assert sup._load_actual_state() is True
-    assert "vps/outline" in sup.actual_state["services"]
-
-    # File becomes empty (race condition)
-    (tmp_path / "world" / "services.json").write_text("")
-    assert sup._load_actual_state() is False
-
-    # State must be unchanged from the previous good load
-    assert "vps/outline" in sup.actual_state["services"], \
-        "Last-known-good state must be preserved on parse failure"
-
-
-def test_healthy_service_does_not_generate_action(tmp_path, monkeypatch):
-    """A desired service that appears healthy in world state generates no action."""
-    sup = _setup_supervisor(tmp_path, monkeypatch)
-    _populate_desired(sup)
-
-    services = {"vps/outline": {"node": "vps", "service": "outline", "status": "healthy"}}
-    (tmp_path / "world" / "services.json").write_text(json.dumps(services))
-    (tmp_path / "world" / "nodes.json").write_text("{}")
-    (tmp_path / "world" / "incidents.json").write_text("{}")
-
-    sup.reconcile()
-
-    pending = list((tmp_path / "actions" / "pending").glob("*.json"))
-    assert pending == [], "Healthy service must not generate any action"

services/joplin/docker-compose.yml

@@ -0,0 +1,44 @@
+services:
+  app:
+    image: joplin/server:latest
+    container_name: joplin-server
+    restart: unless-stopped
+    env_file:
+      - /opt/homelab/config/joplin/.env
+    ports:
+      - "127.0.0.1:22300:22300"
+    depends_on:
+      db:
+        condition: service_healthy
+    networks:
+      - joplin_net
+      - npm_default
+
+  db:
+    image: postgres:18
+    container_name: joplin-db
+    restart: unless-stopped
+    env_file:
+      - /opt/homelab/config/joplin/.env
+    volumes:
+      - postgres_data:/var/lib/postgresql
+    networks:
+      - joplin_net
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U joplin -d joplin"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+volumes:
+  postgres_data:
+    external: true
+    name: joplin_postgres_data
+
+networks:
+  joplin_net:
+    driver: bridge
+    name: joplin-net
+  npm_default:
+    external: true
+    name: npm_default

services/joplin/env.example

@@ -0,0 +1,20 @@
+# Joplin Server — /opt/homelab/config/joplin/.env
+# Both the `app` (joplin-server) and `db` (postgres) containers read this file.
+
+# Application
+APP_BASE_URL=https://joplin.example.com
+APP_PORT=22300
+TRUST_PROXY=1
+RUNNING_IN_DOCKER=1
+
+# Database connection (joplin-server reads these)
+DB_CLIENT=pg
+POSTGRES_HOST=db
+POSTGRES_PORT=5432
+POSTGRES_USER=joplin
+POSTGRES_DB=joplin
+POSTGRES_DATABASE=joplin
+POSTGRES_PASSWORD=
+
+# Runtime
+PM2_HOME=/opt/pm2

services/joplin/healthcheck.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Healthcheck for Joplin Server
+
+if ! docker ps --filter "name=joplin-server" --filter "status=running" | grep -q "joplin-server"; then
+    echo "[FAIL] joplin-server container is not running"
+    exit 1
+fi
+
+if ! curl -sf http://localhost:22300/api/ping > /dev/null; then
+    echo "[FAIL] Joplin Server HTTP endpoint not responding"
+    exit 1
+fi
+
+echo "[OK] Joplin Server is healthy"
+exit 0

services/joplin/service.yaml

@@ -0,0 +1,31 @@
+service:
+  name: joplin
+  owner_node: vps
+  exposure: tailscale-internal
+  dependencies:
+    - db
+  ports:
+    - container: 22300
+      host: 22300
+      protocol: tcp
+      bind: 127.0.0.1
+  healthcheck:
+    type: http
+    endpoint: http://localhost:22300/api/ping
+    interval: 30s
+    timeout: 10s
+    retries: 3
+  restart_policy: unless-stopped
+  persistence:
+    paths:
+      - volume:joplin_postgres_data     # Joplin notes DB
+  runtime:
+    env_file: /opt/homelab/config/joplin/.env
+    env_vars:
+      - APP_BASE_URL
+      - APP_PORT
+      - DB_CLIENT
+      - POSTGRES_HOST
+      - POSTGRES_USER
+      - POSTGRES_PASSWORD
+      - POSTGRES_DB

services/node-agent/Dockerfile

@@ -14,11 +14,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # pyyaml      : may be needed for reading host config snippets
 RUN pip install --no-cache-dir "docker>=6.0" psutil pyyaml
 
-RUN useradd -m -u 1000 homelab
-
 COPY src/ /app/src/
 
 ENV PYTHONUNBUFFERED=1
 
-USER homelab
 CMD ["python", "src/node_agent.py"]

services/node-agent/docker-compose.yml

@@ -2,9 +2,6 @@ services:
   node-agent:
     build: .
     container_name: node-agent
-    user: "1000:1000"
-    group_add:
-      - "999"
     restart: unless-stopped
 
     environment:

services/npm/docker-compose.yml

@@ -8,5 +8,7 @@ services:
       - '81:81'
       - '443:443'
     volumes:
-      - /opt/homelab/data/npm/data:/data
+      # Data lives at dockeruser's path — do NOT move these without a migration plan.
-      - /opt/homelab/data/npm/letsencrypt:/etc/letsencrypt
+      # Proxy hosts, SSL certs, and DB are stored here.
+      - /home/dockeruser/docker/npm/data:/data
+      - /home/dockeruser/docker/npm/letsencrypt:/etc/letsencrypt

services/npm/service.yaml

@@ -22,10 +22,6 @@ service:
   restart_policy: unless-stopped
   persistence:
     paths:
-      - /opt/homelab/data/npm/data
+      - /home/dockeruser/docker/npm/data
-      - /opt/homelab/data/npm/letsencrypt
+      - /home/dockeruser/docker/npm/letsencrypt
-  runtime:
-    directories:
-      - /opt/homelab/data/npm/data
-      - /opt/homelab/data/npm/letsencrypt
     env_vars: []

services/outline/docker-compose.yml

@@ -0,0 +1,68 @@
+services:
+  outline:
+    image: outlinewiki/outline:1.6.1
+    container_name: outline-outline-1
+    restart: unless-stopped
+    env_file:
+      - /opt/homelab/config/outline/.env
+    ports:
+      - "3000:3000"
+    volumes:
+      - outline_storage:/var/lib/outline/data
+    depends_on:
+      - postgres
+      - redis
+    networks:
+      - outline_internal
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:3000/_health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: outline-postgres-1
+    restart: unless-stopped
+    env_file:
+      - /opt/homelab/config/outline/.env
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+    networks:
+      - outline_internal
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U outline -d outline"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  redis:
+    image: redis:7-alpine
+    container_name: outline-redis-1
+    restart: unless-stopped
+    volumes:
+      - redis_data:/data
+    networks:
+      - outline_internal
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  outline_storage:
+    external: true
+    name: outline_outline_storage
+  postgres_data:
+    external: true
+    name: outline_postgres_data
+  redis_data:
+    external: true
+    name: outline_redis_data
+
+networks:
+  outline_internal:
+    driver: bridge
+    name: outline_outline_internal

services/outline/env.example

@@ -0,0 +1,40 @@
+# Outline Wiki — /opt/homelab/config/outline/.env
+# Both the `outline` and `postgres` containers read this file.
+
+# Application
+URL=https://outline.example.com
+NODE_ENV=production
+PORT=3000
+FILE_STORAGE=local
+FILE_STORAGE_LOCAL_ROOT_DIR=/var/lib/outline/data
+FORCE_HTTPS=true
+
+# Secrets — generate with: openssl rand -hex 32
+SECRET_KEY=
+UTILS_SECRET=
+
+# Database
+DATABASE_URL=postgres://outline:<password>@postgres:5432/outline
+PGSSLMODE=disable
+
+# Redis
+REDIS_URL=redis://redis:6379
+
+# Postgres sidecar vars (read by the postgres container)
+POSTGRES_USER=outline
+POSTGRES_DB=outline
+POSTGRES_PASSWORD=
+
+# Google OAuth (optional)
+GOOGLE_CLIENT_ID=
+GOOGLE_CLIENT_SECRET=
+
+# SMTP
+SMTP_HOST=
+SMTP_PORT=587
+SMTP_USERNAME=
+SMTP_PASSWORD=
+SMTP_FROM_EMAIL=outline@example.com
+SMTP_REPLY_EMAIL=outline@example.com
+SMTP_SECURE=false
+ALLOWED_DOMAINS=

services/outline/healthcheck.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Healthcheck for Outline Wiki stack
+
+if ! docker ps --filter "name=outline-outline-1" --filter "status=running" | grep -q "outline-outline-1"; then
+    echo "[FAIL] outline container is not running"
+    exit 1
+fi
+
+if ! curl -sf http://localhost:3000/_health > /dev/null; then
+    echo "[FAIL] Outline HTTP health endpoint not responding"
+    exit 1
+fi
+
+echo "[OK] Outline is healthy"
+exit 0

services/outline/service.yaml

@@ -0,0 +1,36 @@
+service:
+  name: outline
+  owner_node: vps
+  exposure: public
+  dependencies:
+    - postgres
+    - redis
+  ports:
+    - container: 3000
+      host: 3000
+      protocol: tcp
+  healthcheck:
+    type: http
+    endpoint: http://localhost:3000/_health
+    interval: 30s
+    timeout: 10s
+    retries: 3
+  restart_policy: unless-stopped
+  persistence:
+    paths:
+      # Docker named volumes — data stays at Docker volume paths
+      - volume:outline_outline_storage   # /var/lib/outline/data inside container
+      - volume:outline_postgres_data     # Postgres data directory
+      - volume:outline_redis_data        # Redis persistence
+  runtime:
+    env_file: /opt/homelab/config/outline/.env
+    env_vars:
+      - URL
+      - DATABASE_URL
+      - REDIS_URL
+      - SECRET_KEY
+      - UTILS_SECRET
+      - FILE_STORAGE
+      - POSTGRES_USER
+      - POSTGRES_PASSWORD
+      - POSTGRES_DB

services/stability-agent/Dockerfile

@@ -5,8 +5,6 @@ WORKDIR /app
 # No extra dependencies needed beyond standard library for the current script
 # But we might need them if we decide to use libraries later.
 
-RUN useradd -m -u 1000 homelab
-
 COPY src/stability_agent.py .
 COPY healthcheck.sh .
 RUN chmod +x healthcheck.sh
@@ -14,5 +12,5 @@ RUN chmod +x healthcheck.sh
 # Create the expected directories
 RUN mkdir -p /opt/homelab/state /opt/homelab/events
 
-USER homelab
+# Run the agent
 CMD ["python", "stability_agent.py"]

services/stability-agent/docker-compose.yml

@@ -2,9 +2,6 @@ services:
   stability-agent:
     build: .
     container_name: stability-agent
-    user: "1000:1000"
-    group_add:
-      - "999"
     restart: unless-stopped
     volumes:
       - /opt/homelab:/opt/homelab