.claude/skills/deploy/SKILL.md

@@ -1,43 +0,0 @@
----
-name: deploy
-description: Deploy, redeploy, or ship homelab services to a target node. Trigger on any request containing deploy / redeploy / wdróż / zredeployuj / ship for targets control-plane, vps, piha, solaria, or chelsty-infra.
----
-
-Always invoke `scripts/deploy/deploy.sh <target> [--dry-run] [--no-gate]` as the **sole entry point**.
-Never call `deploy-control-plane.sh`, `deploy-node.sh`, or `deploy-local.sh` directly.
-
-## Targets
-
-| Target | What it deploys |
-|---|---|
-| `control-plane` | observer, supervisor, executor, operator-ui on VPS |
-| `vps` | all VPS GitOps services (node-agent, npm, outline, joplin, ai-cluster, …) |
-| `piha` | PIHA services (ha-diag-agent, node-agent, redis, …) |
-| `solaria` | SOLARIA compute services |
-| `chelsty-infra` | CHELSTY LTE edge node (30 s SSH timeout) |
-
-## Invocation
-
-```bash
-scripts/deploy/deploy.sh <target>            # full pipeline
-scripts/deploy/deploy.sh <target> --dry-run  # preflight + gate only
-scripts/deploy/deploy.sh <target> --no-gate  # emergency: bypass tests
-```
-
-## Exit Code Handling
-
-| Code | Meaning | Required action |
-|---|---|---|
-| 0 | Success | Report: target, commit hash, gate status, verify status, elapsed time |
-| 1 | Preflight failed | Fix the upstream issue (push commits, wake node, switch to master). Never bypass. |
-| 2 | Gate failed | Show exactly which test/build failed. Do **not** deploy. Fix the failure first. |
-| 3 | Execute failed | Show full deploy output. Ask user whether to investigate or rollback. |
-| 4 | Verify failed | Show docker ps output. Discuss rollback with the user. |
-| 5 | Sudo handoff | Print the exact manual command from stderr **verbatim** and stop. User must run it. |
-
-## Rules
-
-- Never pass `--no-gate` unless the user explicitly requests emergency/bypass mode.
-- Never deploy uncommitted or unpushed code — preflight enforces this; do not help circumvent it.
-- Canonical branch is `master` — preflight enforces this.
-- For exit 5: reproduce the handoff command exactly as printed to stderr, then stop.

.claude/skills/save-session/SKILL.md

@@ -1,65 +0,0 @@
----
-name: save-session
-description: Save and record the current work session to docs/sessions/. Trigger ONLY on explicit "save session", "zapisz sesję", or "wrap up" — never invoke proactively between tasks.
----
-
-**Trigger condition**: user explicitly says "save session", "zapisz sesję", "wrap up", or equivalent.
-Never invoke proactively. Never invoke mid-task.
-
-## 1. Determine Session Boundary
-
-1. Read the latest entry file in `docs/sessions/` — use its last `## Session HH:MM` heading timestamp as the start boundary.
-2. Fallback if no previous entry exists: 24 hours ago.
-
-## 2. Collect Facts (deterministic only — no invention)
-
-Run exactly:
-```bash
-# All commits since boundary
-git --no-pager log --oneline <boundary>..HEAD
-
-# Changed file summary
-git --no-pager diff --stat <boundary>..HEAD
-```
-
-From the visible conversation transcript: deploys run and their outcomes, test results seen.
-
-## 3. Write the Session Entry
-
-**APPEND** to `docs/sessions/YYYY-MM-DD.md` (create the file if it doesn't exist for today).
-Never overwrite existing content.
-
-```markdown
-## Session HH:MM
-
-### Commits
-<output of git log --oneline>
-
-### Files changed
-<output of git diff --stat>
-
-### Deploys
-<list from transcript, or "None recorded">
-
-### Narrative
-> _user-provided summary_
-```
-
-The `> _user-provided summary_` placeholder is **mandatory**. Never fill it in. The user supplies the narrative separately if desired.
-
-## 4. What NOT to Touch
-
-- `backlog.md` — only on explicit "update backlog" instruction
-- `CLAUDE.md` — only on explicit "update CLAUDE.md" instruction
-- Any other file not listed above
-
-## 5. Commit
-
-Stage and commit **only** the session file:
-
-```bash
-git add docs/sessions/YYYY-MM-DD.md
-git commit -m "docs: session YYYY-MM-DD HH:MM"
-```
-
-No other files. No `git add -A`.

scripts/deploy/deploy.sh

@@ -1,321 +1,270 @@
 #!/usr/bin/env bash
-# scripts/deploy/deploy.sh — Saturn-side deploy dispatcher
+# deploy.sh - Staged deployment framework for homelab nodes.
-# Usage: deploy.sh <target> [--dry-run] [--no-gate]
-#   target ∈ {control-plane, vps, piha, solaria, chelsty-infra}
-# Exit codes: 0=ok  1=preflight  2=gate  3=execute  4=verify  5=handoff(sudo)
 
-set -uo pipefail
+set -o pipefail
 
-REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+# --- Configuration ---
-SSH_USER="${SSH_USER:-oskar}"
+export RUNTIME_PATH="/opt/homelab"
-START_TIME=$(date +%s)
+export STATE_DIR="${RUNTIME_PATH}/state/deploy"
-TARGET=""
+export LOG_DIR="${RUNTIME_PATH}/logs/deploy"
-DRY_RUN=false
+export REPO_PATH="${HOME}/homelab-codex-ws"
-NO_GATE=false
+export TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+export LOG_FILE="${LOG_DIR}/deploy_${TIMESTAMP}.log"
 
-usage() {
+# --- Initialization ---
-    cat >&2 <<'EOF'
+mkdir -p "$STATE_DIR" "$LOG_DIR"
-Usage: deploy.sh <target> [--dry-run] [--no-gate]
 
-Targets:
+# Redirection for logging
-  control-plane   observer/supervisor/executor/operator-ui on VPS
+exec > >(tee -a "$LOG_FILE") 2>&1
-  vps             all VPS GitOps services
-  piha            PIHA services
-  solaria         SOLARIA compute services
-  chelsty-infra   CHELSTY edge node (LTE, longer SSH timeout)
 
-Flags:
+# --- Load Libraries ---
-  --dry-run   run preflight + gate only; stop before deploy
+LIB_PATH="${REPO_PATH}/scripts/lib"
-  --no-gate   skip pytest + docker build (emergency only; logged as WARNING)
+source "${LIB_PATH}/log.sh"
+source "${LIB_PATH}/state.sh"
+source "${LIB_PATH}/inventory.sh"
+source "${LIB_PATH}/compose.sh"
+source "${LIB_PATH}/diagnostics.sh"
 
-Exit codes: 0=ok  1=preflight  2=gate  3=execute  4=verify  5=handoff(sudo)
+# --- CLI Parsing ---
-EOF
+TARGET_HOST=$(hostname)
-    exit 1
+TARGET_SERVICE=""
-}
+RESUME=false
+REQUESTED_STAGE=""
 
 while [[ $# -gt 0 ]]; do
     case $1 in
-        control-plane|vps|piha|solaria|chelsty-infra)
+        --host)
-            TARGET="$1"; shift ;;
+            TARGET_HOST="$2"
-        --dry-run)
+            shift 2
-            DRY_RUN=true; shift ;;
+            ;;
-        --no-gate)
+        --service)
-            NO_GATE=true; shift ;;
+            TARGET_SERVICE="$2"
-        -h|--help)
+            shift 2
-            usage ;;
+            ;;
+        --resume)
+            RESUME=true
+            shift
+            ;;
+        --stage)
+            REQUESTED_STAGE="$2"
+            shift 2
+            ;;
         *)
-            echo "Unknown argument: $1" >&2
+            if [[ "$1" =~ ^(prepare|validate|deploy|verify|diagnose|complete)$ ]]; then
-            usage ;;
+                REQUESTED_STAGE="$1"
+            fi
+            shift
+            ;;
     esac
 done
 
-[[ -z "$TARGET" ]] && { echo "Error: target is required." >&2; usage; }
+# --- Stages ---
 
-case "$TARGET" in
+stage_prepare() {
-    control-plane) SSH_HOST="vps" ;;
+    local host=$1
-    *)             SSH_HOST="$TARGET" ;;
+    if is_stage_complete "prepare" && [[ "$RESUME" == "true" ]]; then
-esac
+        log "INFO" "Skipping PREPARE (already complete)"
-
+        return 0
-case "$TARGET" in
-    chelsty-*) SSH_TIMEOUT=30 ;;
-    *)         SSH_TIMEOUT=5 ;;
-esac
-
-# ── PREFLIGHT ────────────────────────────────────────────────────────────────
-
-preflight() {
-    echo "=== PREFLIGHT ==="
-
-    local branch
-    branch=$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)
-    if [[ "$branch" != "master" ]]; then
-        echo "ERROR: On branch '${branch}', not master. Switch to master and push first." >&2
-        exit 1
     fi
-    echo "[ok] branch: master"
 
-    if ! git -C "$REPO_ROOT" diff --quiet; then
+    log "INFO" "Stage: PREPARE ($host)"
-        echo "ERROR: Unstaged changes in working tree. Commit or stash before deploying." >&2
+    set_stage "prepare"
-        exit 1
+    
-    fi
+    emit_event "deployment_started" "info" "deploy.sh" "all" "${TIMESTAMP}" "{\"stage\": \"prepare\"}"
-    if ! git -C "$REPO_ROOT" diff --cached --quiet; then
-        echo "ERROR: Staged but uncommitted changes. Commit before deploying." >&2
-        exit 1
-    fi
-    echo "[ok] working tree clean"
 
-    git -C "$REPO_ROOT" fetch origin master --quiet
+    cd "$REPO_PATH" || exit 1
-    local unpushed
+    log "INFO" "Pulling latest changes..."
-    unpushed=$(git -C "$REPO_ROOT" log origin/master..HEAD --oneline)
+    if ! git pull; then
-    if [[ -n "$unpushed" ]]; then
+        log "WARN" "Git pull failed, proceeding with local state (offline mode or network flap)"
-        echo "ERROR: Unpushed commits on master:" >&2
-        echo "$unpushed" >&2
-        echo "Push first:  git push origin master" >&2
-        exit 1
     fi
-    echo "[ok] no unpushed commits"
 
-    echo "Checking SSH: ${SSH_USER}@${SSH_HOST} (ConnectTimeout=${SSH_TIMEOUT}s)..."
+    # Ensure runtime directories exist
-    if ! ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
+    mkdir -p "${RUNTIME_PATH}/config" "${RUNTIME_PATH}/data" "${RUNTIME_PATH}/state" "${RUNTIME_PATH}/logs"
-            "${SSH_USER}@${SSH_HOST}" true 2>/dev/null; then
+
-        echo "ERROR: Cannot reach ${SSH_HOST} via SSH (timeout ${SSH_TIMEOUT}s)." >&2
+    struct_log "prepare" "$host" "all" "success" "repo_updated"
-        exit 1
+    mark_stage_complete "prepare"
-    fi
-    echo "[ok] ${SSH_HOST} reachable"
 }
 
-# ── GATE ─────────────────────────────────────────────────────────────────────
+stage_validate() {
-
+    local host=$1
-gate() {
+    if is_stage_complete "validate" && [[ "$RESUME" == "true" ]]; then
-    if [[ "$NO_GATE" == "true" ]]; then
+        log "INFO" "Skipping VALIDATE (already complete)"
-        echo "=== GATE: SKIPPED ==="
-        echo "WARNING: --no-gate active — pytest + docker build bypassed (emergency mode)." >&2
         return 0
     fi
 
-    echo "=== GATE ==="
+    log "INFO" "Stage: VALIDATE ($host)"
+    set_stage "validate"
 
-    local services=()
+    for service in "${SERVICES[@]}"; do
-
+        log "INFO" "Validating $service..."
-    if [[ "$TARGET" == "control-plane" ]]; then
+        if [[ ! -d "${REPO_PATH}/services/$service" ]]; then
-        services=("control-plane")
+            log "ERROR" "Service definition not found: $service"
-    else
+            struct_log "validate" "$host" "$service" "fail" "not_found"
-        local svc_yaml="${REPO_ROOT}/hosts/${TARGET}/services.yaml"
+            return 1
-        if [[ ! -f "$svc_yaml" ]]; then
-            echo "ERROR: ${svc_yaml} not found." >&2
-            exit 2
-        fi
-        local svc_list
-        svc_list=$(python3 -c "
-import yaml
-with open('${svc_yaml}') as f:
-    data = yaml.safe_load(f)
-svcs = data.get('services', {})
-if isinstance(svcs, dict):
-    print('\n'.join(svcs.keys()))
-elif isinstance(svcs, list):
-    print('\n'.join(svcs))
-")
-        while IFS= read -r svc; do
-            [[ -z "$svc" ]] && continue
-            if [[ -f "${REPO_ROOT}/services/${svc}/Dockerfile" ]]; then
-                services+=("$svc")
-            fi
-        done <<< "$svc_list"
-    fi
-
-    if [[ ${#services[@]} -eq 0 ]]; then
-        echo "[info] No services with local Dockerfile found for ${TARGET} — gate trivially passes."
-        return 0
-    fi
-
-    echo "Services under gate: ${services[*]}"
-    local gate_failed=false
-
-    for svc in "${services[@]}"; do
-        local svc_dir="${REPO_ROOT}/services/${svc}"
-
-        if [[ -d "${svc_dir}/tests" ]]; then
-            echo "--- pytest: ${svc} ---"
-            if ! python3 -m pytest "${svc_dir}/tests" -q; then
-                echo "GATE FAIL: pytest failed for ${svc}" >&2
-                gate_failed=true
-            fi
-        fi
-
-        echo "--- docker build: ${svc} ---"
-        if ! docker build --quiet "${svc_dir}" >/dev/null; then
-            echo "GATE FAIL: docker build failed for ${svc}" >&2
-            gate_failed=true
         fi
     done
 
-    if [[ "$gate_failed" == "true" ]]; then
+    struct_log "validate" "$host" "all" "success" "validated"
-        exit 2
+    mark_stage_complete "validate"
-    fi
-    echo "[ok] gate passed"
 }
 
-# ── EXECUTE ──────────────────────────────────────────────────────────────────
+stage_deploy() {
-
+    local host=$1
-execute() {
+    if is_stage_complete "deploy" && [[ "$RESUME" == "true" ]]; then
-    echo "=== EXECUTE ==="
+        log "INFO" "Skipping DEPLOY (already complete)"
-
+        return 0
-    local cmd_output
-    local cmd_exit=0
-
-    if [[ "$TARGET" == "control-plane" ]]; then
-        echo "Running deploy-control-plane.sh --ssh..."
-        cmd_output=$("${REPO_ROOT}/scripts/deploy/deploy-control-plane.sh" --ssh 2>&1) \
-            || cmd_exit=$?
-    else
-        echo "SSHing to ${SSH_HOST}: git pull + deploy-node.sh..."
-        cmd_output=$(ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
-            "${SSH_USER}@${SSH_HOST}" \
-            'cd ~/homelab-codex-ws && git pull && ./scripts/deploy/deploy-node.sh' 2>&1) \
-            || cmd_exit=$?
     fi
 
-    echo "$cmd_output"
+    log "INFO" "Stage: DEPLOY ($host)"
+    set_stage "deploy"
 
-    if echo "$cmd_output" | grep -qF "[sudo] password"; then
+    local last_s=$(get_last_service)
-        echo "" >&2
+    local skip=false
-        echo "ERROR (exit 5): Deploy hit an interactive sudo prompt." >&2
+    if [[ "$RESUME" == "true" && -n "$last_s" ]]; then
-        echo "Run manually:" >&2
+        skip=true
-        if [[ "$TARGET" == "control-plane" ]]; then
-            echo "  ssh -t ${SSH_USER}@${SSH_HOST} 'cd ~/homelab-codex-ws && git pull origin master && cd services/control-plane && bash deploy-local.sh'" >&2
-        else
-            echo "  ssh -t ${SSH_USER}@${SSH_HOST} 'cd ~/homelab-codex-ws && git pull && ./scripts/deploy/deploy-node.sh'" >&2
-        fi
-        exit 5
     fi
 
-    if [[ $cmd_exit -ne 0 ]]; then
+    for service in "${SERVICES[@]}"; do
-        echo "ERROR: Deploy command exited ${cmd_exit}." >&2
+        if [[ "$skip" == "true" ]]; then
-        exit 3
+            if [[ "$service" == "$last_s" ]]; then
-    fi
+                skip=false
-
+                log "INFO" "Resuming from $service..."
-    echo "[ok] execute completed"
+            else
-}
+                log "INFO" "Skipping $service (already processed)"
-
+                continue
-# ── VERIFY ───────────────────────────────────────────────────────────────────
-
-verify() {
-    echo "=== VERIFY ==="
-
-    local ps_output
-    local ps_exit=0
-    ps_output=$(ssh -o "ConnectTimeout=${SSH_TIMEOUT}" -o BatchMode=yes \
-        "${SSH_USER}@${SSH_HOST}" \
-        'docker ps --format "{{.Names}}\t{{.Status}}"' 2>&1) \
-        || ps_exit=$?
-
-    if [[ $ps_exit -ne 0 ]]; then
-        echo "ERROR: docker ps failed on ${SSH_HOST}:" >&2
-        echo "$ps_output" >&2
-        exit 4
-    fi
-
-    echo "$ps_output"
-
-    local failed=false
-
-    local not_up
-    not_up=$(echo "$ps_output" | grep -v '^$' | grep -v $'\tUp' || true)
-    if [[ -n "$not_up" ]]; then
-        echo "ERROR: Containers not in Up state:" >&2
-        echo "$not_up" >&2
-        failed=true
-    fi
-
-    local unhealthy
-    unhealthy=$(echo "$ps_output" | grep '(unhealthy)' || true)
-    if [[ -n "$unhealthy" ]]; then
-        echo "ERROR: Unhealthy containers:" >&2
-        echo "$unhealthy" >&2
-        failed=true
-    fi
-
-    if [[ "$TARGET" == "control-plane" ]]; then
-        for cp_svc in supervisor observer executor operator-ui; do
-            if ! echo "$ps_output" | grep -q "$cp_svc"; then
-                echo "ERROR: control-plane component absent from docker ps: ${cp_svc}" >&2
-                failed=true
             fi
-        done
+        fi
-    fi
 
-    if [[ "$failed" == "true" ]]; then
+        log "INFO" "Deploying $service..."
-        echo "" >&2
+        set_last_service "$service"
-        echo "Full docker ps output above." >&2
-        exit 4
-    fi
 
-    echo "[ok] all containers healthy"
+        if ! run_compose_up "$service"; then
+            struct_log "deploy" "$host" "$service" "fail" "docker_compose_failed"
+            collect_diagnostics "$host" "$service"
+            return 1
+        fi
+
+        struct_log "deploy" "$host" "$service" "success" "deployed"
+    done
+    
+    set_last_service ""
+    mark_stage_complete "deploy"
 }
 
-# ── REPORT ───────────────────────────────────────────────────────────────────
+stage_verify() {
-
+    local host=$1
-report() {
+    if is_stage_complete "verify" && [[ "$RESUME" == "true" ]]; then
-    local mode="${1:-deploy}"
+        log "INFO" "Skipping VERIFY (already complete)"
-    local end_time
+        return 0
-    end_time=$(date +%s)
-    local elapsed
-    elapsed=$(( end_time - START_TIME ))
-    local commit_hash
-    commit_hash=$(git -C "$REPO_ROOT" rev-parse --short HEAD)
-    local gate_s verify_s
-
-    if [[ "$NO_GATE" == "true" ]]; then
-        gate_s="skip"
-    else
-        gate_s="ok"
     fi
 
-    if [[ "$mode" == "dry-run" ]]; then
+    log "INFO" "Stage: VERIFY ($host)"
-        verify_s="skip(dry-run)"
+    set_stage "verify"
-    else
-        verify_s="green"
-    fi
 
-    echo ""
+    for service in "${SERVICES[@]}"; do
-    if [[ "$mode" == "dry-run" ]]; then
+        log "INFO" "Verifying $service..."
-        echo "DRY RUN OK | target=${TARGET} | commit=${commit_hash} | gate=${gate_s} | verify=${verify_s} | ${elapsed}s"
+        local health_script="${REPO_PATH}/services/${service}/healthcheck.sh"
-    else
+        if [[ -f "$health_script" ]]; then
-        echo "DEPLOY OK  | target=${TARGET} | commit=${commit_hash} | gate=${gate_s} | verify=${verify_s} | ${elapsed}s"
+            if ! bash "$health_script"; then
-    fi
+                log "ERROR" "Healthcheck failed for $service"
+                struct_log "verify" "$host" "$service" "fail" "healthcheck_failed"
+                collect_diagnostics "$host" "$service"
+                return 1
+            fi
+        else
+            # Generic check if container is running
+            if ! docker ps --filter "name=$service" --filter "status=running" | grep -q "$service"; then
+                log "ERROR" "Container $service is not running"
+                struct_log "verify" "$host" "$service" "fail" "container_not_running"
+                collect_diagnostics "$host" "$service"
+                return 1
+            fi
+        fi
+        struct_log "verify" "$host" "$service" "success" "verified"
+    done
+    mark_stage_complete "verify"
 }
 
-# ── MAIN ─────────────────────────────────────────────────────────────────────
+stage_complete() {
+    local host=$1
+    log "INFO" "Stage: COMPLETE ($host)"
+    set_stage "complete"
+    struct_log "complete" "$host" "all" "success" "deployment_finished"
+    clear_deployment_state
+}
 
-preflight
+# --- Execution Logic ---
-gate
 
-if [[ "$DRY_RUN" == "true" ]]; then
+run_deployment() {
-    report dry-run
+    local start_stage=$1
-    exit 0
+
+    # Sequential execution from start_stage
+    case "$start_stage" in
+        prepare)
+            stage_prepare "$TARGET_HOST" || return 1
+            ;&
+        validate)
+            stage_validate "$TARGET_HOST" || return 1
+            ;&
+        deploy)
+            stage_deploy "$TARGET_HOST" || return 1
+            ;&
+        verify)
+            stage_verify "$TARGET_HOST" || return 1
+            ;&
+        complete)
+            stage_complete "$TARGET_HOST" || return 1
+            ;;
+        *)
+            log "ERROR" "Invalid stage: $start_stage"
+            return 1
+            ;;
+    esac
+}
+
+# --- Main ---
+
+log "INFO" "--- Homelab Deployment Started (Host: $TARGET_HOST, Service: ${TARGET_SERVICE:-all}) ---"
+
+if ! load_inventory "$TARGET_HOST" "$TARGET_SERVICE"; then
+    log "ERROR" "Failed to load inventory"
+    exit 1
 fi
 
-execute
+EXIT_STATUS=0
-verify
+if [[ "$RESUME" == "true" ]]; then
-report
+    CURRENT=$(get_stage)
+    log "INFO" "Resuming from state: $CURRENT"
+    case "$CURRENT" in
+        prepare|validate|deploy|verify)
+            run_deployment "$CURRENT" || EXIT_STATUS=1
+            ;;
+        complete|none)
+            log "INFO" "No interrupted deployment found. Starting from scratch..."
+            run_deployment "prepare" || EXIT_STATUS=1
+            ;;
+        *)
+            log "INFO" "Unknown state. Starting from prepare..."
+            run_deployment "prepare" || EXIT_STATUS=1
+            ;;
+    esac
+elif [[ -n "$REQUESTED_STAGE" ]]; then
+    if [[ "$REQUESTED_STAGE" == "diagnose" ]]; then
+        collect_diagnostics "$TARGET_HOST" "$TARGET_SERVICE"
+    else
+        run_deployment "$REQUESTED_STAGE" || EXIT_STATUS=1
+    fi
+else
+    # New deployment - clear previous state
+    clear_deployment_state
+    run_deployment "prepare" || EXIT_STATUS=1
+fi
+
+if [[ $EXIT_STATUS -eq 0 ]]; then
+    print_summary "$TARGET_HOST" "SUCCESS"
+    log "INFO" "--- Homelab Deployment Finished Successfully ---"
+else
+    print_summary "$TARGET_HOST" "FAILED"
+    log "ERROR" "--- Homelab Deployment Failed ---"
+    exit 1
+fi