homelab-codex-ws/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What This Repo Is

GitOps-lite orchestration for a distributed homelab. The repo is the source of truth for infrastructure definitions; runtime state lives at /opt/homelab/ on each execution node and is never committed.

Node Roles

Host	Role
SATURN	Primary control node — only node where commits are made
SOLARIA	GPU/compute/AI workloads
PIHA	Infra, monitoring
VPS	Public ingress, reverse proxy, control plane host
CHELSTY-INFRA	LTE edge hypervisor (site: chelsty); Zigbee2MQTT, Mosquitto, stability-agent — offline-first
CHELSTY-HA	LTE Home Assistant VM (site: chelsty); connects to CHELSTY-INFRA MQTT broker — offline-first

All nodes communicate over Tailscale. CHELSTY-INFRA and CHELSTY-HA have an intermittent LTE uplink; their services must never depend on SATURN, VPS, or Forgejo at runtime. Full node capabilities: hosts/<node>/capabilities.yaml.

Deployment

scripts/deploy/deploy.sh                        # fresh deploy on current node
scripts/deploy/deploy.sh --resume              # resume after interruption
scripts/deploy/deploy.sh --stage verify        # specific stage only
scripts/deploy/deploy.sh --service mosquitto   # specific service only
./scripts/deploy/deploy-control-plane.sh --ssh # SATURN/SOLARIA → VPS
./scripts/deploy/deploy-node.sh chelsty-infra  # CHELSTY nodes (individually)
./scripts/bootstrap/prepare-node.sh            # general node bootstrap
./scripts/bootstrap/chelsty-runtime.sh         # CHELSTY-specific bootstrap
scripts/onboard/onboard.sh --node <name>       # onboard a new node (idempotent, bash)
scripts/onboard/onboard.sh --node <name> --step 00-access   # single step
scripts/onboard/onboard.sh --node <name> --dry-run          # simulate

Pipeline stages: prepare → validate → deploy → verify → diagnose (on failure) → complete. Stage state persisted in /opt/homelab/state/deploy/.

Node Onboarding

New nodes are onboarded via scripts/onboard/ — an idempotent bash tool driven by hosts/<node>/node.yaml manifests (no Ansible). See scripts/onboard/README.md for the full schema, step status table, and gotchas.

Key fields in node.yaml: ssh_user, first_contact (LAN IP — not .local), tailscale.hostname, deploy_autonomy, git_control, hardware.*.

Service Structure

Every service must follow this layout:

services/<service>/
├── docker-compose.yml
├── service.yaml       # Machine-readable contract (primary source of truth for agents)
├── README.md
├── env.example        # Template — never commit actual secrets
└── healthcheck.sh     # Returns 0 (healthy) or 1 (unhealthy)

service.yaml defines owner_node, exposure, dependencies, healthcheck, restart_policy, persistence.paths, and runtime.env_vars. This is what AI agents read to understand how to manage a service.

Host-specific runtime config and secrets live at /opt/homelab/config/<service>/ on the target node (not in Git). Docker Compose overrides are version-controlled at hosts/<node>/runtime/<service>/docker-compose.override.yml in this repo and applied during deployment.

Shared Python Libraries (packages/)

Reusable Python packages that are not deployed services live in packages/<lib>/. They have a pyproject.toml and src/ layout but no docker-compose.yml or service.yaml.

Services and jobs that depend on them install via Dockerfile:

COPY packages/<lib>/ /packages/<lib>/
RUN pip install /packages/<lib>/

First library: packages/kb-mail/ — KB envelope model, asyncpg DB helpers, append-only .eml archive helper.

Agent System Architecture

The platform uses a multi-agent model with human-in-the-loop for destructive actions:

1. Stability Agent (services/stability-agent/) — Per-node watchdog. Monitors Docker containers, disk, Tailscale, MQTT. Emits filesystem events. Does NOT restart services autonomously.
2. Observer (services/control-plane/src/) — Synthesizes world state from events into /opt/homelab/world/{nodes,services,deployments,incidents}.json.
3. Supervisor — Detects drift between desired state (from hosts/*/services.yaml) and actual state (from Observer output). Writes pending action JSON files.
4. Executor — Executes actions only after they transition to approved.
5. Operator UI + Telegram Bot — Operators review and approve/reject pending actions.

Action approval flow

Agent → /opt/homelab/actions/pending/<id>.json
      → Telegram notification → Operator approves
      → /opt/homelab/actions/approved/<id>.json
      → Executor dispatches to the target node → completed / failed

The executor never connects to a node (deliberate — see docs/backlog.md "Remediacja floty bez SSH"). It writes a dispatch file that the node collects:

Action type	Inbox	Executed on the node by
container_restart	actions/dispatch/<node>/	node-agent (own docker socket)
redeploy	actions/deploy/<node>/	deploy-runner, host-level systemd (jobs/deploy-runner/)

Both report back with an action_result event, which the executor turns into completed / failed. disk_cleanup and alert_only still resolve inside the executor.

Agents must never execute destructive actions (restarts, deploys, config changes) without a corresponding approved action file.

Event System

Events are append-only JSON lines at /opt/homelab/events/YYYY-MM-DD/<node>/events.jsonl.

Emit via scripts/lib/events.sh (shell) or scripts/lib/events.py (Python).

Normalized event types: deployment_started/completed/failed, service_unhealthy/recovered, node_offline/online, healthcheck_failed, remediation_started/completed.

Supervisor event routing table

Event type	Source	Action generated	Cooldown
containers_not_running	stability-agent	container_restart	dedup via stable ID
healthcheck_failed	node-agent	container_restart	dedup via stable ID
service_unhealthy / other	stability-agent	redeploy → dispatched to the node's deploy-runner (jobs/deploy-runner/)	dedup via stable ID
disk_pressure (high)	stability-agent	disk_cleanup	dedup via stable ID
ha_websocket_dead	ha-diag-agent	container_restart (homeassistant)	30 min after completion
ha_websocket_recovered	ha-diag-agent	cancels matching restart	—
ha_integration_failed	ha-diag-agent	alert_only	1 hour
ha_entity_unavailable_long	ha-diag-agent	alert_only	1 hour
ha_automation_failing	ha-diag-agent	alert_only	1 hour
ha_update_available	ha-diag-agent	alert_only	1 hour
ha_recorder_lag	ha-diag-agent	alert_only	1 hour
ha_system_health_degraded	ha-diag-agent	alert_only	1 hour

HA events are routed directly from the events directory by the supervisor (not via world-state drift loop) to avoid conflicts with stability-agent's independent container health tracking. HA events are suppressed if homeassistant had a containers_not_running incident within the last 5 minutes (planned restart/update in progress).

Discovery Entry Points for Agents

When exploring the system, use these files in order:

1. inventory/topology.yaml — node list, roles, mesh type
2. hosts/<node>/capabilities.yaml — hardware and software constraints
3. hosts/<node>/services.yaml — desired services and exposure classes for that host
4. services/<service>/service.yaml — operational contract for a service

VPS-Specific Rules

VPS has 4 GiB RAM, no swap. Every repo-managed service must declare memory limits in its hosts/vps/runtime/<service>/docker-compose.override.yml.

Memory limit convention

Use top-level Compose properties (not deploy.resources.limits, which requires Swarm mode):

services:
  myservice:
    mem_limit: 256m      # cgroup ceiling; Docker restarts on breach
    oom_score_adj: -900  # host kernel OOM-killer will not pick this container

Rules:

• Control-plane containers (executor, observer, supervisor, operator-ui), node-agent, stability-agent: always set oom_score_adj: -900 — these must never be a system-level OOM victim.
• mem_limit still applies even with oom_score_adj: -900; the cgroup OOM killer is independent of the host OOM killer and will restart the container via Docker when the limit is exceeded.
• Budget: OS+Docker reserves ~800 MiB; sum of all mem_limit values must stay ≤ 3200 MiB (3.1 GiB).

Repo-managed services on VPS

All VPS services are now GitOps-managed. Service definitions live in services/<name>/docker-compose.yml; host-specific overrides (mem_limit, env) live in hosts/vps/runtime/<name>/docker-compose.override.yml.

Service	Compose stack	Data path
npm	services/npm/	/home/dockeruser/docker/npm/{data,letsencrypt} (bind mount)
outline	services/outline/	Docker named volumes: outline_outline_storage, outline_postgres_data, outline_redis_data
joplin	services/joplin/	Docker named volume: joplin_postgres_data
ai-cluster	services/ai-cluster/	Mosquitto config bind: /home/dockeruser/docker/ai-cluster/mosquitto/

Data migration rule: data paths stay in place at cutover. Never move volumes or bind-mount sources without a dedicated migration plan.

Cutover checklist (before running docker compose up for any migrated service):

1. git pull on VPS
2. Populate /opt/homelab/config/<service>/.env from the env.example template
3. For ai-cluster: copy /home/dockeruser/docker/ai-cluster/.env to /opt/homelab/config/ai-cluster/.env
4. For mosquitto: config stays at old bind path until explicitly migrated
5. Verify named volumes exist: docker volume ls | grep <project>

ai-cluster architectural note: compute workloads (codex-worker, planner-worker) belong on SOLARIA (GPU/compute node), not the 4 GB ingress VPS. Migrate when feasible; for now, hard mem_limits contain the blast radius.

CHELSTY-Specific Rules

• Zigbee coordinator is SLZB-06U over TCP (192.168.1.105:6638, ezsp adapter). Never use /dev/ttyUSB0.
• CHELSTY nodes run docker-compose v1 (1.29.2) — use docker-compose (hyphenated), not docker compose.
• Critical backup sets: HA config+data, Zigbee2MQTT config+db+network key, Mosquitto config+persistence, SLZB-06U coordinator state.

Runtime Path Conventions

/opt/homelab/ layout on each node:

• data/<service>/ — persistent volumes
• config/<service>/ — secrets and host-local overrides (not in Git)
• logs/<service>/ — service logs
• state/ — deployment stage markers, agent heartbeats
• events/ — append-only event store
• world/ — Observer output (synthesized state)
• actions/ — pending / approved / running / completed / failed

Definition of Done (serwisy)

Before any new or changed service is considered ready:

1. docker build + smoke run — build the image locally and run it for a few seconds; confirm the process starts its main loop without crashing. This catches packaging/import errors (e.g. ModuleNotFoundError) before they reach a node.
2. pytest — run the service's test suite. If no tests exist yet, add a minimal one (at minimum: import passes, core logic has at least one case). Tests live in services/<service>/tests/.
3. Never commit or deploy code that has never been run. If a smoke run or test fails, fix it first.

Naming Conventions

• Hosts: ALL CAPS (SATURN, PIHA)
• Services: kebab-case (stability-agent, zigbee2mqtt)
• Container names must match service names
• Always restart: unless-stopped unless service.yaml says otherwise

Multi-agent worktree mode

~/homelab-codex-ws (main checkout) is deploy-only and belongs to the human operator. Parallel agent tasks run in isolated git worktrees created by scripts/dev/agent.sh new <name>.

DISCIPLINE RULE — enforced after 2026-06-08 session violation: All feature/implementation work MUST happen in a task worktree, never directly in the main checkout. The main checkout is for reading context and running deploys only. If you are about to create a new branch or make implementation commits while pwd is ~/homelab-codex-ws, stop and ask the operator to run agent.sh new <name> first.

If .agent-task exists in your current working directory, you are in a task worktree. You must immediately read .agent-task and load .claude/skills/worktree-aware/SKILL.md before taking any action. That skill defines all branch-hygiene rules for task worktrees.

Worktree lifecycle commands: agent.sh new | list | merge | clean. Agents never invoke these — only the human does.