homelab-codex-ws/services/ai-cluster/CUTOVER.md

ai-cluster — VPS → SOLARIA cutover plan

Not executed. This document is the plan for a later session. As of the last edit the stack runs untouched on the VPS from /home/dockeruser/docker/ai-cluster/, and nothing has been built on SOLARIA.

Rollback posture: the VPS stack stays up and untouched through every step below. It is only stopped in step 7, after SOLARIA has been confirmed healthy and all consumers have been repointed. Until then, rollback is "do nothing".

0. Prerequisites on SOLARIA

sudo mkdir -p /opt/homelab/config/ai-cluster/mosquitto

Populate the secrets file from the template:

sudo cp services/ai-cluster/env.example /opt/homelab/config/ai-cluster/.env
sudo chmod 600 /opt/homelab/config/ai-cluster/.env
# fill in MQTT_PASSWORD

Mosquitto config — the two committed files, plus a passwd generated on the node:

sudo cp services/ai-cluster/src/mosquitto/{mosquitto.conf,acl} \
        /opt/homelab/config/ai-cluster/mosquitto/
docker run --rm -v /opt/homelab/config/ai-cluster/mosquitto:/m eclipse-mosquitto:2 \
  mosquitto_passwd -c -b /m/passwd codex '<same password as MQTT_PASSWORD>'
sudo chmod 600 /opt/homelab/config/ai-cluster/mosquitto/passwd

The password must match MQTT_PASSWORD in .env. Reusing the VPS password keeps any not-yet-migrated client working during the overlap; rotating it is cleaner but means every consumer must be updated in the same window.

Check nothing on SOLARIA already holds 100.100.231.104:8000 or :1883:

ss -tlnp | grep -E ':8000|:1883'

1. Build

cd ~/homelab-codex-ws/services/ai-cluster
docker compose -f docker-compose.yml \
  -f ../../hosts/solaria/runtime/ai-cluster/docker-compose.override.yml build

Four images build from src/: openclaw, and three from src/worker. Nothing is pulled from a registry except the redis:7-alpine and eclipse-mosquitto:2 bases.

2. Start

docker compose -f docker-compose.yml \
  -f ../../hosts/solaria/runtime/ai-cluster/docker-compose.override.yml up -d
docker compose ps

Expect a container-name collision on mosquitto only if something else on SOLARIA already uses that fixed name — the VPS one is a different host, so no conflict.

3. Verify

./healthcheck.sh

Then, individually:

• openclaw: curl -sf http://100.100.231.104:8000/health
• MQTT auth: publish a task and watch for a result —

  mosquitto_sub -h 100.100.231.104 -p 1883 -u codex -P '<pass>' -t 'codex/results' -C 1 &
  mosquitto_pub -h 100.100.231.104 -p 1883 -u codex -P '<pass>' -t 'codex/tasks' \
    -m '{"id":"cutover-1","target":"role:dev","goal":"cutover smoke test"}'
  

• gateway reachability: confirm GATEWAY_BASE_URL actually answers from inside a worker — docker exec ai-cluster-codex-worker-1 python -c "import urllib.request; print(urllib.request.urlopen('http://100.108.208.3:8080').status)"
• memory: docker stats --no-stream — nothing should sit near its mem_limit. Reference RSS measured on the VPS: workers 7–10 MiB, openclaw ~25 MiB.
• service-ops-worker: confirm it can parse the mounted compose file — docker exec ai-cluster-service-ops-worker-1 docker compose ps. This is the step that catches a missing /opt/homelab/config/ai-cluster/.env inside the container.

4. Repoint NPM on the VPS

In the NPM admin UI, change the proxy host that currently forwards to the local openclaw so it forwards to 100.100.231.104 port 8000.

The VPS reaches SOLARIA over Tailscale; confirm first from the VPS shell:

curl -sf --max-time 5 http://100.100.231.104:8000/health

NPM's config is not in Git — this is a UI change, so note the previous value before editing it in case of rollback.

5. Repoint mosquitto consumers

Recon on 2026-07-27 found no repo-managed consumer of the VPS broker at 100.95.58.48:1883. Specifically:

Candidate	Verdict
hosts/chelsty-infra/runtime/stability-agent/	MQTT_HOST=mosquitto — its own host-networked broker on CHELSTY-INFRA. Not this broker. Offline-first, must stay that way.
hosts/chelsty-infra/runtime/zigbee2mqtt/, frigate/	Same local CHELSTY broker. Not this broker.
services/stability-agent/env.example	Default MQTT_HOST=mosquitto; no host override sets a VPS IP.
services/mosquitto/	A separate repo manifest (owner_node: vps, ports 1883+9001) that describes this same container. Overlaps with this stack — see follow-ups.
ai-cluster's own containers	Use the in-network name mosquitto; move with the stack, no change needed.

So the expected consumer list to update is empty — but the grep only covers this repo. Before step 7, confirm there is no out-of-repo client by watching the broker on the VPS for connections that are not ai-cluster containers:

# on the VPS
docker logs --tail 200 mosquitto | grep -i 'new client'

Any client address that is not a 172.x container IP from the ai-cluster network is an external consumer that must be repointed to 100.100.231.104:1883 first.

6. Observation window

Leave both stacks up for a while. SOLARIA serves traffic; the VPS stack is idle but alive. Confirm no errors in docker compose logs -f on SOLARIA and that NPM is not throwing 502s.

7. Stop the VPS stack

Only after steps 3–6 are clean:

# on the VPS
cd /home/dockeruser/docker/ai-cluster
docker compose down

down without -v — it leaves the images, the bind-mounted config, and the directory in place, so docker compose up -d restores the old stack if needed.

8. Confirm the memory win on the VPS

free -m

Expected reclaim is modest — the six containers measured ~58 MiB RSS combined on 2026-07-27, not the several hundred MiB the stack was once assumed to cost. The real benefit is removing four unbounded Python processes from a 4 GiB no-swap box, not the current byte count. Record the before/after in the session notes.

9. Repo cleanup, after the VPS stack is gone

• Delete /home/dockeruser/docker/ai-cluster/ on the VPS (contains the only copy of .env, mosquitto/passwd, and the dead codex-worker/ prototype — check nothing is still wanted before removing).
• Reconcile services/mosquitto/ (see follow-ups).
• Update inventory/topology.yaml: ai-cluster is listed under the VPS services.

Follow-ups, deliberately not done here

• services/mosquitto/ duplicates this stack's broker. It declares owner_node: vps, ports 1883+9001, and /opt/homelab/data/mosquitto/* paths that do not match the running container. Two manifests describe one container. Needs a decision: fold it into ai-cluster, or make it a standalone service on SOLARIA that ai-cluster depends on.
• AGENT_ID=vps-* on SOLARIA. Values kept for routing compatibility. Rename to solaria-* once every task producer is known and can be updated together.
• service-ops-worker has docker.sock read-write on the GPU node. It can restart ollama and anything else on SOLARIA. The in-code allowlist limits it to ps/inspect/logs/restart, but the blast radius is larger here than on the VPS. Worth a security review before, not after, cutover.
• codex-worker/worker.py on the VPS is a dead 609-byte prototype (uses requests, which is in no requirements file; no Dockerfile; not referenced by compose). Not carried into src/. Delete it with the directory in step 9, or rescue it first if it has value.
• telegram-bot is defined in the VPS compose but no container runs. Not carried over. telegram_bot.py still ships because openclaw/Dockerfile COPYs it.