homelab-codex-ws/services/ai-cluster/CUTOVER.md

# ai-cluster — VPS → SOLARIA cutover plan

**Not executed.** This document is the plan for a later session. As of the last
edit the stack runs untouched on the VPS from
`/home/dockeruser/docker/ai-cluster/`, and nothing has been built on SOLARIA.

**Rollback posture:** the VPS stack stays up and untouched through every step
below. It is only stopped in step 7, after SOLARIA has been confirmed healthy and
all consumers have been repointed. Until then, rollback is "do nothing".

## 0. Prerequisites on SOLARIA

```bash
sudo mkdir -p /opt/homelab/config/ai-cluster/mosquitto
```

Populate the secrets file from the template:

```bash
sudo cp services/ai-cluster/env.example /opt/homelab/config/ai-cluster/.env
sudo chmod 600 /opt/homelab/config/ai-cluster/.env
# fill in MQTT_PASSWORD
```

Mosquitto config — the two committed files, plus a `passwd` generated on the node:

```bash
sudo cp services/ai-cluster/src/mosquitto/{mosquitto.conf,acl} \
        /opt/homelab/config/ai-cluster/mosquitto/
docker run --rm -v /opt/homelab/config/ai-cluster/mosquitto:/m eclipse-mosquitto:2 \
  mosquitto_passwd -c -b /m/passwd codex '<same password as MQTT_PASSWORD>'
sudo chmod 600 /opt/homelab/config/ai-cluster/mosquitto/passwd
```

The password must match `MQTT_PASSWORD` in `.env`. Reusing the VPS password keeps
any not-yet-migrated client working during the overlap; rotating it is cleaner but
means every consumer must be updated in the same window.

Check nothing on SOLARIA already holds `100.100.231.104:8000` or `:1883`:

```bash
ss -tlnp | grep -E ':8000|:1883'
```

## 1. Build

```bash
cd ~/homelab-codex-ws/services/ai-cluster
docker compose -f docker-compose.yml \
  -f ../../hosts/solaria/runtime/ai-cluster/docker-compose.override.yml build
```

Four images build from `src/`: openclaw, and three from `src/worker`. Nothing is
pulled from a registry except the `redis:7-alpine` and `eclipse-mosquitto:2` bases.

## 2. Start

```bash
docker compose -f docker-compose.yml \
  -f ../../hosts/solaria/runtime/ai-cluster/docker-compose.override.yml up -d
docker compose ps
```

Expect a container-name collision on `mosquitto` only if something else on SOLARIA
already uses that fixed name — the VPS one is a different host, so no conflict.

## 3. Verify

```bash
./healthcheck.sh
```

Then, individually:

- **openclaw**: `curl -sf http://100.100.231.104:8000/health`
- **MQTT auth**: publish a task and watch for a result —
  ```bash
  mosquitto_sub -h 100.100.231.104 -p 1883 -u codex -P '<pass>' -t 'codex/results' -C 1 &
  mosquitto_pub -h 100.100.231.104 -p 1883 -u codex -P '<pass>' -t 'codex/tasks' \
    -m '{"id":"cutover-1","target":"role:dev","goal":"cutover smoke test"}'
  ```
- **gateway reachability**: confirm `GATEWAY_BASE_URL` actually answers from inside
  a worker — `docker exec ai-cluster-codex-worker-1 python -c "import urllib.request;
  print(urllib.request.urlopen('http://100.108.208.3:8080').status)"`
- **memory**: `docker stats --no-stream` — nothing should sit near its `mem_limit`.
  Reference RSS measured on the VPS: workers 7–10 MiB, openclaw ~25 MiB.
- **service-ops-worker**: confirm it can parse the mounted compose file —
  `docker exec ai-cluster-service-ops-worker-1 docker compose ps`. This is the step
  that catches a missing `/opt/homelab/config/ai-cluster/.env` inside the container.

## 4. Repoint NPM on the VPS

In the NPM admin UI, change the proxy host that currently forwards to the local
openclaw so it forwards to `100.100.231.104` port `8000`.

The VPS reaches SOLARIA over Tailscale; confirm first from the VPS shell:

```bash
curl -sf --max-time 5 http://100.100.231.104:8000/health
```

NPM's config is not in Git — this is a UI change, so note the previous value before
editing it in case of rollback.

## 5. Repoint mosquitto consumers

Recon on 2026-07-27 found **no repo-managed consumer** of the VPS broker at
`100.95.58.48:1883`. Specifically:

| Candidate | Verdict |
|---|---|
| `hosts/chelsty-infra/runtime/stability-agent/` | `MQTT_HOST=mosquitto` — its own **host-networked** broker on CHELSTY-INFRA. Not this broker. Offline-first, must stay that way. |
| `hosts/chelsty-infra/runtime/zigbee2mqtt/`, `frigate/` | Same local CHELSTY broker. Not this broker. |
| `services/stability-agent/env.example` | Default `MQTT_HOST=mosquitto`; no host override sets a VPS IP. |
| `services/mosquitto/` | A **separate** repo manifest (`owner_node: vps`, ports `1883`+`9001`) that describes this same container. Overlaps with this stack — see follow-ups. |
| ai-cluster's own containers | Use the in-network name `mosquitto`; move with the stack, no change needed. |

So the expected consumer list to update is **empty** — but the grep only covers
this repo. Before step 7, confirm there is no out-of-repo client by watching the
broker on the VPS for connections that are not ai-cluster containers:

```bash
# on the VPS
docker logs --tail 200 mosquitto | grep -i 'new client'
```

Any client address that is not a `172.x` container IP from the ai-cluster network
is an external consumer that must be repointed to `100.100.231.104:1883` first.

## 6. Observation window

Leave both stacks up for a while. SOLARIA serves traffic; the VPS stack is idle but
alive. Confirm no errors in `docker compose logs -f` on SOLARIA and that NPM is not
throwing 502s.

## 7. Stop the VPS stack

Only after steps 3–6 are clean:

```bash
# on the VPS
cd /home/dockeruser/docker/ai-cluster
docker compose down
```

`down` without `-v` — it leaves the images, the bind-mounted config, and the
directory in place, so `docker compose up -d` restores the old stack if needed.

## 8. Confirm the memory win on the VPS

```bash
free -m
```

Expected reclaim is modest — the six containers measured ~58 MiB RSS combined on
2026-07-27, not the several hundred MiB the stack was once assumed to cost. The
real benefit is removing four unbounded Python processes from a 4 GiB no-swap box,
not the current byte count. Record the before/after in the session notes.

## 9. Repo cleanup, after the VPS stack is gone

- Delete `/home/dockeruser/docker/ai-cluster/` on the VPS (contains the only copy
  of `.env`, `mosquitto/passwd`, and the dead `codex-worker/` prototype — check
  nothing is still wanted before removing).
- Reconcile `services/mosquitto/` (see follow-ups).
- Update `inventory/topology.yaml`: `ai-cluster` is listed under the VPS services.

## Follow-ups, deliberately not done here

- **`services/mosquitto/` duplicates this stack's broker.** It declares
  `owner_node: vps`, ports 1883+9001, and `/opt/homelab/data/mosquitto/*` paths that
  do not match the running container. Two manifests describe one container. Needs a
  decision: fold it into ai-cluster, or make it a standalone service on SOLARIA that
  ai-cluster depends on.
- **`AGENT_ID=vps-*` on SOLARIA.** Values kept for routing compatibility. Rename to
  `solaria-*` once every task producer is known and can be updated together.
- **`service-ops-worker` has `docker.sock` read-write on the GPU node.** It can
  restart `ollama` and anything else on SOLARIA. The in-code allowlist limits it to
  ps/inspect/logs/restart, but the blast radius is larger here than on the VPS.
  Worth a security review before, not after, cutover.
- **`codex-worker/worker.py` on the VPS** is a dead 609-byte prototype (uses
  `requests`, which is in no requirements file; no Dockerfile; not referenced by
  compose). Not carried into `src/`. Delete it with the directory in step 9, or
  rescue it first if it has value.
- **`telegram-bot`** is defined in the VPS compose but no container runs. Not carried
  over. `telegram_bot.py` still ships because `openclaw/Dockerfile` COPYs it.