homelab-codex-ws/services/ai-cluster/README.md

# ai-cluster

Small MQTT/Redis task pipeline: an API front-end (`openclaw`) hands tasks to
role-scoped Python workers.

**Status: authored, not deployed.** The stack currently runs on the **VPS**, outside
GitOps, from `/home/dockeruser/docker/ai-cluster/`. This directory is the target
definition for **SOLARIA**. Nothing here has been built or started yet — see
[CUTOVER.md](CUTOVER.md).

## Why SOLARIA

These are compute workloads pointed at an LLM gateway. The VPS is a 4 GiB
public-ingress box with no swap; SOLARIA is the 64 GiB compute/GPU node. The
earlier plan (branch `feat/vps-service-migration`) was to bring the stack into
GitOps *in place* on the VPS. That intermediate step is skipped: the stack moves
straight to SOLARIA, and the VPS keeps only the ingress role via NPM.

## Architecture

```
client ──HTTP──> openclaw ──┬── Redis  (primary queue, result store)
                            └── MQTT   (secondary transport, codex/tasks)
                                  │
                                  ├── codex-worker        (ROLE=dev)
                                  ├── planner-worker      (ROLE=planner)
                                  └── service-ops-worker  (ROLE=service-ops)
```

Tasks are routed by a `target` field: `role:<ROLE>` or an exact `AGENT_ID`.
Results go back to Redis (`result:{id}`, read by `GET /result/{id}`) and to the
`codex/results` MQTT topic.

## Layout

| Path | What |
|---|---|
| `src/openclaw/` | FastAPI app + build context for the `openclaw` image |
| `src/worker/` | Three Dockerfiles over one context: `Dockerfile` (codex-worker), `Dockerfile.planner`, `Dockerfile.service-ops` |
| `src/mosquitto/` | `mosquitto.conf` + `acl` templates — `passwd` is generated on the node, never committed |
| `docker-compose.yml` | Base stack; builds from `src/`, no registry pulls |
| `../../hosts/solaria/runtime/ai-cluster/docker-compose.override.yml` | mem limits + the repo-path bind mount |

## Networking

| What | Bind | Reachable from |
|---|---|---|
| openclaw API | `100.100.231.104:8000` | Tailscale mesh only |
| mosquitto | `100.100.231.104:1883` | Tailscale mesh only, auth required |

Neither is published on `0.0.0.0`. Public access to openclaw goes through NPM on
the VPS, which proxies over Tailscale to `100.100.231.104:8000`.

There is **no loopback bind** on 8000 — this deviates from the `ollama` compose on
the same node, which binds both `127.0.0.1` and the Tailscale IP. `healthcheck.sh`
therefore probes the Tailscale IP. If you'd rather have the loopback convenience
bind, add it in the host override.

## Gotcha: `piha` does not resolve

`GATEWAY_BASE_URL` defaults to `http://100.108.208.3:8080` (PIHA's Tailscale IP),
**not** `http://piha:8080`. Tailscale DNS is disabled on SOLARIA:

```
$ tailscale dns status
Tailscale DNS: disabled.
```

so no MagicDNS name resolves — not the short name, not the FQDN. The same check on
the VPS shows `piha` never resolved there either, so the original default was
already dead. Use IPs in this stack.

## Secrets

Everything sensitive comes from `/opt/homelab/config/ai-cluster/.env` via
`env_file:` — there is no `${VAR:?}` interpolation in the compose file, so the
stack does not care which directory you run `docker compose` from. Start from
[`env.example`](env.example).

Mosquitto's `passwd` is generated on the node and lives next to the config at
`/opt/homelab/config/ai-cluster/mosquitto/passwd`.

## Health

```bash
./healthcheck.sh          # containers + openclaw /health + mosquitto port
```

`openclaw` also has a container-level healthcheck. It uses `python -c urllib...`
rather than `wget`/`curl` because the `python:3.12-slim` base has neither —
verified against the running image.