homelab-codex-ws/docs/infra/ollama-solaria-cutover-2026-07-15.md

# Ollama SOLARIA: manual → declarative cutover runbook

Date: 2026-07-15
Status: repo changes done (this doc), **live cutover not yet executed** — no SSH
access to SOLARIA from the worktree this was prepared in. Run on SATURN or by
the operator directly on SOLARIA.

## Background

`services/ollama/service.yaml` declared `owner_node: solaria` but was never
added to `hosts/solaria/services.yaml` — the file `deploy-node.sh` actually
reads to decide what to bring up on a host. That's the whole gap; Ollama has
been running natively/manually (systemd) the entire time instead. This change
adds the missing `hosts/solaria/services.yaml` entry and fixes the compose
port bind (see below) so `deploy-node.sh` can manage it going forward.

The running native instance already has models pulled, including large coder
models (`qwen2.5-coder:14b`, `qwen3-coder:30b`, `deepseek-coder:latest`,
`deepcoder:14b`) actively used by `llm-gateway@PIHA`, plus `bge-m3` (pulled
manually ahead of this task for the KB embeddings job). **Do not re-pull —
bind-mount the existing model directory.**

## Pre-cutover checks (read-only, on SOLARIA)

1. Confirm current model store path and owning user:
   ```bash
   sudo systemctl show ollama -p Environment    # look for OLLAMA_MODELS override
   sudo ls -la /usr/share/ollama/.ollama/models 2>&1   # default for systemd-installed ollama
   ls -la ~/.ollama/models 2>&1                        # default for a user-run instance
   ```
2. Confirm `nvidia-container-toolkit` is installed and Docker sees the GPU runtime:
   ```bash
   docker info | grep -i nvidia
   nvidia-smi
   ```
3. Confirm the current OLLAMA_HOST bind (how llm-gateway@PIHA is reaching it today):
   ```bash
   sudo systemctl show ollama -p Environment | grep -o 'OLLAMA_HOST=[^ ]*'
   ```
4. **Timing: do this when llm-gateway@PIHA is not actively serving a request.**
   The cutover has a brief availability gap between disabling the native
   service and the container passing its healthcheck — pick a quiet window,
   don't run it mid-request.

## Cutover steps

1. `git pull` on SOLARIA (picks up this branch once merged to master).
2. Populate `services/ollama/.env` from `services/ollama/env.example`
   (`TAILSCALE_BIND_IP=100.100.231.104` — verify with `tailscale ip -4`).
3. **Relocate the model store BEFORE the first `docker compose up`:**
   ```bash
   sudo mkdir -p /opt/homelab/data/ollama
   # Adjust source path per the pre-check above — this assumes the systemd
   # default. Use rsync (not mv) so nothing is lost if the path guess is wrong;
   # delete the source only after the post-start verification below passes.
   sudo rsync -a /usr/share/ollama/.ollama/ /opt/homelab/data/ollama/
   sudo chown -R "$(id -u):$(id -g)" /opt/homelab/data/ollama   # container runs as root by default; adjust if the image user differs
   ```
   An empty bind mount here is a silent failure mode: the container starts
   fine, `ollama list` comes back empty, and llm-gateway's `/api/generate`
   calls start 404-ing on model names that used to work — no crash, no loud
   error, just wrong answers for whoever's calling it next.
4. Disable (not just stop) the native service so it can't come back and fight
   the container for port 11434:
   ```bash
   sudo systemctl disable --now ollama
   ```
5. Bring up the declarative stack:
   ```bash
   cd ~/homelab-codex-ws
   ./scripts/deploy/deploy-node.sh
   # or directly: docker compose -f services/ollama/docker-compose.yml --env-file services/ollama/.env up -d
   ```

## Post-cutover verification

1. **Model completeness — do this before anything else calls the new instance:**
   ```bash
   docker exec ollama ollama list
   ```
   Confirm it lists ALL of: `qwen2.5-coder:14b`, `qwen3-coder:30b`,
   `deepseek-coder:latest`, `deepcoder:14b`, `bge-m3` — not just a subset. If
   any are missing, the bind mount points at the wrong source directory; stop
   and fix before letting llm-gateway traffic resume (it depends on these
   coder models being present).
2. Container is on GPU, not CPU fallback:
   ```bash
   nvidia-smi                 # ollama process should appear once a model is loaded
   docker exec ollama ollama ps
   ```
3. Embeddings endpoint + vector dimension (deferred check from
   `docs/kb/modules/05-faza2-plan.md` §6 step 2):
   ```bash
   curl -s http://localhost:11434/api/embeddings -d '{"model":"bge-m3","prompt":"test"}' \
     | python3 -c "import json,sys; v=json.load(sys.stdin)['embedding']; print(len(v))"
   # expect: 1024
   ```
4. Reachability from PIHA (llm-gateway's actual path):
   ```bash
   curl -sf http://solaria:11434/api/tags   # from PIHA, over Tailscale
   ```
5. `services/ollama/healthcheck.sh` passes on SOLARIA.

## Rollback

If the container fails to come up healthy or the model list is incomplete:
```bash
docker compose -f services/ollama/docker-compose.yml down
sudo systemctl enable --now ollama
```
The native install's model directory was copied (not moved) in step 3, so it
is untouched and the manual instance comes back exactly as it was.