homelab-codex-ws/docs/infra/ollama-solaria-cutover-2026-07-15.md

Ollama SOLARIA: manual → declarative cutover runbook

Date: 2026-07-15 Status: repo changes done (this doc), live cutover not yet executed — no SSH access to SOLARIA from the worktree this was prepared in. Run on SATURN or by the operator directly on SOLARIA.

Background

services/ollama/service.yaml declared owner_node: solaria but was never added to hosts/solaria/services.yaml — the file deploy-node.sh actually reads to decide what to bring up on a host. That's the whole gap; Ollama has been running natively/manually (systemd) the entire time instead. This change adds the missing hosts/solaria/services.yaml entry and fixes the compose port bind (see below) so deploy-node.sh can manage it going forward.

The running native instance already has models pulled, including large coder models (qwen2.5-coder:14b, qwen3-coder:30b, deepseek-coder:latest, deepcoder:14b) actively used by llm-gateway@PIHA, plus bge-m3 (pulled manually ahead of this task for the KB embeddings job). Do not re-pull — bind-mount the existing model directory.

Pre-cutover checks (read-only, on SOLARIA)

1. Confirm current model store path and owning user:

   sudo systemctl show ollama -p Environment    # look for OLLAMA_MODELS override
   sudo ls -la /usr/share/ollama/.ollama/models 2>&1   # default for systemd-installed ollama
   ls -la ~/.ollama/models 2>&1                        # default for a user-run instance
   

2. Confirm nvidia-container-toolkit is installed and Docker sees the GPU runtime:

   docker info | grep -i nvidia
   nvidia-smi
   

3. Confirm the current OLLAMA_HOST bind (how llm-gateway@PIHA is reaching it today):

   sudo systemctl show ollama -p Environment | grep -o 'OLLAMA_HOST=[^ ]*'
   

4. Timing: do this when llm-gateway@PIHA is not actively serving a request. The cutover has a brief availability gap between disabling the native service and the container passing its healthcheck — pick a quiet window, don't run it mid-request.

Cutover steps

1. git pull on SOLARIA (picks up this branch once merged to master).
2. Populate services/ollama/.env from services/ollama/env.example (TAILSCALE_BIND_IP=100.100.231.104 — verify with tailscale ip -4).
3. Relocate the model store BEFORE the first docker compose up:

   sudo mkdir -p /opt/homelab/data/ollama
   # Adjust source path per the pre-check above — this assumes the systemd
   # default. Use rsync (not mv) so nothing is lost if the path guess is wrong;
   # delete the source only after the post-start verification below passes.
   sudo rsync -a /usr/share/ollama/.ollama/ /opt/homelab/data/ollama/
   sudo chown -R "$(id -u):$(id -g)" /opt/homelab/data/ollama   # container runs as root by default; adjust if the image user differs
   

   An empty bind mount here is a silent failure mode: the container starts fine, ollama list comes back empty, and llm-gateway's /api/generate calls start 404-ing on model names that used to work — no crash, no loud error, just wrong answers for whoever's calling it next.
4. Disable (not just stop) the native service so it can't come back and fight the container for port 11434:

   sudo systemctl disable --now ollama
   

5. Bring up the declarative stack:

   cd ~/homelab-codex-ws
   ./scripts/deploy/deploy-node.sh
   # or directly: docker compose -f services/ollama/docker-compose.yml --env-file services/ollama/.env up -d
   

Post-cutover verification

1. Model completeness — do this before anything else calls the new instance:

   docker exec ollama ollama list
   

   Confirm it lists ALL of: qwen2.5-coder:14b, qwen3-coder:30b, deepseek-coder:latest, deepcoder:14b, bge-m3 — not just a subset. If any are missing, the bind mount points at the wrong source directory; stop and fix before letting llm-gateway traffic resume (it depends on these coder models being present).
2. Container is on GPU, not CPU fallback:

   nvidia-smi                 # ollama process should appear once a model is loaded
   docker exec ollama ollama ps
   

3. Embeddings endpoint + vector dimension (deferred check from docs/kb/modules/05-faza2-plan.md §6 step 2):

   curl -s http://localhost:11434/api/embeddings -d '{"model":"bge-m3","prompt":"test"}' \
     | python3 -c "import json,sys; v=json.load(sys.stdin)['embedding']; print(len(v))"
   # expect: 1024
   

4. Reachability from PIHA (llm-gateway's actual path):

   curl -sf http://solaria:11434/api/tags   # from PIHA, over Tailscale
   

5. services/ollama/healthcheck.sh passes on SOLARIA.

Rollback

If the container fails to come up healthy or the model list is incomplete:

docker compose -f services/ollama/docker-compose.yml down
sudo systemctl enable --now ollama

The native install's model directory was copied (not moved) in step 3, so it is untouched and the manual instance comes back exactly as it was.