Ollama SOLARIA: manual → declarative cutover runbook

Date: 2026-07-15 Status: executed 2026-07-15 (CPU-only, see "Wykonanie" below); GPU restored 2026-07-16 once the missing NVIDIA driver was fixed.

Background

services/ollama/service.yaml declared owner_node: solaria but was never added to hosts/solaria/services.yaml — the file deploy-node.sh actually reads to decide what to bring up on a host. That's the whole gap; Ollama has been running natively/manually (systemd) the entire time instead. This change adds the missing hosts/solaria/services.yaml entry and fixes the compose port bind (see below) so deploy-node.sh can manage it going forward.

The running native instance already has models pulled, including large coder models (qwen2.5-coder:14b, qwen3-coder:30b, deepseek-coder:latest, deepcoder:14b) actively used by llm-gateway@PIHA, plus bge-m3 (pulled manually ahead of this task for the KB embeddings job). Do not re-pull — bind-mount the existing model directory.

Pre-cutover checks (read-only, on SOLARIA)

Confirm current model store path and owning user:

sudo systemctl show ollama -p Environment    # look for OLLAMA_MODELS override
sudo ls -la /usr/share/ollama/.ollama/models 2>&1   # default for systemd-installed ollama
ls -la ~/.ollama/models 2>&1                        # default for a user-run instance

Confirm nvidia-container-toolkit is installed and Docker sees the GPU runtime:
```
docker info | grep -i nvidia
nvidia-smi
```
Confirm the current OLLAMA_HOST bind (how llm-gateway@PIHA is reaching it today):
```
sudo systemctl show ollama -p Environment | grep -o 'OLLAMA_HOST=[^ ]*'
```
Timing: do this when llm-gateway@PIHA is not actively serving a request. The cutover has a brief availability gap between disabling the native service and the container passing its healthcheck — pick a quiet window, don't run it mid-request.

Cutover steps

git pull on SOLARIA (picks up this branch once merged to master).
Populate services/ollama/.env from services/ollama/env.example (TAILSCALE_BIND_IP=100.100.231.104 — verify with tailscale ip -4).

Relocate the model store BEFORE the first docker compose up:

sudo mkdir -p /opt/homelab/data/ollama
# Adjust source path per the pre-check above — this assumes the systemd
# default. Use rsync (not mv) so nothing is lost if the path guess is wrong;
# delete the source only after the post-start verification below passes.
sudo rsync -a /usr/share/ollama/.ollama/ /opt/homelab/data/ollama/
sudo chown -R "$(id -u):$(id -g)" /opt/homelab/data/ollama   # container runs as root by default; adjust if the image user differs

An empty bind mount here is a silent failure mode: the container starts fine, ollama list comes back empty, and llm-gateway's /api/generate calls start 404-ing on model names that used to work — no crash, no loud error, just wrong answers for whoever's calling it next.

Disable (not just stop) the native service so it can't come back and fight the container for port 11434:
```
sudo systemctl disable --now ollama
```

Bring up the declarative stack:

cd ~/homelab-codex-ws
./scripts/deploy/deploy-node.sh
# or directly: docker compose -f services/ollama/docker-compose.yml --env-file services/ollama/.env up -d

Post-cutover verification

Model completeness — do this before anything else calls the new instance:
```
docker exec ollama ollama list
```
Confirm it lists ALL of: qwen2.5-coder:14b, qwen3-coder:30b, deepseek-coder:latest, deepcoder:14b, bge-m3 — not just a subset. If any are missing, the bind mount points at the wrong source directory; stop and fix before letting llm-gateway traffic resume (it depends on these coder models being present).

Container is on GPU, not CPU fallback:

nvidia-smi                 # ollama process should appear once a model is loaded
docker exec ollama ollama ps

Embeddings endpoint + vector dimension (deferred check from docs/kb/modules/05-faza2-plan.md §6 step 2):

curl -s http://localhost:11434/api/embeddings -d '{"model":"bge-m3","prompt":"test"}' \
  | python3 -c "import json,sys; v=json.load(sys.stdin)['embedding']; print(len(v))"
# expect: 1024

Reachability from PIHA (llm-gateway's actual path):

curl -sf http://solaria:11434/api/tags   # from PIHA, over Tailscale

services/ollama/healthcheck.sh passes on SOLARIA.

Rollback

If the container fails to come up healthy or the model list is incomplete:

docker compose -f services/ollama/docker-compose.yml down
sudo systemctl enable --now ollama

The native install's model directory was copied (not moved) in step 3, so it is untouched and the manual instance comes back exactly as it was.

(See "Wykonanie" below — the actual run used mv, not rsync, so this rollback description no longer matches what happened live.)

Wykonanie (2026-07-15, GPU restored 2026-07-16)

Deviations from the plan above, discovered while executing it live on SOLARIA:

Step 3 (relocate model store) used mv, not rsync -a. The model store (36G) was moved in one shot rather than copied. Consequence: the rollback described above ("source untouched since copied") does not apply as executed — a real rollback would need the reverse mv (/opt/homelab/data/ollama/* → /usr/share/ollama/.ollama/), not just re-enabling the native systemd service against an already-relocated directory.
Step 2 pre-check (nvidia-container-toolkit) failed the first time: it was not installed. Installed as a prerequisite before continuing — this wasn't previously flagged as something that could be missing.
Step 2 pre-check (driver) failed harder than expected: nvidia-smi did not exist on the host at all — no NVIDIA driver installed on SOLARIA. hosts/solaria/services.yaml had described ollama as GPU-backed for a while; that was aspirational, not actual — Ollama had been running CPU-only the entire time regardless of what the manifest said.
Given the missing driver, the cutover proceeded in CPU-only mode: the deploy.resources GPU reservation was commented out in services/ollama/docker-compose.yml (commit f57a01a), and the driver fix was filed as a backlog item (see docs/backlog.md) blocking the module 5 mail-embedding phase.
2026-07-16: driver fixed. Installed nvidia-driver-595-open from the distro repository — not the old ppa:graphics-drivers/ppa (jammy), which was deactivated by renaming its sources.list.d entry to .disabled. Result: RTX 4070 Ti SUPER 16GB, CUDA 13.2, nvidia-smi working on the host. nvidia-container-toolkit from the earlier prerequisite install was already in place (nvidia runtime registered in daemon.json), so no further toolkit work was needed. The GPU reservation was restored in docker-compose.yml and the container recreated.
GPU vs CPU embedding throughput (measured 2026-07-16): GPU 207ms/embed vs CPU ≈790ms — ~3.8× faster sequential (50 calls, ~600-tok prompt, bge-m3 at 100% GPU per ollama ps). Single-request overhead (HTTP/tokenization) dominates; batching remains the real lever (backlog).

Nota operacyjna: kontener zniknął po reboocie (2026-07-15 wieczór)

Po reboocie hosta wieczorem 2026-07-15 kontener ollama zniknął (docker ps -a nie pokazywał go w ogóle — nie Exited), mimo restart: unless-stopped. Jednorazowe zdarzenie: po boocie 2026-07-16 kontener wstał poprawnie i automatycznie. Przyczyna nie została odtworzona — flagowane jako obserwacja, nie bug do śledzenia, chyba że się powtórzy.

7.7 KiB Raw Permalink Blame History Unescape Escape