diff --git a/docs/infra/ollama-solaria-cutover-2026-07-15.md b/docs/infra/ollama-solaria-cutover-2026-07-15.md new file mode 100644 index 0000000..0a44743 --- /dev/null +++ b/docs/infra/ollama-solaria-cutover-2026-07-15.md @@ -0,0 +1,112 @@ +# Ollama SOLARIA: manual → declarative cutover runbook + +Date: 2026-07-15 +Status: repo changes done (this doc), **live cutover not yet executed** — no SSH +access to SOLARIA from the worktree this was prepared in. Run on SATURN or by +the operator directly on SOLARIA. + +## Background + +`services/ollama/service.yaml` declared `owner_node: solaria` but was never +added to `hosts/solaria/services.yaml` — the file `deploy-node.sh` actually +reads to decide what to bring up on a host. That's the whole gap; Ollama has +been running natively/manually (systemd) the entire time instead. This change +adds the missing `hosts/solaria/services.yaml` entry and fixes the compose +port bind (see below) so `deploy-node.sh` can manage it going forward. + +The running native instance already has models pulled, including large coder +models (`qwen2.5-coder:14b`, `qwen3-coder:30b`, `deepseek-coder:latest`, +`deepcoder:14b`) actively used by `llm-gateway@PIHA`, plus `bge-m3` (pulled +manually ahead of this task for the KB embeddings job). **Do not re-pull — +bind-mount the existing model directory.** + +## Pre-cutover checks (read-only, on SOLARIA) + +1. Confirm current model store path and owning user: + ```bash + sudo systemctl show ollama -p Environment # look for OLLAMA_MODELS override + sudo ls -la /usr/share/ollama/.ollama/models 2>&1 # default for systemd-installed ollama + ls -la ~/.ollama/models 2>&1 # default for a user-run instance + ``` +2. Confirm `nvidia-container-toolkit` is installed and Docker sees the GPU runtime: + ```bash + docker info | grep -i nvidia + nvidia-smi + ``` +3. Confirm the current OLLAMA_HOST bind (how llm-gateway@PIHA is reaching it today): + ```bash + sudo systemctl show ollama -p Environment | grep -o 'OLLAMA_HOST=[^ ]*' + ``` +4. **Timing: do this when llm-gateway@PIHA is not actively serving a request.** + The cutover has a brief availability gap between disabling the native + service and the container passing its healthcheck — pick a quiet window, + don't run it mid-request. + +## Cutover steps + +1. `git pull` on SOLARIA (picks up this branch once merged to master). +2. Populate `services/ollama/.env` from `services/ollama/env.example` + (`TAILSCALE_BIND_IP=100.100.231.104` — verify with `tailscale ip -4`). +3. **Relocate the model store BEFORE the first `docker compose up`:** + ```bash + sudo mkdir -p /opt/homelab/data/ollama + # Adjust source path per the pre-check above — this assumes the systemd + # default. Use rsync (not mv) so nothing is lost if the path guess is wrong; + # delete the source only after the post-start verification below passes. + sudo rsync -a /usr/share/ollama/.ollama/ /opt/homelab/data/ollama/ + sudo chown -R "$(id -u):$(id -g)" /opt/homelab/data/ollama # container runs as root by default; adjust if the image user differs + ``` + An empty bind mount here is a silent failure mode: the container starts + fine, `ollama list` comes back empty, and llm-gateway's `/api/generate` + calls start 404-ing on model names that used to work — no crash, no loud + error, just wrong answers for whoever's calling it next. +4. Disable (not just stop) the native service so it can't come back and fight + the container for port 11434: + ```bash + sudo systemctl disable --now ollama + ``` +5. Bring up the declarative stack: + ```bash + cd ~/homelab-codex-ws + ./scripts/deploy/deploy-node.sh + # or directly: docker compose -f services/ollama/docker-compose.yml --env-file services/ollama/.env up -d + ``` + +## Post-cutover verification + +1. **Model completeness — do this before anything else calls the new instance:** + ```bash + docker exec ollama ollama list + ``` + Confirm it lists ALL of: `qwen2.5-coder:14b`, `qwen3-coder:30b`, + `deepseek-coder:latest`, `deepcoder:14b`, `bge-m3` — not just a subset. If + any are missing, the bind mount points at the wrong source directory; stop + and fix before letting llm-gateway traffic resume (it depends on these + coder models being present). +2. Container is on GPU, not CPU fallback: + ```bash + nvidia-smi # ollama process should appear once a model is loaded + docker exec ollama ollama ps + ``` +3. Embeddings endpoint + vector dimension (deferred check from + `docs/kb/modules/05-faza2-plan.md` §6 step 2): + ```bash + curl -s http://localhost:11434/api/embeddings -d '{"model":"bge-m3","prompt":"test"}' \ + | python3 -c "import json,sys; v=json.load(sys.stdin)['embedding']; print(len(v))" + # expect: 1024 + ``` +4. Reachability from PIHA (llm-gateway's actual path): + ```bash + curl -sf http://solaria:11434/api/tags # from PIHA, over Tailscale + ``` +5. `services/ollama/healthcheck.sh` passes on SOLARIA. + +## Rollback + +If the container fails to come up healthy or the model list is incomplete: +```bash +docker compose -f services/ollama/docker-compose.yml down +sudo systemctl enable --now ollama +``` +The native install's model directory was copied (not moved) in step 3, so it +is untouched and the manual instance comes back exactly as it was. diff --git a/hosts/solaria/services.yaml b/hosts/solaria/services.yaml index 2cddb0f..9fd54f7 100644 --- a/hosts/solaria/services.yaml +++ b/hosts/solaria/services.yaml @@ -13,3 +13,20 @@ services: config_path: /opt/homelab/config/node-agent data_path: /opt/homelab/state logs_path: /opt/homelab/events + + ollama: + role: llm-inference # GPU-backed inference: embeddings (bge-m3) + coder models + deployment_model: docker-compose + exposure: private # Tailscale-only bind (TAILSCALE_BIND_IP) + loopback; consumed by llm-gateway@PIHA over mesh + offline_required: false + depends_on: + local: [] + external: [] + ports: + - name: http + container_port: 11434 + protocol: tcp + runtime: + # .env (TAILSCALE_BIND_IP) lives alongside the compose file, matching llm-gateway + config_path: services/ollama + data_path: /opt/homelab/data/ollama diff --git a/services/ollama/docker-compose.yml b/services/ollama/docker-compose.yml index 2bf73d9..09176a2 100644 --- a/services/ollama/docker-compose.yml +++ b/services/ollama/docker-compose.yml @@ -4,7 +4,13 @@ services: container_name: ollama restart: unless-stopped ports: - - '11434:11434' + # Loopback: service.yaml's healthcheck + healthcheck.sh curl localhost + # directly on the node. Tailscale IP: llm-gateway@PIHA reaches this over + # the mesh as http://solaria:11434 (MagicDNS). No 0.0.0.0 — this is a + # private service, don't expose wider than local + internal network. + # Requires .env (from env.example) next to this file at deploy. + - "127.0.0.1:11434:11434" + - "${TAILSCALE_BIND_IP}:11434:11434" volumes: - /opt/homelab/data/ollama:/root/.ollama deploy: diff --git a/services/ollama/env.example b/services/ollama/env.example index 29f3edf..5983b8f 100644 --- a/services/ollama/env.example +++ b/services/ollama/env.example @@ -1,2 +1,8 @@ -# No specific environment variables required by default. +# Copy to .env next to docker-compose.yml (gitignored); docker compose picks +# it up automatically. Same convention as services/llm-gateway. + +# Tailscale IP of the SOLARIA node. Bind the mesh-facing publish ONLY to this +# IP — never 0.0.0.0. Verify when rebuilding the host: tailscale ip -4. +TAILSCALE_BIND_IP=100.100.231.104 + # CUDA_VISIBLE_DEVICES=0