fix(ollama): declare service on SOLARIA, restrict bind, add cutover runbook

Ollama's service.yaml/docker-compose.yml declared owner_node: solaria but
was never added to hosts/solaria/services.yaml, the manifest deploy-node.sh
actually reads — so it stayed running manually/natively instead of via the
declarative pipeline. Add the missing entry.

Also switch the port publish from 0.0.0.0 to loopback + TAILSCALE_BIND_IP
(same convention as llm-gateway@PIHA), since this is a private service and
llm-gateway@PIHA is the only consumer beyond the host itself.

Live cutover (relocating the existing native model store, disabling the
systemd unit, bringing up the container, verifying bge-m3 embeddings +
GPU use) is documented in docs/infra/ollama-solaria-cutover-2026-07-15.md
but not executed here — no SSH access to SOLARIA from this worktree.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/infra/ollama-solaria-cutover-2026-07-15.md

@@ -0,0 +1,112 @@
+# Ollama SOLARIA: manual → declarative cutover runbook
+
+Date: 2026-07-15
+Status: repo changes done (this doc), **live cutover not yet executed** — no SSH
+access to SOLARIA from the worktree this was prepared in. Run on SATURN or by
+the operator directly on SOLARIA.
+
+## Background
+
+`services/ollama/service.yaml` declared `owner_node: solaria` but was never
+added to `hosts/solaria/services.yaml` — the file `deploy-node.sh` actually
+reads to decide what to bring up on a host. That's the whole gap; Ollama has
+been running natively/manually (systemd) the entire time instead. This change
+adds the missing `hosts/solaria/services.yaml` entry and fixes the compose
+port bind (see below) so `deploy-node.sh` can manage it going forward.
+
+The running native instance already has models pulled, including large coder
+models (`qwen2.5-coder:14b`, `qwen3-coder:30b`, `deepseek-coder:latest`,
+`deepcoder:14b`) actively used by `llm-gateway@PIHA`, plus `bge-m3` (pulled
+manually ahead of this task for the KB embeddings job). **Do not re-pull —
+bind-mount the existing model directory.**
+
+## Pre-cutover checks (read-only, on SOLARIA)
+
+1. Confirm current model store path and owning user:
+   ```bash
+   sudo systemctl show ollama -p Environment    # look for OLLAMA_MODELS override
+   sudo ls -la /usr/share/ollama/.ollama/models 2>&1   # default for systemd-installed ollama
+   ls -la ~/.ollama/models 2>&1                        # default for a user-run instance
+   ```
+2. Confirm `nvidia-container-toolkit` is installed and Docker sees the GPU runtime:
+   ```bash
+   docker info | grep -i nvidia
+   nvidia-smi
+   ```
+3. Confirm the current OLLAMA_HOST bind (how llm-gateway@PIHA is reaching it today):
+   ```bash
+   sudo systemctl show ollama -p Environment | grep -o 'OLLAMA_HOST=[^ ]*'
+   ```
+4. **Timing: do this when llm-gateway@PIHA is not actively serving a request.**
+   The cutover has a brief availability gap between disabling the native
+   service and the container passing its healthcheck — pick a quiet window,
+   don't run it mid-request.
+
+## Cutover steps
+
+1. `git pull` on SOLARIA (picks up this branch once merged to master).
+2. Populate `services/ollama/.env` from `services/ollama/env.example`
+   (`TAILSCALE_BIND_IP=100.100.231.104` — verify with `tailscale ip -4`).
+3. **Relocate the model store BEFORE the first `docker compose up`:**
+   ```bash
+   sudo mkdir -p /opt/homelab/data/ollama
+   # Adjust source path per the pre-check above — this assumes the systemd
+   # default. Use rsync (not mv) so nothing is lost if the path guess is wrong;
+   # delete the source only after the post-start verification below passes.
+   sudo rsync -a /usr/share/ollama/.ollama/ /opt/homelab/data/ollama/
+   sudo chown -R "$(id -u):$(id -g)" /opt/homelab/data/ollama   # container runs as root by default; adjust if the image user differs
+   ```
+   An empty bind mount here is a silent failure mode: the container starts
+   fine, `ollama list` comes back empty, and llm-gateway's `/api/generate`
+   calls start 404-ing on model names that used to work — no crash, no loud
+   error, just wrong answers for whoever's calling it next.
+4. Disable (not just stop) the native service so it can't come back and fight
+   the container for port 11434:
+   ```bash
+   sudo systemctl disable --now ollama
+   ```
+5. Bring up the declarative stack:
+   ```bash
+   cd ~/homelab-codex-ws
+   ./scripts/deploy/deploy-node.sh
+   # or directly: docker compose -f services/ollama/docker-compose.yml --env-file services/ollama/.env up -d
+   ```
+
+## Post-cutover verification
+
+1. **Model completeness — do this before anything else calls the new instance:**
+   ```bash
+   docker exec ollama ollama list
+   ```
+   Confirm it lists ALL of: `qwen2.5-coder:14b`, `qwen3-coder:30b`,
+   `deepseek-coder:latest`, `deepcoder:14b`, `bge-m3` — not just a subset. If
+   any are missing, the bind mount points at the wrong source directory; stop
+   and fix before letting llm-gateway traffic resume (it depends on these
+   coder models being present).
+2. Container is on GPU, not CPU fallback:
+   ```bash
+   nvidia-smi                 # ollama process should appear once a model is loaded
+   docker exec ollama ollama ps
+   ```
+3. Embeddings endpoint + vector dimension (deferred check from
+   `docs/kb/modules/05-faza2-plan.md` §6 step 2):
+   ```bash
+   curl -s http://localhost:11434/api/embeddings -d '{"model":"bge-m3","prompt":"test"}' \
+     | python3 -c "import json,sys; v=json.load(sys.stdin)['embedding']; print(len(v))"
+   # expect: 1024
+   ```
+4. Reachability from PIHA (llm-gateway's actual path):
+   ```bash
+   curl -sf http://solaria:11434/api/tags   # from PIHA, over Tailscale
+   ```
+5. `services/ollama/healthcheck.sh` passes on SOLARIA.
+
+## Rollback
+
+If the container fails to come up healthy or the model list is incomplete:
+```bash
+docker compose -f services/ollama/docker-compose.yml down
+sudo systemctl enable --now ollama
+```
+The native install's model directory was copied (not moved) in step 3, so it
+is untouched and the manual instance comes back exactly as it was.

hosts/solaria/services.yaml

@@ -13,3 +13,20 @@ services:
       config_path: /opt/homelab/config/node-agent
       data_path: /opt/homelab/state
       logs_path: /opt/homelab/events
+
+  ollama:
+    role: llm-inference          # GPU-backed inference: embeddings (bge-m3) + coder models
+    deployment_model: docker-compose
+    exposure: private            # Tailscale-only bind (TAILSCALE_BIND_IP) + loopback; consumed by llm-gateway@PIHA over mesh
+    offline_required: false
+    depends_on:
+      local: []
+      external: []
+    ports:
+      - name: http
+        container_port: 11434
+        protocol: tcp
+    runtime:
+      # .env (TAILSCALE_BIND_IP) lives alongside the compose file, matching llm-gateway
+      config_path: services/ollama
+      data_path: /opt/homelab/data/ollama

services/ollama/docker-compose.yml

@@ -4,7 +4,13 @@ services:
     container_name: ollama
     restart: unless-stopped
     ports:
-      - '11434:11434'
+      # Loopback: service.yaml's healthcheck + healthcheck.sh curl localhost
+      # directly on the node. Tailscale IP: llm-gateway@PIHA reaches this over
+      # the mesh as http://solaria:11434 (MagicDNS). No 0.0.0.0 — this is a
+      # private service, don't expose wider than local + internal network.
+      # Requires .env (from env.example) next to this file at deploy.
+      - "127.0.0.1:11434:11434"
+      - "${TAILSCALE_BIND_IP}:11434:11434"
     volumes:
       - /opt/homelab/data/ollama:/root/.ollama
     deploy:

services/ollama/env.example

@@ -1,2 +1,8 @@
-# No specific environment variables required by default.
+# Copy to .env next to docker-compose.yml (gitignored); docker compose picks
+# it up automatically. Same convention as services/llm-gateway.
+
+# Tailscale IP of the SOLARIA node. Bind the mesh-facing publish ONLY to this
+# IP — never 0.0.0.0. Verify when rebuilding the host: tailscale ip -4.
+TAILSCALE_BIND_IP=100.100.231.104
+
 # CUDA_VISIBLE_DEVICES=0