56 lines
2 KiB
Markdown
56 lines
2 KiB
Markdown
|
# llm-gateway
|
||
|
|
|
||
|
|
Small FastAPI router in front of Ollama on **SOLARIA**. Runs on the **PIHA**
|
||
|
|
node, bound to PIHA's Tailscale IP only (`http://piha:8080` on the mesh; no
|
||
|
|
public ingress). Stateless — no volumes, no secrets.
|
||
|
|
|
||
|
|
Origin: migrated 2026-07-03 from an unmanaged shadow deployment at
|
||
|
|
`/opt/llm-gateway` on PIHA into GitOps. The old copy is kept on disk as
|
||
|
|
rollback until explicitly retired.
|
||
|
|
|
||
|
|
## Endpoints
|
||
|
|
|
||
|
|
| Endpoint | Method | Model | Purpose |
|
||
|
|
|---|---|---|---|
|
||
|
|
| `/` | GET | — | Health: `{"status":"gateway ok"}` |
|
||
|
|
| `/api/chat` | POST | `deepcoder:14b` | General chat completion |
|
||
|
|
| `/api/code` | POST | `deepseek-coder:latest` | Code/command generation ("return only code" system prompt) |
|
||
|
|
|
||
|
|
Request body for both POST routes: `{"prompt": "...", "stream": false}`.
|
||
|
|
|
||
|
|
`/api/code` quirk (by design): a single-line prompt whose first word is not a
|
||
|
|
natural-language starter (write/create/explain/…) is treated as an
|
||
|
|
already-formed shell command and echoed back verbatim as `response`.
|
||
|
|
|
||
|
|
## Upstream dependency
|
||
|
|
|
||
|
|
Calls Ollama at `http://solaria:11434/api/generate` (override via `OLLAMA_URL`).
|
||
|
|
SOLARIA is a GPU node that may be powered down — the gateway stays healthy
|
||
|
|
(`/` still answers) but proxied routes return **502** until Ollama is back.
|
||
|
|
`solaria` resolves via the host's Tailscale MagicDNS, which Docker's embedded
|
||
|
|
DNS forwards to.
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
- `.env` — **gitignored**, copy from `env.example`. Holds `TAILSCALE_BIND_IP`
|
||
|
|
(required for the mesh-only port bind) and optional `OLLAMA_URL`.
|
||
|
|
- No secrets anywhere in the stack.
|
||
|
|
|
||
|
|
## Deploy (PIHA)
|
||
|
|
|
||
|
|
1. `git pull` on PIHA (`~/homelab-codex-ws`).
|
||
|
|
2. `cp services/llm-gateway/env.example services/llm-gateway/.env` (defaults
|
||
|
|
are correct for PIHA).
|
||
|
|
3. ```
|
||
|
|
docker compose -f services/llm-gateway/docker-compose.yml \
|
||
|
|
-f hosts/piha/runtime/llm-gateway/docker-compose.override.yml up -d --build
|
||
|
|
```
|
||
|
|
4. Verify: `services/llm-gateway/healthcheck.sh`, then from any mesh node:
|
||
|
|
`curl http://piha:8080/` → `{"status":"gateway ok"}`.
|
||
|
|
|
||
|
|
## Tests
|
||
|
|
|
||
|
|
```
|
||
|
|
cd services/llm-gateway && pytest
|
||
|
|
```
|