70ff08b4a0
Kod zyl tylko na dysku PIHA w /opt/llm-gateway (bez gita). Przeniesiony do
services/llm-gateway/ pod pelny wzorzec homelaba:
- app/main.py: kod 1:1 z PIHA + OLLAMA_URL/CHAT_MODEL/CODE_MODEL
nadpisywalne przez env (defaulty bez zmian)
- docker-compose.yml: bind TYLKO do Tailscale IP (${TAILSCALE_BIND_IP},
wzorzec fleet-prometheus), nie 0.0.0.0 jak w starym compose
- service.yaml, env.example (bez sekretow), healthcheck.sh, README, testy
- hosts/piha/runtime/llm-gateway: mem_limit 256m (PIHA jest RAM-bound)
- rejestracja w hosts/piha/services.yaml i inventory/topology.yaml
Zepsuty /opt/llm-gateway/docker-compose.yml (zduplikowany klucz ports)
celowo NIE przeniesiony.
DoD: pytest 4 passed; docker build + smoke run OK (GET / -> gateway ok).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
56 lines
2 KiB
Markdown
56 lines
2 KiB
Markdown
# llm-gateway
|
|
|
|
Small FastAPI router in front of Ollama on **SOLARIA**. Runs on the **PIHA**
|
|
node, bound to PIHA's Tailscale IP only (`http://piha:8080` on the mesh; no
|
|
public ingress). Stateless — no volumes, no secrets.
|
|
|
|
Origin: migrated 2026-07-03 from an unmanaged shadow deployment at
|
|
`/opt/llm-gateway` on PIHA into GitOps. The old copy is kept on disk as
|
|
rollback until explicitly retired.
|
|
|
|
## Endpoints
|
|
|
|
| Endpoint | Method | Model | Purpose |
|
|
|---|---|---|---|
|
|
| `/` | GET | — | Health: `{"status":"gateway ok"}` |
|
|
| `/api/chat` | POST | `deepcoder:14b` | General chat completion |
|
|
| `/api/code` | POST | `deepseek-coder:latest` | Code/command generation ("return only code" system prompt) |
|
|
|
|
Request body for both POST routes: `{"prompt": "...", "stream": false}`.
|
|
|
|
`/api/code` quirk (by design): a single-line prompt whose first word is not a
|
|
natural-language starter (write/create/explain/…) is treated as an
|
|
already-formed shell command and echoed back verbatim as `response`.
|
|
|
|
## Upstream dependency
|
|
|
|
Calls Ollama at `http://solaria:11434/api/generate` (override via `OLLAMA_URL`).
|
|
SOLARIA is a GPU node that may be powered down — the gateway stays healthy
|
|
(`/` still answers) but proxied routes return **502** until Ollama is back.
|
|
`solaria` resolves via the host's Tailscale MagicDNS, which Docker's embedded
|
|
DNS forwards to.
|
|
|
|
## Configuration
|
|
|
|
- `.env` — **gitignored**, copy from `env.example`. Holds `TAILSCALE_BIND_IP`
|
|
(required for the mesh-only port bind) and optional `OLLAMA_URL`.
|
|
- No secrets anywhere in the stack.
|
|
|
|
## Deploy (PIHA)
|
|
|
|
1. `git pull` on PIHA (`~/homelab-codex-ws`).
|
|
2. `cp services/llm-gateway/env.example services/llm-gateway/.env` (defaults
|
|
are correct for PIHA).
|
|
3. ```
|
|
docker compose -f services/llm-gateway/docker-compose.yml \
|
|
-f hosts/piha/runtime/llm-gateway/docker-compose.override.yml up -d --build
|
|
```
|
|
4. Verify: `services/llm-gateway/healthcheck.sh`, then from any mesh node:
|
|
`curl http://piha:8080/` → `{"status":"gateway ok"}`.
|
|
|
|
## Tests
|
|
|
|
```
|
|
cd services/llm-gateway && pytest
|
|
```
|