3d4ee3818d
Last missing core piece of KB phase 4: kb-query no longer hard-fails /search when Ollama@SOLARIA is unreachable. app/fallback.py implements the plan's circuit-breaker exactly (30s cached health probe, 3s hard embed timeout on SOLARIA, one-shot same-request switch to a new local ollama-piha@PIHA container on timeout/error). sol_status in /healthz and /search now reflects the real breaker state instead of a hardcoded "up". New services/ollama-piha (bge-m3, OLLAMA_KEEP_ALIVE=0, arm64/no-GPU) is the local fallback leg. Live calibration on PIHA (2026-07-27, normal load): embed latency 4.2-5.2s, RAM peak ~983MiB against a 2.5GiB ceiling -- both inside the plan's go-bar, so the fallback is enabled by default rather than gated behind a flag. Calibration also surfaced and disabled (not removed) a previously-undocumented orphaned native ollama.service on PIHA that had been conflicting with the container's port. The embed-model invariant (query embedding == document_chunk.model) still enforces once at startup, since both fallback legs share one EMBED_MODEL constant by construction; a redundant per-request DB check was deliberately skipped and the invariant is instead proven structurally by test. retrieval_eval.py gains --transport http (plan §2 decision 6/§9), previously unimplemented. Verified live: HTTP transport is bit-identical to direct transport against the same live SOLARIA (0 mismatches), and a live sol-down simulation (kb-query's own OLLAMA_URL pointed at a dead address, no other Ollama consumer touched) shows the PIHA fallback answering with the same hit@3 gate outcome and dist within ~3e-4 of the SOLARIA baseline. Zero changes to DB schema or kb_retrieval's retrieval logic -- only the embed + health layer, per task constraints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|---|---|---|
| .claude/skills | ||
| backups/zigbee | ||
| docs | ||
| dotfiles | ||
| hardware/esp/ir-ac-ha-integration | ||
| hosts | ||
| inventory | ||
| jobs | ||
| packages | ||
| scripts | ||
| services | ||
| .codex | ||
| .gitignore | ||
| CLAUDE.md | ||
| codex_context | ||
| codex_context.yaml | ||
| deploy_agent.py | ||
| ollama_client.py | ||
| README.md | ||
| start-aider.sh | ||
| start-codex.sh | ||
| sync-context.sh | ||
| tech-debt.md | ||
| update-context.md | ||
Homelab Codex
GitOps-lite orchestration for a distributed homelab environment.
Architecture
The homelab consists of several nodes connected via a Tailscale internal mesh.
| Host | Role | Description |
|---|---|---|
| SATURN | Primary Node | Development, orchestration, and git source of truth (commit node). |
| SOLARIA | Compute Node | GPU, inference, and heavy compute workloads. |
| PIHA | Infra Node | Core infrastructure services, automation, and monitoring. |
| VPS | Edge Node | Public ingress, reverse proxy, and edge services. |
Agent System
The homelab uses a multi-agent orchestration model with human-in-the-loop for destructive actions:
| Agent | Node | Role |
|---|---|---|
| stability-agent | all nodes | Per-node watchdog — monitors Docker, disk, Tailscale, MQTT; emits events |
| node-agent | all nodes | Publishes container health events to Redis pub/sub |
| observer | VPS | Synthesizes world state from events into /opt/homelab/world/*.json |
| supervisor | VPS | Detects drift between desired and actual state; writes pending actions |
| planner-agent | SOLARIA | LLM-powered diagnosis — listens to Redis, proposes remediation actions |
| executor | VPS | Executes actions only after operator approval |
| operator-ui + telegram-bot | VPS / PIHA | Operator reviews and approves/rejects pending actions |
Action approval flow: pending/ → operator approves → approved/ → executor runs.
Repository Structure
docs/: Infrastructure Standards and Deployment Conventions.hosts/: Host-specific configurations and service assignments.services/: Reusable Docker Compose service definitions.scripts/: Deployment and management scripts.
Getting Started
- Standardization: Follow the Infrastructure Standards.
- Deployment: See Deployment Conventions for how to roll out changes.
- SATURN: Remember that SATURN is the only node where commits should be made.
Documentation Index
- Infrastructure Standards
- Agent Operating Procedures (For AI/Non-Human Agents)
- Deployment Conventions
- Hardware
- Networking
- Services
- Node Capabilities
- Action Model
Note: This repository documents the state of the homelab. Runtime state lives outside the repository in /opt/homelab.