09bb59a1db
Adds documents-ingest-embed: chunks source='paperless' envelope content (paragraph-preferring, ~600 tok/chunk, ~150 tok overlap, hard char-fallback for oversized paragraphs per plan §2 decision 3), embeds each chunk via Ollama (bge-m3, dim validated against document_chunk's VECTOR(1024) on every response) and inserts into document_chunk. Lives in documents-ingest per the plan's own recommendation (§6 step 6) rather than a new package — reuses the job family's existing idempotency/stats-balance/dry-run conventions (paperless_adapter.py, gmail-header-backfill). A 5-angle multi-agent code review of the initial implementation surfaced three real bugs, fixed here: hard_split() could infinite-loop if --chunk-overlap >= --chunk-size (now guarded in both hard_split() and main()); insert_chunk() wasn't error-isolated like embed_chunk(), so a DB write failure would crash the whole run instead of being counted and skipped; and ON CONFLICT DO NOTHING's outcome was discarded, so a silently skipped row (the known gap where document_chunk's UNIQUE constraint doesn't include `model`) would have been miscounted as a successful insert - now tracked separately as chunks_conflict_skipped and treated as a run failure. Smoke-tested and run to completion live on SOLARIA against the real Ollama instance and kb-postgres@PIHA: dry-run matched the known phase-2-step-5 figures exactly (186 fetched, 26 empty_content, 2684 chunks planned), a --limit 10 apply + idempotent re-run + DB/distance sanity checks all passed, and the full 186-document run inserted 2683/2684 chunks (1 isolated error - Ollama's runtime context window rejected one pathological dot-leader table-of-contents chunk that tokenized far more densely than estimated; documented as a known limitation, not fixed here given it's a single-chunk edge case). Timing: ~0.79s/chunk average on CPU (SOLARIA's Ollama runs GPU-less per the recent GPU-reservation-disabled fix), ~35 min wall-clock for the full pilot - the real input for scoping the later mail-corpus embedding phase (plan §7's GPU-based estimate doesn't hold here). pytest: 101 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|---|---|---|
| .claude/skills | ||
| backups/zigbee | ||
| docs | ||
| dotfiles | ||
| hosts | ||
| inventory | ||
| jobs | ||
| packages/kb-mail | ||
| scripts | ||
| services | ||
| .codex | ||
| .gitignore | ||
| CLAUDE.md | ||
| codex_context | ||
| codex_context.yaml | ||
| deploy_agent.py | ||
| ollama_client.py | ||
| README.md | ||
| start-aider.sh | ||
| start-codex.sh | ||
| sync-context.sh | ||
| tech-debt.md | ||
| update-context.md | ||
Homelab Codex
GitOps-lite orchestration for a distributed homelab environment.
Architecture
The homelab consists of several nodes connected via a Tailscale internal mesh.
| Host | Role | Description |
|---|---|---|
| SATURN | Primary Node | Development, orchestration, and git source of truth (commit node). |
| SOLARIA | Compute Node | GPU, inference, and heavy compute workloads. |
| PIHA | Infra Node | Core infrastructure services, automation, and monitoring. |
| VPS | Edge Node | Public ingress, reverse proxy, and edge services. |
Agent System
The homelab uses a multi-agent orchestration model with human-in-the-loop for destructive actions:
| Agent | Node | Role |
|---|---|---|
| stability-agent | all nodes | Per-node watchdog — monitors Docker, disk, Tailscale, MQTT; emits events |
| node-agent | all nodes | Publishes container health events to Redis pub/sub |
| observer | VPS | Synthesizes world state from events into /opt/homelab/world/*.json |
| supervisor | VPS | Detects drift between desired and actual state; writes pending actions |
| planner-agent | SOLARIA | LLM-powered diagnosis — listens to Redis, proposes remediation actions |
| executor | VPS | Executes actions only after operator approval |
| operator-ui + telegram-bot | VPS / PIHA | Operator reviews and approves/rejects pending actions |
Action approval flow: pending/ → operator approves → approved/ → executor runs.
Repository Structure
docs/: Infrastructure Standards and Deployment Conventions.hosts/: Host-specific configurations and service assignments.services/: Reusable Docker Compose service definitions.scripts/: Deployment and management scripts.
Getting Started
- Standardization: Follow the Infrastructure Standards.
- Deployment: See Deployment Conventions for how to roll out changes.
- SATURN: Remember that SATURN is the only node where commits should be made.
Documentation Index
- Infrastructure Standards
- Agent Operating Procedures (For AI/Non-Human Agents)
- Deployment Conventions
- Hardware
- Networking
- Services
- Node Capabilities
- Action Model
Note: This repository documents the state of the homelab. Runtime state lives outside the repository in /opt/homelab.