Domkniecie Kroku 7. Realizuje Decyzje (d) reconu (host schedulera + korekta
kadencji indeksowania) i doklada dokumentacje wg konwencji OKF.
Scheduler (NIEAKTYWOWANY — wlacza operator):
- jobs/mail-imap-sync/systemd/{service,timer,run.sh} — wzorzec 1:1 z kb-ingest,
OnCalendar=hourly, Persistent=true, log do pliku (nigdy sam journal).
- hosts/piha/jobs.yaml — deklaracja jednostek host-level na PIHA. Nowy plik, bo
services.yaml jest dla kontenerow (supervisor dopasowuje jego wpisy do world-state
i wpis niekontenerowy dryfowalby wiecznie jako missing_service). Nic tego pliku
nie czyta — istnieje po to, zeby "shadow-deploy family" z otwartego pytania 5
reconu multiagentowego byla spisana, a nie tylko na nodzie.
Takt indeksowania (Decyzja (d), recon §3.3):
- kb-ingest.timer: 03:30 raz na dobe -> co 2 h. O 03:30 SOLARIA prawie na pewno spi
(potwierdzone odczytem kb_ingest_embed_skipped 1 z 2026-08-06), a tick dostaje
teraz etap mailowy: ~60 nowych chunkow na dobe pomijanych kazdej nocy sprawiloby,
ze backlog rosnie monotonicznie i KbEmbedBacklogGrowing zapala sie NA STALE.
Co 2 h zamiast stalej godziny — probe Ollamy sam wybiera okno, wiec ktorys tick
w nie trafi niezaleznie od nawykow operatora.
- cyclic_ingest: etap mailowy (mail_body_ingest --only-unchunked), import miekki,
wiec venv bez tego pakietu pomija etap zamiast wywracac wrapper. Predykat bledu
JEST luzniejszy niz wlasne main() tamtego joba i to jedyne takie miejsce w tym
wrapperze: pojedynczy trwale nieparsowalny mail nie moze zamrozic
last_success_timestamp i zapalic KbIngestStale na zawsze. Bledy per-mail sa
publikowane jako kb_ingest_mail_parse_errors, nie chowane.
Obserwowalnosc: KbMailSyncStale (6 h bez udanego ticku). Alert na cisze w skrzynce
ODRZUCONY (decyzja operatora, zgodna z reconem §3.4) — zero nowych maili to legalny
stan skrzynki, a alert zapalajacy sie na zdrowym systemie zostaje wyciszony
i przestaje dzialac wtedy, gdy jest potrzebny.
Dokumentacja:
- kb/services/job-mail-imap-sync.md (OKF), kb/runbooks/mail-sync-run.md — 9 krokow
pierwszego uruchomienia, w tym checklista 4 punktow [do weryfikacji na zywo]
z reconu (polityki dostawcow — do sprawdzenia, nie do zgadniecia) oraz pomiar
STATUS (MESSAGES) na Fastmailu, na ktorym zapada ODLOZONA decyzja o historii.
- kb-mail-pillar.md: KOREKTA JMAP -> IMAP dla Fastmaila jako decyzja 2026-08-06;
stary zapis zostaje jako historia z data. Zamkniete "unifikacja adaptera"
i "sizing Gmaila"; otwarte zostaje "sizing Fastmaila" — celowo, bo rozstrzyga
je pomiar, nie dyskusja.
- kb-m5-faza-mailowa.md: Krok 7 IN PROGRESS + tabela zakresu wdrozonego,
kb-m5-faza3.md: korekta harmonogramu i sekwencji wrappera,
pkg-kb-mail.md: rozpisany ze stubu, kb-postgres.md: lista migracji + 005.
Testy: 642 passed (calosc kb-mail, kb-retrieval i jobs). systemd-analyze verify
na timerze przechodzi, OnCalendar=0/2:00:00 normalizuje sie do co 2 h.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
117 lines
4.4 KiB
Markdown
117 lines
4.4 KiB
Markdown
---
|
|
okf: "0.1"
|
|
type: service
|
|
visibility: private
|
|
status: active
|
|
updated: 2026-06-22
|
|
links: []
|
|
---
|
|
|
|
# kb-postgres
|
|
|
|
Postgres 16 + pgvector — KB spine on **PIHA** (Raspberry Pi 5, always-on). Stores the frozen envelope schema shared by all KB pillars (mails, documents, photos, transactions).
|
|
|
|
Runs here because the KB store must answer queries 24/7; SOLARIA (GPU/compute) is powered down intermittently. Embeddings/models still run on SOLARIA's GPU — only the Postgres+pgvector store lives on PIHA. The `pgvector/pgvector:pg16` image is multi-arch and runs natively on arm64 (the Pi 5).
|
|
|
|
Port: **5433** on PIHA (Tailscale-accessible to other nodes).
|
|
|
|
## Standard deploy (from SATURN)
|
|
|
|
```bash
|
|
# On SATURN — pushes to master, then deploy.sh SSHes to PIHA and runs deploy-node.sh
|
|
git push origin master
|
|
scripts/deploy/deploy.sh piha
|
|
```
|
|
|
|
`deploy-node.sh` on PIHA automatically picks up the per-host override:
|
|
```
|
|
docker compose \
|
|
-f services/kb-postgres/docker-compose.yml \
|
|
-f hosts/piha/runtime/kb-postgres/docker-compose.override.yml \
|
|
up -d --remove-orphans
|
|
```
|
|
|
|
The PIHA override caps memory (`mem_limit: 1g`) and tunes Postgres for a tight,
|
|
HA-shared RAM budget. Data lives in the `kb_postgres_data` named volume, which
|
|
**must** land on the NVMe (Docker data-root on `/home`), never the SD card —
|
|
verify before first deploy (see the override file's DATA PLACEMENT note):
|
|
|
|
```bash
|
|
# On PIHA
|
|
docker info -f '{{.DockerRootDir}}' # expect an NVMe path
|
|
df -h "$(docker info -f '{{.DockerRootDir}}')" # confirm it's the NVMe
|
|
```
|
|
|
|
## First-time setup on PIHA (before first deploy)
|
|
|
|
The `.env` file must exist at `services/kb-postgres/.env` in the PIHA repo checkout
|
|
(alongside the compose file — that's where `env_file: .env` resolves to):
|
|
|
|
```bash
|
|
# On PIHA
|
|
cd ~/homelab-codex-ws
|
|
cp services/kb-postgres/env.example services/kb-postgres/.env
|
|
# Edit .env: set POSTGRES_PASSWORD to something strong
|
|
```
|
|
|
|
`.env` is gitignored (`*.env` rule in root `.gitignore`) — it will never be committed.
|
|
|
|
## Manual one-off (debugging / first boot)
|
|
|
|
```bash
|
|
# On PIHA, from repo root
|
|
docker compose \
|
|
-f services/kb-postgres/docker-compose.yml \
|
|
-f hosts/piha/runtime/kb-postgres/docker-compose.override.yml \
|
|
up -d
|
|
```
|
|
|
|
## Verify after first boot
|
|
|
|
```bash
|
|
# Host-side healthcheck
|
|
./services/kb-postgres/healthcheck.sh
|
|
|
|
# Inside the container
|
|
docker exec kb-postgres psql -U kb -d kb -c '\d envelope'
|
|
docker exec kb-postgres psql -U kb -d kb \
|
|
-c "SELECT extname FROM pg_extension WHERE extname = 'vector';"
|
|
```
|
|
|
|
Expected `\d envelope` output:
|
|
|
|
```
|
|
Table "public.envelope"
|
|
Column | Type | Nullable | Default
|
|
----------+--------------------------+----------+-----------
|
|
id | text | not null |
|
|
source | text | not null |
|
|
ts | timestamp with time zone | not null |
|
|
geo | jsonb | |
|
|
raw_ref | text | not null |
|
|
entities | jsonb | not null | '[]'::jsonb
|
|
Indexes:
|
|
"envelope_pkey" PRIMARY KEY, btree (id)
|
|
"envelope_source_idx" btree (source)
|
|
"envelope_ts_idx" btree (ts)
|
|
```
|
|
|
|
## Schema contract
|
|
|
|
The `envelope` table is the frozen cross-source envelope (see `kb/subsystems/kb-overview.md` §Zasady przekrojowe). Adding columns is OK; removing or renaming existing ones is NOT.
|
|
|
|
Future migrations go in `init/` as `002_*.sql`, `003_*.sql`, …. Postgres runs `initdb` scripts only on a fresh volume — for existing instances apply migrations with `psql` directly.
|
|
|
|
Applied so far: `001` envelope, `002` document_chunk, `003` chunk model key + `excluded_reason`, `004` document_summary, `005` `mail_sync_state`.
|
|
|
|
`005_mail_sync_state.sql` (2026-08-06) adds the per-folder IMAP sync cursor for `jobs/mail-imap-sync`, keyed `(account, folder)`. A table rather than a file under `/opt/homelab/state/` for one decisive reason: the cursor and the envelopes it describes must restore together or not at all. A state file surviving a DB restore would make the poller silently skip everything between the restored rows and the file's `last_uid` — a failure with no symptom. Apply it by hand on the live instance (`kb/runbooks/mail-sync-run.md` §4); the DDL is `IF NOT EXISTS`, so repeating it is safe.
|
|
|
|
## Connection string
|
|
|
|
```
|
|
postgresql://kb:<POSTGRES_PASSWORD>@piha:5433/kb
|
|
```
|
|
|
|
Set `KB_TEST_DSN` to this value when running integration tests from `packages/kb-mail/`
|
|
(host = `piha` over Tailscale).
|