Migracja 004: document_summary (envelope_id, summary, tags JSONB, model, embedding
VECTOR(1024) + HNSW cosine, embedding_model, UNIQUE(envelope_id, model) od razu — wzorzec
002/003). Zastosowana na żywej bazie kb-postgres@PIHA.
Job documents-ingest-summarize: wejście = document_chunk.text WHERE excluded_reason IS NULL
per koperta source='paperless' (duplikaty przez entities[duplicate_of] pomijane w całości),
wymuszony JSON {summary, tags} przez --backend ollama|anthropic, słownik tagów kontrolowany
(tags-vocab.yaml) z max 3 free-form, map-reduce dla dokumentów >200k znaków (grupy ~20
chunków), bilans + idempotencja + izolacja błędów per wiersz wg wzorców rodziny jobów.
Osobny --embed-summaries (bge-m3, reużywa chunk_embed.embed_chunk).
Bug znaleziony i naprawiony w trakcie pilota: brak options.num_ctx w wywołaniach Ollamy
powodował, że gemma3:12b używał domyślnego runtime kontekstu (~2048 tok), nie
zadeklarowanego 128k — dla 71/157 dokumentów (45%, >8k znaków) treść była cicho ucinana
(zweryfikowane: prompt_eval_count=2051 dla dokumentu 93k znaków). Naprawa: compute_num_ctx()
liczy num_ctx z długości promptu (~3 znaki/token, cap 131072). Cały tor lokalny przeliczony
od zera po naprawie.
Wynik pilota (186 dok. paperless, 3 duplikaty, 26 bez aktywnych chunków → 157 oczekiwanych
na tor): tor lokalny (gemma3:12b) 155/157 (2 izolowane błędy JSON po retry: paperless:24,
paperless:61), tor referencyjny (claude-haiku-4-5) 157/157, 0 błędów JSON, 0 tagów
ucinanych — słownik przestrzegany w 100% przypadków. Oba komplety zembedowane (bge-m3).
Znaleziony przy okazji: prompt do tagów wymagał dopracowania — pierwsza wersja pozwalała
modelowi zwracać tagi po angielsku spoza słownika; wzmocniona instrukcja (słownik w
pierwszej kolejności, "nigdy po angielsku") poprawiła zgodność w 2/3 przypadków testowych.
Porównanie A/B (~15 dok.) i weryfikacja końcowa (bilans, sanity SQL, retrieval po summary)
odłożone do następnej sesji.
Testy: 157 (152 nowe/summarize.py + istniejące), mocki API/Ollama/DB, bilans, idempotencja,
regresja num_ctx.
Co najmniej 3 decyzje wymagały zatrzymania i potwierdzenia z Oskarem w sesji (sposób podania
klucza API, wybór modelu lokalnego gemma3:12b, naprawa+przeliczenie całego toru lokalnego po
odkryciu buga num_ctx) — udokumentowane w transkrypcie sesji.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-embed: chunks source='paperless' envelope content
(paragraph-preferring, ~600 tok/chunk, ~150 tok overlap, hard char-fallback
for oversized paragraphs per plan §2 decision 3), embeds each chunk via
Ollama (bge-m3, dim validated against document_chunk's VECTOR(1024) on
every response) and inserts into document_chunk. Lives in documents-ingest
per the plan's own recommendation (§6 step 6) rather than a new package —
reuses the job family's existing idempotency/stats-balance/dry-run
conventions (paperless_adapter.py, gmail-header-backfill).
A 5-angle multi-agent code review of the initial implementation surfaced
three real bugs, fixed here: hard_split() could infinite-loop if
--chunk-overlap >= --chunk-size (now guarded in both hard_split() and
main()); insert_chunk() wasn't error-isolated like embed_chunk(), so a DB
write failure would crash the whole run instead of being counted and
skipped; and ON CONFLICT DO NOTHING's outcome was discarded, so a silently
skipped row (the known gap where document_chunk's UNIQUE constraint
doesn't include `model`) would have been miscounted as a successful insert
- now tracked separately as chunks_conflict_skipped and treated as a
run failure.
Smoke-tested and run to completion live on SOLARIA against the real Ollama
instance and kb-postgres@PIHA: dry-run matched the known phase-2-step-5
figures exactly (186 fetched, 26 empty_content, 2684 chunks planned), a
--limit 10 apply + idempotent re-run + DB/distance sanity checks all
passed, and the full 186-document run inserted 2683/2684 chunks (1 isolated
error - Ollama's runtime context window rejected one pathological
dot-leader table-of-contents chunk that tokenized far more densely than
estimated; documented as a known limitation, not fixed here given it's a
single-chunk edge case). Timing: ~0.79s/chunk average on CPU (SOLARIA's
Ollama runs GPU-less per the recent GPU-reservation-disabled fix), ~35 min
wall-clock for the full pilot - the real input for scoping the later
mail-corpus embedding phase (plan §7's GPU-based estimate doesn't hold
here).
pytest: 101 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds documents-ingest-paperless: paginated GET /api/documents/, maps each
doc to a source='paperless' envelope per plan §4.2-4.3, reusing
kb_mail.Envelope/insert_envelope unchanged (packages/kb-mail not touched).
Cross-source link (source_mail entity) is a deterministic join of
original_file_name against the phase-1 registry.json consume_name index —
no heuristics, no correspondent guessing (plan decision 4). Stats always
balance (fetched = already_in_db + inserted + errors) and main() now also
exits non-zero on imbalance, not just on errors>0, matching the exit-code
convention already established in gmail-bulk-import.
Verified live on PIHA (rsync to /tmp, ~/kb/venv, PIHA checkout untouched):
dry-run then --apply inserted 186/186 paperless envelopes (0 errors,
180 source_mail links), a second --apply reported inserted=0/already_in_db=186
(idempotent), gmail rows stayed at 225030 and document_chunk stayed empty.
Rotated the kb-ingest Paperless API token after it was accidentally
partially echoed during recon (old token now dead).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of
PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive
and drops them into Paperless consume/ for OCR, so the RAG layer has real
documents to work with before the full Paperless/Nextcloud envelope adapter
is built. sha256-first attachment matching (manifest filenames can carry raw
RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default
decodes today — confirmed against live data, ~10% of candidates were affected).
Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write.
Verified end-to-end on PIHA: dry-run + --apply both run against live
kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run
duplicates correctly deduped), Paperless picked them up and started OCR
immediately.
