Batched embedding client (input as a list, batch 64 measured live at ~8-18
ms/chunk vs ~150-200 ms/chunk sequential through embed_chunk's
/api/embeddings) for jobs/mail-body-ingest -- embed_chunk stays the
single-text path for kb-query and paperless cyclic ingest. Validates the
response embedding count and each item's dimension (1024), raising
EmbeddingDimensionError (abort-run) on mismatch so a bad batch never silently
writes a mismatched vector (docs/kb/modules/05-faza-mailowa-plan.md, §4,
decision 5).
chunk_text/hard_split/split_paragraphs + 2400/600-char constants move out of
documents_ingest.chunk_embed into kb_mail.chunking so the upcoming
jobs/mail-body-ingest shares the exact same chunker instead of a copy-pasted
drift (the phase-4 lesson for retrieval.py -> packages/kb-retrieval, applied
again). chunk_embed.py re-exports unchanged; zero behavior change, tests moved
1:1 alongside the code (docs/kb/modules/05-faza-mailowa-plan.md, §3).
Module 5 phase 4 step 0 (docs/kb/modules/05-faza4-plan.md, §3, decision 1):
kb-query is a long-lived Docker service, documents-ingest is a venv job with
an `anthropic` dependency and CLI scripts it doesn't need. Move
embed_chunk/_vector_literal/cascade_query/flat_query into a shared package
with minimal deps (asyncpg, aiohttp only) so both can depend on the same
tested retrieval code without the service image pulling in the job's extras.
documents_ingest.chunk_embed/retrieval keep thin re-exports so nothing
importing the old paths breaks. Pure refactor: retrieval_eval.py run live
against kb-postgres@PIHA + Ollama@SOLARIA before/after gives byte-identical
`dist`/hit@3/gate results (still PASS) — zero behavior change in the cascade.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four audit findings (2026-07-14), each reproduced on crafted mboxes; the
Takeout corpus has proven 8-bit header bytes, so all are real re-import risks.
1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an
email.header.Header (not str) for raw 8-bit bytes; the old
Header.strip() raised AttributeError. _message_id ran BEFORE the
per-message try, so one bad header killed the entire import.
Fix: str() + sanitize_surrogates() before strip; and move
_message_id/_parse_date/_parse_attachments INSIDE the per-message try —
a broken message is now errors += 1, never run death.
2. Poison batch. pending.clear() ran only AFTER a successful insert, so a
failed flush (DB down / bad row) left pending intact and every later
message re-flushed the doomed batch; the final flush sat in try/finally
with no except and propagated out, losing all stats. Fix: _flush always
clears pending and counts a failed insert as db_insert_failed; the run
always reaches import_complete.
3. Stats didn't reconcile with the DB. imported counts archive writes, not
DB rows, so a partial-insert drift was invisible. Fix: separate
db_inserted/db_insert_failed counters; main() exits non-zero on any
error, DB drift, or a processed = imported + skipped + errors imbalance.
4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header)
raised even when str(header) parses fine. Fix: str() before the epoch
fallback.
Shared helper: _sanitize moved from gmail-header-backfill into
packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs;
gmail-header-backfill now depends on kb-mail.
Tests: regression coverage for all four findings in gmail-bulk-import
(8-bit id/date, per-message guard, failed-insert non-poisoning, stats
balance) plus kb_mail.text unit tests. Full suites green:
kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Krok 1 planu docs/kb/modules/05-faza2-plan.md §3/§6 (chunk-level embeddings,
1:N do envelope). Addytywna — 001_envelope.sql nietknięta (zweryfikowane
\d envelope po migracji: identyczny schemat + FK jako "Referenced by").
Schemat wg rekomendacji recon (§2 decyzja 1+2): osobna tabela (nie kolumna
w envelope, bo N-wartościowy chunking jest obowiązkowy przy dokumentach
>8k tokenów), embedding VECTOR(1024) pod bge-m3 (dense), HNSW cosine index,
kolumna `model` do trywialnego re-indexu przy zmianie modelu (kb-00 zasada
#1: indeks odtwarzalny). Idempotentna (CREATE TABLE/INDEX IF NOT EXISTS,
zweryfikowane podwójnym uruchomieniem na kb-postgres@PIHA — drugi run same
NOTICE "already exists, skipping").
Zastosowana na żywej bazie: ssh piha docker exec kb-postgres psql, po
potwierdzeniu SQL przez Oskara. \dt + \d document_chunk + \d envelope
zweryfikowane po migracji.
Testy: dopisane sanity-testy 002 do packages/kb-mail/tests/test_migration.py
(wzorzec 001 — statyczne assercje na treści SQL, bez DB), 13/13 zielone.
Co NIE jest częścią tego kroku (§3 planu, odłożone): entity/entity_link
(graf encji) — szkic na przyszłość, nie blokuje domknięcia modułu 5.
Co dalej (plan §6, poza zakresem tego kroku): ollama pull bge-m3 na SOLARII,
token API Paperless, jobs/gmail-header-backfill/, adapter Paperless→koperta,
chunking+embed job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
