homelab-codex-ws/jobs
oskar ad0ef408f9 feat(mail-body-ingest): new job to chunk+embed gmail body content (faza mailowa Krok 2)
Second full pass over the gmail .eml archive (gmail-bulk-import's first pass
skipped inline text/plain and text/html on purpose). Per envelope: typed
parse with compat32 fallback -> body extraction (inline text/plain preferred,
HTML->text via a small stdlib HTMLParser otherwise) -> quote-strip (reply
markers + `>`-quoted lines, EN/PL/Outlook patterns) -> newsletter
classification (List-Unsubscribe/List-Id/Precedence, chunked but not
embedded, excluded_reason='newsletter') -> Temat/Od/Data prefix from the
already-backfilled entities[type=headers] (zero header re-parse) -> chunk via
kb_mail.chunking -> batched embed_batch (64) -> INSERT document_chunk.
In-Reply-To/References are appended as entities[type=threading] during the
same read (idempotent WHERE NOT EXISTS append, 1:1 with
gmail-header-backfill) -- the only DB writes are document_chunk INSERTs and
an additive envelope.entities UPDATE; the .eml archive stays read-only.

--dsn/KB_DSN, --archive-root, --since/--limit/--offset, --batch-size,
--apply (dry-run default). Idempotency keys on (envelope_id, chunk_index)
pre-fetched scoped to --model, built correctly from the start per the
plan's flagged chunk_embed.py precedent. A failed embed batch is isolated
(chunks_errors, no abort) for Ollama's documented instability; a wrong
embedding dimension aborts the whole run.

48 tests, DoD smoke run against live kb-postgres@PIHA confirmed wiring
(archive not yet rsync'd to SOLARIA, so all 5 rows correctly reported
missing_file). docs/kb/modules/05-faza-mailowa-plan.md, §5.
2026-07-23 17:06:49 +02:00
..
documents-ingest refactor(kb-mail): extract chunker to packages/kb-mail (faza mailowa Krok 0) 2026-07-23 17:06:48 +02:00
gmail-bulk-import fix(gmail-bulk-import): harden re-import against 8-bit headers + poison batches 2026-07-15 14:28:09 +02:00
gmail-header-backfill fix(gmail-bulk-import): harden re-import against 8-bit headers + poison batches 2026-07-15 14:28:09 +02:00
mail-body-ingest feat(mail-body-ingest): new job to chunk+embed gmail body content (faza mailowa Krok 2) 2026-07-23 17:06:49 +02:00