homelab-codex-ws/kb/services/job-mail-body-ingest.md

okf	type	visibility	status	updated	links
0.1	service	private	active	2026-07-22	../runbooks/mail-body-ingest-run.md

mail-body-ingest

Module 5, faza mailowa (docs/kb/modules/05-faza-mailowa-plan.md, §5, Krok 2). Second full pass over the gmail .eml archive — gmail-bulk-import deliberately skipped inline text/plain/text/html parts (_parse_attachments does continue on them); this job reads exactly the content that gap left out, chunks it, embeds it, and inserts it into document_chunk alongside the existing paperless chunks.

Why a separate job, not an extension of documents-ingest's chunk_embed

chunk_embed.py is wired to source='paperless' + entities[type=content] (pre-extracted text already in the DB). Mail content isn't in the DB yet — it has to be read from .eml files, MIME-walked, quote-stripped, and classified, none of which paperless chunks need. The only thing genuinely shared is the chunker itself, which is why it was extracted to kb_mail.chunking first (Krok 0) instead of being copy-pasted here.

Where it runs

On SOLARIA (needs Ollama on localhost for /api/embed), against kb-postgres@PIHA over Tailscale. The .eml archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow and fragile.

Install (from repo root, on SOLARIA):

pip install -e packages/kb-mail/
pip install -e packages/kb-retrieval/
pip install -e jobs/mail-body-ingest/

Pipeline (per envelope)

1. Read archive_root / raw_ref — missing_file/read_error counted like gmail-header-backfill.
2. Parse: typed (policy.default) with a compat32 fallback (same ~9/225030 failure mode gmail-header-backfill documents).
3. Body extraction: inline text/plain preferred; HTML->text via a small stdlib HTMLParser when the mail is HTML-only (plan §1.3: 15% of the corpus) — zero new dependencies, skips style/script/head content.
4. Quote-strip (Decyzja 2): truncate at the earliest reply marker (On ... wrote:, Dnia ... napisał(a):, W dniu ... pisze:, -----Original Message-----, Outlook's underscore separator), then drop remaining >-quoted lines. In HTML, blockquote and div.gmail_quote subtrees are skipped before conversion to text. quoted_chars_stripped is tallied for calibration review (plan §7).
5. Classification: newsletter (List-Unsubscribe/List-Id/Precedence: bulk|list, read from the same parsed message); body_empty (after quote-strip — an empty mail is still counted, just produces zero chunks).
6. Prefix (Decyzya 3): Temat: ... | Od: ... | Data: YYYY-MM-DD built from the already-backfilled entities[type=headers] + envelope.ts — zero header re-parse.
7. Chunk: kb_mail.chunking.chunk_text (2400/600 chars, same as paperless).
8. Embed + insert: newsletter chunks are inserted immediately with excluded_reason='newsletter', embedding=NULL (no Ollama call, reversible later); everything else is buffered up to --batch-size (default 64) and sent through kb_retrieval.embed.embed_batch (/api/embed) before inserting. ON CONFLICT (envelope_id, chunk_index, model) DO NOTHING is checked via the command tag, so a silent no-op counts as chunks_conflict_skipped, never chunks_inserted.
9. Threading append (Decyzja 10): In-Reply-To/References (angle brackets stripped, matching envelope.id's bare-Message-ID convention) appended as entities[type=threading] via the same idempotent WHERE NOT EXISTS UPDATE pattern as gmail-header-backfill — done for every parsed envelope regardless of newsletter/ body_empty status, since it's the same read either way.

Stats must balance

mails_scanned = missing_file + read_errors + parse_errors + body_empty + mails_chunked
chunks_total  = chunks_inserted + chunks_newsletter_flagged + chunks_already_embedded
                + chunks_conflict_skipped + chunks_errors

Any non-zero read_errors/parse_errors/missing_file/chunks_errors/ chunks_conflict_skipped, or an unbalanced sum, makes the CLI exit 1 — same convention as gmail-header-backfill/documents-ingest's chunk_embed.

Reading exit 1 on a full-corpus run: it is a "look at this", not "the run failed". Across 225k mails a handful of parse_errors is expected (plan §1.5 documents ~9 mails that need the compat32 fallback), and any one of them alone trips exit 1. The verdict is the balance and the counters in the summary line, not the exit code. Exit 2 is different — see below.

Exit codes

Code	Meaning
0	Balanced, zero errors
1	Balanced-but-imperfect (any parse_errors/missing_file/read_errors/chunks_errors/chunks_conflict_skipped), an unbalanced sum, or an embedding-dimension abort
2	--max-embed-failures consecutive embed batches failed — the embed backend is down; re-run once it is back

Ollama-offline tolerance and the circuit breaker

A failed embed_batch() call is caught per-batch (aiohttp.ClientError -> the whole batch, up to --batch-size chunks, counts as chunks_errors; the run logs a warning and continues). Those chunks never enter the idempotency set, so a later re-run retries them automatically — no separate checkpointing needed.

Tolerating a flaky backend is right; surviving a dead one is not. The archive is parsed single-threaded ahead of the GPU, so on a full-corpus run (Etap B) a dead Ollama would let the job chew through 200k+ mails at parse speed, mark every chunk chunks_errors, and throw away a multi-hour pass. --max-embed-failures (default 5, 0 disables) therefore stops the run after that many consecutive failed batches, with exit code 2; a single successful batch resets the counter. Ollama@SOLARIA's known failure mode is total (container vanishes, network-detached — 4 incidents, plan §1.4/§7), so the breaker trips within seconds of it. On abort, pending entities[type=threading] appends are flushed first: they don't depend on Ollama, they're idempotent, and re-deriving them would mean re-reading the same 27 GB.

Only a wrong embedding dimension is more severe (EmbeddingDimensionError, exit 1) — it aborts immediately, since that would otherwise silently index a vector that doesn't match document_chunk.embedding VECTOR(1024).

Idempotency

Pre-fetched (envelope_id, chunk_index) pairs, scoped server-side to --model (WHERE model = $1), skip chunks already inserted — the pair doesn't need model redundantly since the fetch is already scoped to it. Built this way from the start per the plan's note that chunk_embed.py's otherwise-equivalent pre-fetch is easy to mis-key across runs that mix models (not an active bug there today, since one run always uses one model, but worth not repeating the ambiguity here).

Definition of Done

Per CLAUDE.md: pytest passes (48/48) + a --limit 5 dry-run smoke against live kb-postgres@PIHA before committing (confirms DSN/query wiring; a missing local archive mirror correctly reports missing_file rather than crashing). The Etap A pilot (--since 2025-07-01 --apply, plan §7) is a separate, explicitly-confirmed run — not part of this job's DoD, since it's the first real write against production data.