homelab-codex-ws/kb/services/job-mail-body-ingest.md

okf	type	visibility	status	updated	links
0.1	service	private	active	2026-08-06	../runbooks/mail-body-ingest-run.md job-mail-imap-sync.md

mail-body-ingest

Module 5, faza mailowa (kb/phases/kb-m5-faza-mailowa.md, §5, Krok 2). Second full pass over the gmail .eml archive — gmail-bulk-import deliberately skipped inline text/plain/text/html parts (_parse_attachments does continue on them); this job reads exactly the content that gap left out, chunks it, embeds it, and inserts it into document_chunk alongside the existing paperless chunks.

Why a separate job, not an extension of documents-ingest's chunk_embed

chunk_embed.py is wired to source='paperless' + entities[type=content] (pre-extracted text already in the DB). Mail content isn't in the DB yet — it has to be read from .eml files, MIME-walked, quote-stripped, and classified, none of which paperless chunks need. The only thing genuinely shared is the chunker itself, which is why it was extracted to kb_mail.chunking first (Krok 0) instead of being copy-pasted here.

Where it runs

On SOLARIA for backfill runs (needs Ollama on localhost for /api/embed), against kb-postgres@PIHA over Tailscale. The .eml archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow and fragile.

Also on PIHA since 2026-08-06, as the mail stage of kb-ingest (see "Two modes" below). There it reads the canonical archive locally and calls Ollama@SOLARIA over Tailscale — which is fine at this scale: the daily increment is ~60 chunks at 8-18 ms each, measured at 8.2 ms round-trip from PIHA. The argument against embedding from PIHA (recon §3.1) was about 271k backfill chunks and does not carry over.

Install (from repo root, on SOLARIA):

pip install -e packages/kb-mail/
pip install -e packages/kb-retrieval/
pip install -e jobs/mail-body-ingest/

Pipeline (per envelope)

1. Read archive_root / raw_ref — missing_file/read_error counted like gmail-header-backfill.
2. Parse: typed (policy.default) with a compat32 fallback (same ~9/225030 failure mode gmail-header-backfill documents).
3. Body extraction: inline text/plain preferred; HTML->text via a small stdlib HTMLParser when the mail is HTML-only (plan §1.3: 15% of the corpus) — zero new dependencies, skips style/script/head content.
4. Quote-strip (Decyzja 2): truncate at the earliest reply marker (On ... wrote:, Dnia ... napisał(a):, W dniu ... pisze:, -----Original Message-----, Outlook's underscore separator), then drop remaining >-quoted lines. In HTML, blockquote and div.gmail_quote subtrees are skipped before conversion to text. quoted_chars_stripped is tallied for calibration review (plan §7).
5. Classification: newsletter (List-Unsubscribe/List-Id/Precedence: bulk|list, read from the same parsed message); body_empty (after quote-strip — an empty mail is still counted, just produces zero chunks).
6. Prefix (Decyzya 3): Temat: ... | Od: ... | Data: YYYY-MM-DD built from the already-backfilled entities[type=headers] + envelope.ts — zero header re-parse.
7. Chunk: kb_mail.chunking.chunk_text (2400/600 chars, same as paperless).
8. Embed + insert: newsletter chunks are inserted immediately with excluded_reason='newsletter', embedding=NULL (no Ollama call, reversible later); everything else is buffered up to --batch-size (default 64) and sent through kb_retrieval.embed.embed_batch_resilient (/api/embed with input as a list) before inserting. ON CONFLICT (envelope_id, chunk_index, model) DO NOTHING is checked via the command tag, so a silent no-op counts as chunks_conflict_skipped, never chunks_inserted.
9. Threading append (Decyzja 10): In-Reply-To/References (angle brackets stripped, matching envelope.id's bare-Message-ID convention) appended as entities[type=threading] via the same idempotent WHERE NOT EXISTS UPDATE pattern as gmail-header-backfill — done for every parsed envelope regardless of newsletter/ body_empty status, since it's the same read either way.

Two modes: backfill slice vs cyclic queue

The job was written for supervised full-corpus slices (--limit/--offset/--since). Since 2026-08-06 it also runs as a stage of kb-ingest every 2 hours, draining what the incremental IMAP sync fetched:

mail-body-ingest --only-unchunked --apply          # cyclic mode

--only-unchunked restricts the set to envelopes with no document_chunk rows at all — the queue that the data itself defines (recon kb/audits/mail-sync-2026-08-06.md §3.2). It needs no separate state and is self-healing: an interrupted run, a mail fetched while Ollama was down, a message inserted with a backdated header — each stays pending until it has chunks.

--since was explicitly rejected for that role. It filters on envelope.ts, the sender's Date: header, so a mail delivered today carrying a month-old date would fall outside the window and never be chunked at all. --since stays what it was: a tool for staging manual runs.

--sources defaults to gmail,fastmail. The pre-fetched chunk-key set is scoped to the working set (WHERE model = $1 AND envelope_id = ANY($2)); unbounded it read all 389 012 keys (~26 MB, ~1.0 s) on every run, which is nothing once for a 50k slice and wasteful every two hours on a node with 2.4 GB available (recon §2.5 iv).

Stats must balance

mails_scanned = missing_file + read_errors + parse_errors + body_empty + mails_chunked
chunks_total  = chunks_inserted + chunks_newsletter_flagged + chunks_already_embedded
                + chunks_conflict_skipped + chunks_errors

Any non-zero read_errors/parse_errors/missing_file/chunks_errors/ chunks_conflict_skipped, or an unbalanced sum, makes the CLI exit 1 — same convention as gmail-header-backfill/documents-ingest's chunk_embed.

Reading exit 1 on a full-corpus run: it is a "look at this", not "the run failed". Across 225k mails a handful of parse_errors is expected (plan §1.5 documents ~9 mails that need the compat32 fallback), and any one of them alone trips exit 1. The verdict is the balance and the counters in the summary line, not the exit code. Exit 2 is different — see below.

Exit codes

Code	Meaning
0	Balanced, zero errors
1	Balanced-but-imperfect (any parse_errors/missing_file/read_errors/chunks_errors/chunks_conflict_skipped), an unbalanced sum, or an embedding-dimension abort
2	--max-embed-failures consecutive embed batches gave up — the embed backend is down; re-run once it is back

Ollama-offline tolerance and the circuit breaker

Three failure modes, three responses — the first two live in kb_retrieval.embed.embed_batch_resilient, the third here:

1. Transient blip — the batch is retried --embed-retries times (default 2) with exponential backoff (--embed-backoff, default 1 s, doubled per attempt). Costs seconds, loses nothing.
2. One poison chunk — /api/embed is all-or-nothing, so a single pathological text used to cost the entire 64-chunk batch it happened to land in. When the retries are exhausted but /api/tags says the backend is alive, the batch is bisected until the bad inputs are isolated; the healthy remainder embeds normally and only the genuinely bad chunks count as chunks_errors. One bad chunk costs ~log2(batch) extra requests instead of 64 embeddings.
3. Dead backend — the probe says down, so the batch gives up without bisecting (splitting against a dead backend would burn 2n-1 requests and delay the breaker exactly when it needs to trip).

Tolerating a flaky backend is right; surviving a dead one is not. The archive is parsed single-threaded ahead of the GPU, so on a full-corpus run (Etap B) a dead Ollama would let the job chew through 200k+ mails at parse speed, mark every chunk chunks_errors, and throw away a multi-hour pass. --max-embed-failures (default 5, 0 disables) therefore stops the run after that many consecutive give-ups, with exit code 2; any successful batch resets the counter. Ollama@SOLARIA's known failure mode is total (container vanishes, network-detached — 4 incidents, plan §1.4/§7), so the breaker trips within seconds of it. On abort, rows already embedded in the dying batch are committed and pending entities[type=threading] appends are flushed first: they don't depend on Ollama, they're idempotent, and re-deriving them would mean re-reading the same 27 GB.

A partial failure against a live backend deliberately does not advance the breaker: those chunks were never inserted, so the next run retries them through the ordinary idempotency path, and letting a handful of bad chunks abort a 50k slice would be strictly worse than skipping them. embed_items_failed and the embed_requests_total/embed_calls ratio in the summary line are what to read — a ratio of 1.0 means a clean run with no retries or bisection.

Timeouts are part of this, not an exception to it: aiohttp raises a bare builtins.TimeoutError when ClientTimeout(total=...) expires, and that is not a subclass of aiohttp.ClientError. An earlier version of this job caught ClientError alone, so an Ollama that accepted the connection and then hung — its actual failure mode — crashed the run outright, with no breaker and no threading flush. kb_retrieval.embed.TRANSIENT_EMBED_ERRORS now names both; catch that tuple, never ClientError on its own.

Only a wrong embedding dimension is more severe (EmbeddingDimensionError, exit 1) — it aborts immediately, since that would otherwise silently index a vector that doesn't match document_chunk.embedding VECTOR(1024).

No SOLARIA -> PIHA fallback here (deliberate)

kb-query fails a query over to Ollama@PIHA when SOLARIA is down (app/embed_router.py). This job does not, by decision: ~271k active chunks at PIHA's ~790 ms/embed CPU is ~60 h on an 8 GB node already shared with Home Assistant, Paperless and kb-postgres, and CPU embedding does not batch-scale the way the GPU does. A fallback would quietly turn "abort and resume the slice once Ollama is fixed" into a two-day run suffocating the infra node. The correct answer to a dead primary in the backfill path is to stop and resume — which costs nothing, because the job is idempotent. The two backend paths stay separate on purpose.

Batch-size benchmark

mail-body-ingest-bench (same package) sweeps batch sizes over real mail chunks and prints ms/chunk, chunks/s and a projected wall-clock for the full corpus. Read-only — SELECTs and inference, with no code path that can write — so it is safe to point at live kb-postgres@PIHA.

mail-body-ingest-bench --dsn postgresql://kb:<pw>@piha:5433/kb \
    --archive-root /home/oskar/kb/mail/archive --sample-envelopes 200

It warms the model up before measuring (the first call after an idle period pays the model load) and measures the same chunk set at every size, since ms/chunk depends heavily on text length. --max-chunks caps the set — batch=1 otherwise dominates the wall clock.

Idempotency

Pre-fetched (envelope_id, chunk_index) pairs, scoped server-side to --model (WHERE model = $1), skip chunks already inserted — the pair doesn't need model redundantly since the fetch is already scoped to it. Built this way from the start per the plan's note that chunk_embed.py's otherwise-equivalent pre-fetch is easy to mis-key across runs that mix models (not an active bug there today, since one run always uses one model, but worth not repeating the ambiguity here).

Definition of Done

Per CLAUDE.md: pytest passes (62/62 for the job, 22/22 for kb-retrieval's embed client)

• a --limit 5 dry-run smoke against live kb-postgres@PIHA before committing (confirms DSN/query wiring; a missing local archive mirror correctly reports missing_file rather than crashing). The Etap A pilot (--since 2025-07-01 --apply, plan §7) is a separate, explicitly-confirmed run — not part of this job's DoD, since it's the first real write against production data.