--- okf: "0.1" type: service visibility: private status: active updated: 2026-08-05 links: - ../runbooks/mail-body-ingest-run.md --- # mail-body-ingest Module 5, faza mailowa (`kb/phases/kb-m5-faza-mailowa.md`, §5, Krok 2). Second full pass over the gmail `.eml` archive — `gmail-bulk-import` deliberately skipped inline `text/plain`/`text/html` parts (`_parse_attachments` does `continue` on them); this job reads exactly the content that gap left out, chunks it, embeds it, and inserts it into `document_chunk` alongside the existing paperless chunks. ## Why a separate job, not an extension of documents-ingest's chunk_embed `chunk_embed.py` is wired to `source='paperless'` + `entities[type=content]` (pre-extracted text already in the DB). Mail content isn't in the DB yet — it has to be read from `.eml` files, MIME-walked, quote-stripped, and classified, none of which paperless chunks need. The only thing genuinely shared is the chunker itself, which is why it was extracted to `kb_mail.chunking` first (Krok 0) instead of being copy-pasted here. ## Where it runs **On SOLARIA** (needs Ollama on `localhost` for `/api/embed`), against `kb-postgres@PIHA` over Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow and fragile. Install (from repo root, on SOLARIA): ```bash pip install -e packages/kb-mail/ pip install -e packages/kb-retrieval/ pip install -e jobs/mail-body-ingest/ ``` ## Pipeline (per envelope) 1. **Read** `archive_root / raw_ref` — `missing_file`/`read_error` counted like `gmail-header-backfill`. 2. **Parse**: typed (`policy.default`) with a `compat32` fallback (same ~9/225030 failure mode `gmail-header-backfill` documents). 3. **Body extraction**: inline `text/plain` preferred; HTML->text via a small stdlib `HTMLParser` when the mail is HTML-only (plan §1.3: 15% of the corpus) — zero new dependencies, skips `style`/`script`/`head` content. 4. **Quote-strip** (Decyzja 2): truncate at the earliest reply marker (`On ... wrote:`, `Dnia ... napisał(a):`, `W dniu ... pisze:`, `-----Original Message-----`, Outlook's underscore separator), then drop remaining `>`-quoted lines. In HTML, `blockquote` and `div.gmail_quote` subtrees are skipped before conversion to text. `quoted_chars_stripped` is tallied for calibration review (plan §7). 5. **Classification**: `newsletter` (`List-Unsubscribe`/`List-Id`/`Precedence: bulk|list`, read from the same parsed message); `body_empty` (after quote-strip — an empty mail is still counted, just produces zero chunks). 6. **Prefix** (Decyzya 3): `Temat: ... | Od: ... | Data: YYYY-MM-DD` built from the already-backfilled `entities[type=headers]` + `envelope.ts` — zero header re-parse. 7. **Chunk**: `kb_mail.chunking.chunk_text` (2400/600 chars, same as paperless). 8. **Embed + insert**: newsletter chunks are inserted immediately with `excluded_reason='newsletter'`, `embedding=NULL` (no Ollama call, reversible later); everything else is buffered up to `--batch-size` (default 64) and sent through `kb_retrieval.embed.embed_batch_resilient` (`/api/embed` with `input` as a list) before inserting. `ON CONFLICT (envelope_id, chunk_index, model) DO NOTHING` is checked via the command tag, so a silent no-op counts as `chunks_conflict_skipped`, never `chunks_inserted`. 9. **Threading append** (Decyzja 10): `In-Reply-To`/`References` (angle brackets stripped, matching `envelope.id`'s bare-Message-ID convention) appended as `entities[type=threading]` via the same idempotent `WHERE NOT EXISTS` UPDATE pattern as `gmail-header-backfill` — done for every parsed envelope regardless of newsletter/ body_empty status, since it's the same read either way. ## Stats must balance ``` mails_scanned = missing_file + read_errors + parse_errors + body_empty + mails_chunked chunks_total = chunks_inserted + chunks_newsletter_flagged + chunks_already_embedded + chunks_conflict_skipped + chunks_errors ``` Any non-zero `read_errors`/`parse_errors`/`missing_file`/`chunks_errors`/ `chunks_conflict_skipped`, or an unbalanced sum, makes the CLI exit 1 — same convention as `gmail-header-backfill`/`documents-ingest`'s `chunk_embed`. **Reading exit 1 on a full-corpus run**: it is a "look at this", not "the run failed". Across 225k mails a handful of `parse_errors` is expected (plan §1.5 documents ~9 mails that need the compat32 fallback), and any one of them alone trips exit 1. The verdict is the balance and the counters in the `summary` line, not the exit code. Exit 2 is different — see below. ## Exit codes | Code | Meaning | |---|---| | 0 | Balanced, zero errors | | 1 | Balanced-but-imperfect (any `parse_errors`/`missing_file`/`read_errors`/`chunks_errors`/`chunks_conflict_skipped`), an unbalanced sum, or an embedding-dimension abort | | 2 | `--max-embed-failures` consecutive embed batches gave up — the embed backend is down; re-run once it is back | ## Ollama-offline tolerance and the circuit breaker Three failure modes, three responses — the first two live in `kb_retrieval.embed.embed_batch_resilient`, the third here: 1. **Transient blip** — the batch is retried `--embed-retries` times (default 2) with exponential backoff (`--embed-backoff`, default 1 s, doubled per attempt). Costs seconds, loses nothing. 2. **One poison chunk** — `/api/embed` is all-or-nothing, so a single pathological text used to cost the entire 64-chunk batch it happened to land in. When the retries are exhausted but `/api/tags` says the backend is alive, the batch is bisected until the bad inputs are isolated; the healthy remainder embeds normally and only the genuinely bad chunks count as `chunks_errors`. One bad chunk costs ~log2(batch) extra requests instead of 64 embeddings. 3. **Dead backend** — the probe says down, so the batch gives up *without* bisecting (splitting against a dead backend would burn 2n-1 requests and delay the breaker exactly when it needs to trip). Tolerating a *flaky* backend is right; surviving a *dead* one is not. The archive is parsed single-threaded ahead of the GPU, so on a full-corpus run (Etap B) a dead Ollama would let the job chew through 200k+ mails at parse speed, mark every chunk `chunks_errors`, and throw away a multi-hour pass. `--max-embed-failures` (default 5, `0` disables) therefore stops the run after that many *consecutive* give-ups, with exit code 2; any successful batch resets the counter. Ollama@SOLARIA's known failure mode is total (container vanishes, network-detached — 4 incidents, plan §1.4/§7), so the breaker trips within seconds of it. On abort, rows already embedded in the dying batch are committed and pending `entities[type=threading]` appends are flushed first: they don't depend on Ollama, they're idempotent, and re-deriving them would mean re-reading the same 27 GB. A partial failure against a *live* backend deliberately does **not** advance the breaker: those chunks were never inserted, so the next run retries them through the ordinary idempotency path, and letting a handful of bad chunks abort a 50k slice would be strictly worse than skipping them. `embed_items_failed` and the `embed_requests_total`/`embed_calls` ratio in the `summary` line are what to read — a ratio of 1.0 means a clean run with no retries or bisection. Timeouts are part of this, not an exception to it: aiohttp raises a bare `builtins.TimeoutError` when `ClientTimeout(total=...)` expires, and that is **not** a subclass of `aiohttp.ClientError`. An earlier version of this job caught `ClientError` alone, so an Ollama that accepted the connection and then hung — its actual failure mode — crashed the run outright, with no breaker and no threading flush. `kb_retrieval.embed.TRANSIENT_EMBED_ERRORS` now names both; catch that tuple, never `ClientError` on its own. Only a wrong embedding dimension is more severe (`EmbeddingDimensionError`, exit 1) — it aborts immediately, since that would otherwise silently index a vector that doesn't match `document_chunk.embedding VECTOR(1024)`. ## No SOLARIA -> PIHA fallback here (deliberate) `kb-query` fails a *query* over to Ollama@PIHA when SOLARIA is down (`app/embed_router.py`). This job does not, by decision: ~271k active chunks at PIHA's ~790 ms/embed CPU is ~60 h on an 8 GB node already shared with Home Assistant, Paperless and kb-postgres, and CPU embedding does not batch-scale the way the GPU does. A fallback would quietly turn "abort and resume the slice once Ollama is fixed" into a two-day run suffocating the infra node. The correct answer to a dead primary in the backfill path is to stop and resume — which costs nothing, because the job is idempotent. The two backend paths stay separate on purpose. ## Batch-size benchmark `mail-body-ingest-bench` (same package) sweeps batch sizes over real mail chunks and prints ms/chunk, chunks/s and a projected wall-clock for the full corpus. Read-only — `SELECT`s and inference, with no code path that can write — so it is safe to point at live kb-postgres@PIHA. ```bash mail-body-ingest-bench --dsn postgresql://kb:@piha:5433/kb \ --archive-root /home/oskar/kb/mail/archive --sample-envelopes 200 ``` It warms the model up before measuring (the first call after an idle period pays the model load) and measures the *same* chunk set at every size, since ms/chunk depends heavily on text length. `--max-chunks` caps the set — batch=1 otherwise dominates the wall clock. ## Idempotency Pre-fetched `(envelope_id, chunk_index)` pairs, scoped server-side to `--model` (`WHERE model = $1`), skip chunks already inserted — the pair doesn't need `model` redundantly since the fetch is already scoped to it. Built this way from the start per the plan's note that `chunk_embed.py`'s otherwise-equivalent pre-fetch is easy to mis-key across runs that mix models (not an active bug there today, since one run always uses one model, but worth not repeating the ambiguity here). ## Definition of Done Per `CLAUDE.md`: `pytest` passes (62/62 for the job, 22/22 for `kb-retrieval`'s embed client) + a `--limit 5` dry-run smoke against live `kb-postgres@PIHA` before committing (confirms DSN/query wiring; a missing local archive mirror correctly reports `missing_file` rather than crashing). The Etap A pilot (`--since 2025-07-01 --apply`, plan §7) is a separate, explicitly-confirmed run — not part of this job's DoD, since it's the first real write against production data.