17640526ef
Tolerancja pojedynczego nieudanego batcha jest słuszna, przeżycie martwej Ollamy już nie. Parse archiwum jest jednowątkowy i wyprzedza GPU, więc przy Etapie B (~212k kopert bez --since) padnięta Ollama przemieliłaby resztę korpusu z prędkością parse'u, oznaczając każdy chunk jako chunks_errors — bez ani jednego zapisu, ale kosztem ~2 h przebiegu do powtórzenia. Znany tryb awarii Ollama@SOLARIA jest totalny (zniknięcie kontenera / network-detach, 4 incydenty, §1.4/§7), nie częściowy, więc próg z kolejnych porażek trafia w niego od razu. --max-embed-failures N (domyślnie 5, 0 wyłącza) → EmbedBackendUnavailableError i exit 2, odrębny od exit 1 (który pełny korpus osiąga legalnie na pojedynczych parse_errors — §1.5). Licznik zeruje się po udanym batchu, więc kryterium jest "kolejnych", nie "łącznie". Przy abortcie dopychane są zaległe wpisy entities[type=threading]: nie zależą od Ollamy, są idempotentne, a ich odtworzenie oznaczałoby ponowny odczyt tych samych 27 GB. Nowy licznik embed_batch_failures jest wyłącznie diagnostyczny — równania bilansu bez zmian. Plan §9: dopisane decyzje operatora do Etapu B (plastry po 50k, breaker, pominięty dry-run całości, hybrid default poza zakresem) + nota jak czytać exit 1 vs exit 2. Testy: 4 nowe (trip po N kolejnych, reset po sukcesie, 0 wyłącza, flush threadingu przy abortcie); 55 passed mail-body-ingest, 25 passed kb-retrieval. Smoke: --limit 5 dry-run na żywym kb-postgres@PIHA — bilans domknięty, zero zapisów, zero wywołań Ollamy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
164 lines
8.8 KiB
Markdown
164 lines
8.8 KiB
Markdown
# mail-body-ingest
|
|
|
|
Module 5, faza mailowa (`docs/kb/modules/05-faza-mailowa-plan.md`, §5, Krok 2). Second full
|
|
pass over the gmail `.eml` archive — `gmail-bulk-import` deliberately skipped inline
|
|
`text/plain`/`text/html` parts (`_parse_attachments` does `continue` on them); this job reads
|
|
exactly the content that gap left out, chunks it, embeds it, and inserts it into
|
|
`document_chunk` alongside the existing paperless chunks.
|
|
|
|
## Why a separate job, not an extension of documents-ingest's chunk_embed
|
|
|
|
`chunk_embed.py` is wired to `source='paperless'` + `entities[type=content]` (pre-extracted
|
|
text already in the DB). Mail content isn't in the DB yet — it has to be read from `.eml`
|
|
files, MIME-walked, quote-stripped, and classified, none of which paperless chunks need. The
|
|
only thing genuinely shared is the chunker itself, which is why it was extracted to
|
|
`kb_mail.chunking` first (Krok 0) instead of being copy-pasted here.
|
|
|
|
## Where it runs
|
|
|
|
**On SOLARIA** (needs Ollama on `localhost` for `/api/embed`), against `kb-postgres@PIHA` over
|
|
Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than
|
|
read live over the network — 225k small files over Tailscale would be slow and fragile.
|
|
|
|
Install (from repo root, on SOLARIA):
|
|
|
|
```bash
|
|
pip install -e packages/kb-mail/
|
|
pip install -e packages/kb-retrieval/
|
|
pip install -e jobs/mail-body-ingest/
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero
|
|
# DB writes (including the threading UPDATE):
|
|
mail-body-ingest --dsn postgresql://kb:<pw>@piha:5433/kb --archive-root /home/oskar/kb/mail/archive
|
|
|
|
# Etap A pilot — last 12 months only (plan Decyzja 9):
|
|
mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1
|
|
|
|
# Etap B — full archive in 50k slices (plan §9; ORDER BY id is stable, so slices are
|
|
# reproducible, and idempotency covers their boundaries):
|
|
nice -n 10 ionice -c2 -n7 mail-body-ingest --dsn ... --apply \
|
|
--limit 50000 --offset 0 > mail-ingest-etapB-0.log 2>&1
|
|
|
|
# Smoke-test slice:
|
|
mail-body-ingest --dsn ... --apply --limit 10
|
|
```
|
|
|
|
DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default
|
|
`http://localhost:11434` — this job is meant to run where Ollama lives).
|
|
|
|
## Pipeline (per envelope)
|
|
|
|
1. **Read** `archive_root / raw_ref` — `missing_file`/`read_error` counted like
|
|
`gmail-header-backfill`.
|
|
2. **Parse**: typed (`policy.default`) with a `compat32` fallback (same ~9/225030 failure mode
|
|
`gmail-header-backfill` documents).
|
|
3. **Body extraction**: inline `text/plain` preferred; HTML->text via a small stdlib
|
|
`HTMLParser` when the mail is HTML-only (plan §1.3: 15% of the corpus) — zero new
|
|
dependencies, skips `style`/`script`/`head` content.
|
|
4. **Quote-strip** (Decyzja 2): truncate at the earliest reply marker (`On ... wrote:`,
|
|
`Dnia ... napisał(a):`, `W dniu ... pisze:`, `-----Original Message-----`, Outlook's
|
|
underscore separator), then drop remaining `>`-quoted lines. In HTML, `blockquote` and
|
|
`div.gmail_quote` subtrees are skipped before conversion to text. `quoted_chars_stripped`
|
|
is tallied for calibration review (plan §7).
|
|
5. **Classification**: `newsletter` (`List-Unsubscribe`/`List-Id`/`Precedence: bulk|list`,
|
|
read from the same parsed message); `body_empty` (after quote-strip — an empty mail is
|
|
still counted, just produces zero chunks).
|
|
6. **Prefix** (Decyzya 3): `Temat: ... | Od: ... | Data: YYYY-MM-DD` built from the
|
|
already-backfilled `entities[type=headers]` + `envelope.ts` — zero header re-parse.
|
|
7. **Chunk**: `kb_mail.chunking.chunk_text` (2400/600 chars, same as paperless).
|
|
8. **Embed + insert**: newsletter chunks are inserted immediately with
|
|
`excluded_reason='newsletter'`, `embedding=NULL` (no Ollama call, reversible later);
|
|
everything else is buffered up to `--batch-size` (default 64) and sent through
|
|
`kb_retrieval.embed.embed_batch` (`/api/embed`) before inserting. `ON CONFLICT
|
|
(envelope_id, chunk_index, model) DO NOTHING` is checked via the command tag, so a
|
|
silent no-op counts as `chunks_conflict_skipped`, never `chunks_inserted`.
|
|
9. **Threading append** (Decyzja 10): `In-Reply-To`/`References` (angle brackets stripped,
|
|
matching `envelope.id`'s bare-Message-ID convention) appended as
|
|
`entities[type=threading]` via the same idempotent `WHERE NOT EXISTS` UPDATE pattern as
|
|
`gmail-header-backfill` — done for every parsed envelope regardless of newsletter/
|
|
body_empty status, since it's the same read either way.
|
|
|
|
## Stats must balance
|
|
|
|
```
|
|
mails_scanned = missing_file + read_errors + parse_errors + body_empty + mails_chunked
|
|
chunks_total = chunks_inserted + chunks_newsletter_flagged + chunks_already_embedded
|
|
+ chunks_conflict_skipped + chunks_errors
|
|
```
|
|
|
|
Any non-zero `read_errors`/`parse_errors`/`missing_file`/`chunks_errors`/
|
|
`chunks_conflict_skipped`, or an unbalanced sum, makes the CLI exit 1 — same convention as
|
|
`gmail-header-backfill`/`documents-ingest`'s `chunk_embed`.
|
|
|
|
**Reading exit 1 on a full-corpus run**: it is a "look at this", not "the run failed". Across
|
|
225k mails a handful of `parse_errors` is expected (plan §1.5 documents ~9 mails that need the
|
|
compat32 fallback), and any one of them alone trips exit 1. The verdict is the balance and the
|
|
counters in the `summary` line, not the exit code. Exit 2 is different — see below.
|
|
|
|
## Exit codes
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| 0 | Balanced, zero errors |
|
|
| 1 | Balanced-but-imperfect (any `parse_errors`/`missing_file`/`read_errors`/`chunks_errors`/`chunks_conflict_skipped`), an unbalanced sum, or an embedding-dimension abort |
|
|
| 2 | `--max-embed-failures` consecutive embed batches failed — the embed backend is down; re-run once it is back |
|
|
|
|
## Ollama-offline tolerance and the circuit breaker
|
|
|
|
A failed `embed_batch()` call is caught per-batch (`aiohttp.ClientError` -> the whole batch,
|
|
up to `--batch-size` chunks, counts as `chunks_errors`; the run logs a warning and continues).
|
|
Those chunks never enter the idempotency set, so a later re-run retries them automatically —
|
|
no separate checkpointing needed.
|
|
|
|
Tolerating a *flaky* backend is right; surviving a *dead* one is not. The archive is parsed
|
|
single-threaded ahead of the GPU, so on a full-corpus run (Etap B) a dead Ollama would let the
|
|
job chew through 200k+ mails at parse speed, mark every chunk `chunks_errors`, and throw away a
|
|
multi-hour pass. `--max-embed-failures` (default 5, `0` disables) therefore stops the run after
|
|
that many *consecutive* failed batches, with exit code 2; a single successful batch resets the
|
|
counter. Ollama@SOLARIA's known failure mode is total (container vanishes, network-detached —
|
|
4 incidents, plan §1.4/§7), so the breaker trips within seconds of it. On abort, pending
|
|
`entities[type=threading]` appends are flushed first: they don't depend on Ollama, they're
|
|
idempotent, and re-deriving them would mean re-reading the same 27 GB.
|
|
|
|
Only a wrong embedding dimension is more severe (`EmbeddingDimensionError`, exit 1) — it aborts
|
|
immediately, since that would otherwise silently index a vector that doesn't match
|
|
`document_chunk.embedding VECTOR(1024)`.
|
|
|
|
## Idempotency
|
|
|
|
Pre-fetched `(envelope_id, chunk_index)` pairs, scoped server-side to `--model` (`WHERE model
|
|
= $1`), skip chunks already inserted — the pair doesn't need `model` redundantly since the
|
|
fetch is already scoped to it. Built this way from the start per the plan's note that
|
|
`chunk_embed.py`'s otherwise-equivalent pre-fetch is easy to mis-key across runs that mix
|
|
models (not an active bug there today, since one run always uses one model, but worth not
|
|
repeating the ambiguity here).
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pip install -e "jobs/mail-body-ingest[dev]"
|
|
cd jobs/mail-body-ingest && pytest
|
|
```
|
|
|
|
Pure unit tests (55), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect`
|
|
and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers:
|
|
quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/
|
|
gmail_quote skipping), newsletter classification, threading extraction, prefix building,
|
|
body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse
|
|
fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks
|
|
never reaching Ollama, Ollama-offline batch isolation, dimension-mismatch abort, and the
|
|
circuit breaker (trips on N consecutive failures, resets on a success, disabled by `0`,
|
|
flushes pending threading on abort).
|
|
|
|
## Definition of Done
|
|
|
|
Per `CLAUDE.md`: `pytest` passes (48/48) + a `--limit 5` dry-run smoke against live
|
|
`kb-postgres@PIHA` before committing (confirms DSN/query wiring; a missing local archive
|
|
mirror correctly reports `missing_file` rather than crashing). The Etap A pilot (`--since
|
|
2025-07-01 --apply`, plan §7) is a separate, explicitly-confirmed run — not part of this
|
|
job's DoD, since it's the first real write against production data.
|