3eec5182a0
Domkniecie Kroku 7. Realizuje Decyzje (d) reconu (host schedulera + korekta
kadencji indeksowania) i doklada dokumentacje wg konwencji OKF.
Scheduler (NIEAKTYWOWANY — wlacza operator):
- jobs/mail-imap-sync/systemd/{service,timer,run.sh} — wzorzec 1:1 z kb-ingest,
OnCalendar=hourly, Persistent=true, log do pliku (nigdy sam journal).
- hosts/piha/jobs.yaml — deklaracja jednostek host-level na PIHA. Nowy plik, bo
services.yaml jest dla kontenerow (supervisor dopasowuje jego wpisy do world-state
i wpis niekontenerowy dryfowalby wiecznie jako missing_service). Nic tego pliku
nie czyta — istnieje po to, zeby "shadow-deploy family" z otwartego pytania 5
reconu multiagentowego byla spisana, a nie tylko na nodzie.
Takt indeksowania (Decyzja (d), recon §3.3):
- kb-ingest.timer: 03:30 raz na dobe -> co 2 h. O 03:30 SOLARIA prawie na pewno spi
(potwierdzone odczytem kb_ingest_embed_skipped 1 z 2026-08-06), a tick dostaje
teraz etap mailowy: ~60 nowych chunkow na dobe pomijanych kazdej nocy sprawiloby,
ze backlog rosnie monotonicznie i KbEmbedBacklogGrowing zapala sie NA STALE.
Co 2 h zamiast stalej godziny — probe Ollamy sam wybiera okno, wiec ktorys tick
w nie trafi niezaleznie od nawykow operatora.
- cyclic_ingest: etap mailowy (mail_body_ingest --only-unchunked), import miekki,
wiec venv bez tego pakietu pomija etap zamiast wywracac wrapper. Predykat bledu
JEST luzniejszy niz wlasne main() tamtego joba i to jedyne takie miejsce w tym
wrapperze: pojedynczy trwale nieparsowalny mail nie moze zamrozic
last_success_timestamp i zapalic KbIngestStale na zawsze. Bledy per-mail sa
publikowane jako kb_ingest_mail_parse_errors, nie chowane.
Obserwowalnosc: KbMailSyncStale (6 h bez udanego ticku). Alert na cisze w skrzynce
ODRZUCONY (decyzja operatora, zgodna z reconem §3.4) — zero nowych maili to legalny
stan skrzynki, a alert zapalajacy sie na zdrowym systemie zostaje wyciszony
i przestaje dzialac wtedy, gdy jest potrzebny.
Dokumentacja:
- kb/services/job-mail-imap-sync.md (OKF), kb/runbooks/mail-sync-run.md — 9 krokow
pierwszego uruchomienia, w tym checklista 4 punktow [do weryfikacji na zywo]
z reconu (polityki dostawcow — do sprawdzenia, nie do zgadniecia) oraz pomiar
STATUS (MESSAGES) na Fastmailu, na ktorym zapada ODLOZONA decyzja o historii.
- kb-mail-pillar.md: KOREKTA JMAP -> IMAP dla Fastmaila jako decyzja 2026-08-06;
stary zapis zostaje jako historia z data. Zamkniete "unifikacja adaptera"
i "sizing Gmaila"; otwarte zostaje "sizing Fastmaila" — celowo, bo rozstrzyga
je pomiar, nie dyskusja.
- kb-m5-faza-mailowa.md: Krok 7 IN PROGRESS + tabela zakresu wdrozonego,
kb-m5-faza3.md: korekta harmonogramu i sekwencji wrappera,
pkg-kb-mail.md: rozpisany ze stubu, kb-postgres.md: lista migracji + 005.
Testy: 642 passed (calosc kb-mail, kb-retrieval i jobs). systemd-analyze verify
na timerze przechodzi, OnCalendar=0/2:00:00 normalizuje sie do co 2 h.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
218 lines
12 KiB
Markdown
218 lines
12 KiB
Markdown
---
|
|
okf: "0.1"
|
|
type: service
|
|
visibility: private
|
|
status: active
|
|
updated: 2026-08-06
|
|
links:
|
|
- ../runbooks/mail-body-ingest-run.md
|
|
- job-mail-imap-sync.md
|
|
---
|
|
|
|
# mail-body-ingest
|
|
|
|
Module 5, faza mailowa (`kb/phases/kb-m5-faza-mailowa.md`, §5, Krok 2). Second full
|
|
pass over the gmail `.eml` archive — `gmail-bulk-import` deliberately skipped inline
|
|
`text/plain`/`text/html` parts (`_parse_attachments` does `continue` on them); this job reads
|
|
exactly the content that gap left out, chunks it, embeds it, and inserts it into
|
|
`document_chunk` alongside the existing paperless chunks.
|
|
|
|
## Why a separate job, not an extension of documents-ingest's chunk_embed
|
|
|
|
`chunk_embed.py` is wired to `source='paperless'` + `entities[type=content]` (pre-extracted
|
|
text already in the DB). Mail content isn't in the DB yet — it has to be read from `.eml`
|
|
files, MIME-walked, quote-stripped, and classified, none of which paperless chunks need. The
|
|
only thing genuinely shared is the chunker itself, which is why it was extracted to
|
|
`kb_mail.chunking` first (Krok 0) instead of being copy-pasted here.
|
|
|
|
## Where it runs
|
|
|
|
**On SOLARIA** for backfill runs (needs Ollama on `localhost` for `/api/embed`), against
|
|
`kb-postgres@PIHA` over Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7,
|
|
Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow
|
|
and fragile.
|
|
|
|
**Also on PIHA since 2026-08-06**, as the mail stage of `kb-ingest` (see "Two modes" below).
|
|
There it reads the canonical archive locally and calls Ollama@SOLARIA over Tailscale — which
|
|
is fine at this scale: the daily increment is ~60 chunks at 8-18 ms each, measured at 8.2 ms
|
|
round-trip from PIHA. The argument against embedding from PIHA (recon §3.1) was about 271k
|
|
backfill chunks and does not carry over.
|
|
|
|
Install (from repo root, on SOLARIA):
|
|
|
|
```bash
|
|
pip install -e packages/kb-mail/
|
|
pip install -e packages/kb-retrieval/
|
|
pip install -e jobs/mail-body-ingest/
|
|
```
|
|
|
|
## Pipeline (per envelope)
|
|
|
|
1. **Read** `archive_root / raw_ref` — `missing_file`/`read_error` counted like
|
|
`gmail-header-backfill`.
|
|
2. **Parse**: typed (`policy.default`) with a `compat32` fallback (same ~9/225030 failure mode
|
|
`gmail-header-backfill` documents).
|
|
3. **Body extraction**: inline `text/plain` preferred; HTML->text via a small stdlib
|
|
`HTMLParser` when the mail is HTML-only (plan §1.3: 15% of the corpus) — zero new
|
|
dependencies, skips `style`/`script`/`head` content.
|
|
4. **Quote-strip** (Decyzja 2): truncate at the earliest reply marker (`On ... wrote:`,
|
|
`Dnia ... napisał(a):`, `W dniu ... pisze:`, `-----Original Message-----`, Outlook's
|
|
underscore separator), then drop remaining `>`-quoted lines. In HTML, `blockquote` and
|
|
`div.gmail_quote` subtrees are skipped before conversion to text. `quoted_chars_stripped`
|
|
is tallied for calibration review (plan §7).
|
|
5. **Classification**: `newsletter` (`List-Unsubscribe`/`List-Id`/`Precedence: bulk|list`,
|
|
read from the same parsed message); `body_empty` (after quote-strip — an empty mail is
|
|
still counted, just produces zero chunks).
|
|
6. **Prefix** (Decyzya 3): `Temat: ... | Od: ... | Data: YYYY-MM-DD` built from the
|
|
already-backfilled `entities[type=headers]` + `envelope.ts` — zero header re-parse.
|
|
7. **Chunk**: `kb_mail.chunking.chunk_text` (2400/600 chars, same as paperless).
|
|
8. **Embed + insert**: newsletter chunks are inserted immediately with
|
|
`excluded_reason='newsletter'`, `embedding=NULL` (no Ollama call, reversible later);
|
|
everything else is buffered up to `--batch-size` (default 64) and sent through
|
|
`kb_retrieval.embed.embed_batch_resilient` (`/api/embed` with `input` as a list) before
|
|
inserting. `ON CONFLICT (envelope_id, chunk_index, model) DO NOTHING` is checked via the
|
|
command tag, so a silent no-op counts as `chunks_conflict_skipped`, never
|
|
`chunks_inserted`.
|
|
9. **Threading append** (Decyzja 10): `In-Reply-To`/`References` (angle brackets stripped,
|
|
matching `envelope.id`'s bare-Message-ID convention) appended as
|
|
`entities[type=threading]` via the same idempotent `WHERE NOT EXISTS` UPDATE pattern as
|
|
`gmail-header-backfill` — done for every parsed envelope regardless of newsletter/
|
|
body_empty status, since it's the same read either way.
|
|
|
|
## Two modes: backfill slice vs cyclic queue
|
|
|
|
The job was written for supervised full-corpus slices (`--limit`/`--offset`/`--since`). Since
|
|
2026-08-06 it also runs as a stage of `kb-ingest` every 2 hours, draining what the incremental
|
|
IMAP sync fetched:
|
|
|
|
```bash
|
|
mail-body-ingest --only-unchunked --apply # cyclic mode
|
|
```
|
|
|
|
`--only-unchunked` restricts the set to envelopes with **no `document_chunk` rows at all** —
|
|
the queue that the data itself defines (recon `kb/audits/mail-sync-2026-08-06.md` §3.2). It
|
|
needs no separate state and is self-healing: an interrupted run, a mail fetched while Ollama
|
|
was down, a message inserted with a backdated header — each stays pending until it has chunks.
|
|
|
|
`--since` was explicitly rejected for that role. It filters on `envelope.ts`, the sender's
|
|
`Date:` header, so a mail delivered today carrying a month-old date would fall outside the
|
|
window and never be chunked at all. `--since` stays what it was: a tool for staging manual runs.
|
|
|
|
`--sources` defaults to `gmail,fastmail`. The pre-fetched chunk-key set is scoped to the
|
|
working set (`WHERE model = $1 AND envelope_id = ANY($2)`); unbounded it read all 389 012 keys
|
|
(~26 MB, ~1.0 s) on every run, which is nothing once for a 50k slice and wasteful every two
|
|
hours on a node with 2.4 GB available (recon §2.5 iv).
|
|
|
|
## Stats must balance
|
|
|
|
```
|
|
mails_scanned = missing_file + read_errors + parse_errors + body_empty + mails_chunked
|
|
chunks_total = chunks_inserted + chunks_newsletter_flagged + chunks_already_embedded
|
|
+ chunks_conflict_skipped + chunks_errors
|
|
```
|
|
|
|
Any non-zero `read_errors`/`parse_errors`/`missing_file`/`chunks_errors`/
|
|
`chunks_conflict_skipped`, or an unbalanced sum, makes the CLI exit 1 — same convention as
|
|
`gmail-header-backfill`/`documents-ingest`'s `chunk_embed`.
|
|
|
|
**Reading exit 1 on a full-corpus run**: it is a "look at this", not "the run failed". Across
|
|
225k mails a handful of `parse_errors` is expected (plan §1.5 documents ~9 mails that need the
|
|
compat32 fallback), and any one of them alone trips exit 1. The verdict is the balance and the
|
|
counters in the `summary` line, not the exit code. Exit 2 is different — see below.
|
|
|
|
## Exit codes
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| 0 | Balanced, zero errors |
|
|
| 1 | Balanced-but-imperfect (any `parse_errors`/`missing_file`/`read_errors`/`chunks_errors`/`chunks_conflict_skipped`), an unbalanced sum, or an embedding-dimension abort |
|
|
| 2 | `--max-embed-failures` consecutive embed batches gave up — the embed backend is down; re-run once it is back |
|
|
|
|
## Ollama-offline tolerance and the circuit breaker
|
|
|
|
Three failure modes, three responses — the first two live in
|
|
`kb_retrieval.embed.embed_batch_resilient`, the third here:
|
|
|
|
1. **Transient blip** — the batch is retried `--embed-retries` times (default 2) with
|
|
exponential backoff (`--embed-backoff`, default 1 s, doubled per attempt). Costs seconds,
|
|
loses nothing.
|
|
2. **One poison chunk** — `/api/embed` is all-or-nothing, so a single pathological text used to
|
|
cost the entire 64-chunk batch it happened to land in. When the retries are exhausted but
|
|
`/api/tags` says the backend is alive, the batch is bisected until the bad inputs are
|
|
isolated; the healthy remainder embeds normally and only the genuinely bad chunks count as
|
|
`chunks_errors`. One bad chunk costs ~log2(batch) extra requests instead of 64 embeddings.
|
|
3. **Dead backend** — the probe says down, so the batch gives up *without* bisecting (splitting
|
|
against a dead backend would burn 2n-1 requests and delay the breaker exactly when it needs
|
|
to trip).
|
|
|
|
Tolerating a *flaky* backend is right; surviving a *dead* one is not. The archive is parsed
|
|
single-threaded ahead of the GPU, so on a full-corpus run (Etap B) a dead Ollama would let the
|
|
job chew through 200k+ mails at parse speed, mark every chunk `chunks_errors`, and throw away a
|
|
multi-hour pass. `--max-embed-failures` (default 5, `0` disables) therefore stops the run after
|
|
that many *consecutive* give-ups, with exit code 2; any successful batch resets the counter.
|
|
Ollama@SOLARIA's known failure mode is total (container vanishes, network-detached — 4
|
|
incidents, plan §1.4/§7), so the breaker trips within seconds of it. On abort, rows already
|
|
embedded in the dying batch are committed and pending `entities[type=threading]` appends are
|
|
flushed first: they don't depend on Ollama, they're idempotent, and re-deriving them would mean
|
|
re-reading the same 27 GB.
|
|
|
|
A partial failure against a *live* backend deliberately does **not** advance the breaker: those
|
|
chunks were never inserted, so the next run retries them through the ordinary idempotency path,
|
|
and letting a handful of bad chunks abort a 50k slice would be strictly worse than skipping
|
|
them. `embed_items_failed` and the `embed_requests_total`/`embed_calls` ratio in the `summary`
|
|
line are what to read — a ratio of 1.0 means a clean run with no retries or bisection.
|
|
|
|
Timeouts are part of this, not an exception to it: aiohttp raises a bare `builtins.TimeoutError`
|
|
when `ClientTimeout(total=...)` expires, and that is **not** a subclass of
|
|
`aiohttp.ClientError`. An earlier version of this job caught `ClientError` alone, so an Ollama
|
|
that accepted the connection and then hung — its actual failure mode — crashed the run outright,
|
|
with no breaker and no threading flush. `kb_retrieval.embed.TRANSIENT_EMBED_ERRORS` now names
|
|
both; catch that tuple, never `ClientError` on its own.
|
|
|
|
Only a wrong embedding dimension is more severe (`EmbeddingDimensionError`, exit 1) — it aborts
|
|
immediately, since that would otherwise silently index a vector that doesn't match
|
|
`document_chunk.embedding VECTOR(1024)`.
|
|
|
|
## No SOLARIA -> PIHA fallback here (deliberate)
|
|
|
|
`kb-query` fails a *query* over to Ollama@PIHA when SOLARIA is down (`app/embed_router.py`).
|
|
This job does not, by decision: ~271k active chunks at PIHA's ~790 ms/embed CPU is ~60 h on an
|
|
8 GB node already shared with Home Assistant, Paperless and kb-postgres, and CPU embedding does
|
|
not batch-scale the way the GPU does. A fallback would quietly turn "abort and resume the slice
|
|
once Ollama is fixed" into a two-day run suffocating the infra node. The correct answer to a
|
|
dead primary in the backfill path is to stop and resume — which costs nothing, because the job
|
|
is idempotent. The two backend paths stay separate on purpose.
|
|
|
|
## Batch-size benchmark
|
|
|
|
`mail-body-ingest-bench` (same package) sweeps batch sizes over real mail chunks and prints
|
|
ms/chunk, chunks/s and a projected wall-clock for the full corpus. Read-only — `SELECT`s and
|
|
inference, with no code path that can write — so it is safe to point at live kb-postgres@PIHA.
|
|
|
|
```bash
|
|
mail-body-ingest-bench --dsn postgresql://kb:<pw>@piha:5433/kb \
|
|
--archive-root /home/oskar/kb/mail/archive --sample-envelopes 200
|
|
```
|
|
|
|
It warms the model up before measuring (the first call after an idle period pays the model
|
|
load) and measures the *same* chunk set at every size, since ms/chunk depends heavily on text
|
|
length. `--max-chunks` caps the set — batch=1 otherwise dominates the wall clock.
|
|
|
|
## Idempotency
|
|
|
|
Pre-fetched `(envelope_id, chunk_index)` pairs, scoped server-side to `--model` (`WHERE model
|
|
= $1`), skip chunks already inserted — the pair doesn't need `model` redundantly since the
|
|
fetch is already scoped to it. Built this way from the start per the plan's note that
|
|
`chunk_embed.py`'s otherwise-equivalent pre-fetch is easy to mis-key across runs that mix
|
|
models (not an active bug there today, since one run always uses one model, but worth not
|
|
repeating the ambiguity here).
|
|
|
|
## Definition of Done
|
|
|
|
Per `CLAUDE.md`: `pytest` passes (62/62 for the job, 22/22 for `kb-retrieval`'s embed client)
|
|
+ a `--limit 5` dry-run smoke against live
|
|
`kb-postgres@PIHA` before committing (confirms DSN/query wiring; a missing local archive
|
|
mirror correctly reports `missing_file` rather than crashing). The Etap A pilot (`--since
|
|
2025-07-01 --apply`, plan §7) is a separate, explicitly-confirmed run — not part of this
|
|
job's DoD, since it's the first real write against production data.
|