feat(mail-body-ingest): new job to chunk+embed gmail body content (faza mailowa Krok 2)
Second full pass over the gmail .eml archive (gmail-bulk-import's first pass
skipped inline text/plain and text/html on purpose). Per envelope: typed
parse with compat32 fallback -> body extraction (inline text/plain preferred,
HTML->text via a small stdlib HTMLParser otherwise) -> quote-strip (reply
markers + `>`-quoted lines, EN/PL/Outlook patterns) -> newsletter
classification (List-Unsubscribe/List-Id/Precedence, chunked but not
embedded, excluded_reason='newsletter') -> Temat/Od/Data prefix from the
already-backfilled entities[type=headers] (zero header re-parse) -> chunk via
kb_mail.chunking -> batched embed_batch (64) -> INSERT document_chunk.
In-Reply-To/References are appended as entities[type=threading] during the
same read (idempotent WHERE NOT EXISTS append, 1:1 with
gmail-header-backfill) -- the only DB writes are document_chunk INSERTs and
an additive envelope.entities UPDATE; the .eml archive stays read-only.
--dsn/KB_DSN, --archive-root, --since/--limit/--offset, --batch-size,
--apply (dry-run default). Idempotency keys on (envelope_id, chunk_index)
pre-fetched scoped to --model, built correctly from the start per the
plan's flagged chunk_embed.py precedent. A failed embed batch is isolated
(chunks_errors, no abort) for Ollama's documented instability; a wrong
embedding dimension aborts the whole run.
48 tests, DoD smoke run against live kb-postgres@PIHA confirmed wiring
(archive not yet rsync'd to SOLARIA, so all 5 rows correctly reported
missing_file). docs/kb/modules/05-faza-mailowa-plan.md, §5.
2026-07-22 19:01:14 +02:00
|
|
|
[build-system]
|
|
|
|
|
requires = ["setuptools>=68"]
|
|
|
|
|
build-backend = "setuptools.build_meta"
|
|
|
|
|
|
|
|
|
|
[project]
|
|
|
|
|
name = "mail-body-ingest"
|
|
|
|
|
version = "0.1.0"
|
|
|
|
|
requires-python = ">=3.11"
|
|
|
|
|
dependencies = [
|
|
|
|
|
"asyncpg>=0.29",
|
|
|
|
|
"aiohttp>=3.9",
|
|
|
|
|
"structlog>=24.1",
|
|
|
|
|
"kb-mail",
|
|
|
|
|
"kb-retrieval",
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
[project.optional-dependencies]
|
|
|
|
|
dev = [
|
|
|
|
|
"pytest>=8.1",
|
|
|
|
|
"pytest-asyncio>=0.23",
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
[project.scripts]
|
|
|
|
|
mail-body-ingest = "mail_body_ingest.ingest:main"
|
feat(kb-mail-batching): retry + izolacja trujacego chunka w torze embed + benchmark
Batching /api/embed juz istnial (Krok 1 fazy mailowej, batch 64). Recon przed
Etapem B wykazal w torze backfillu blad blokujacy i dwie luki.
BUG (blokujacy dla Etapu B): flush_embed_buffer lapal wylacznie
aiohttp.ClientError, a wyczerpanie ClientTimeout(total=...) rzuca goly
builtins.TimeoutError, ktory NIE jest jego podklasa (zweryfikowane empirycznie
na aiohttp 3.14.3). Zawieszona Ollama — czyli jej udokumentowany failure mode,
"przyjmuje polaczenie i milczy" — wywalala caly run nieobsluzonym wyjatkiem,
bez breakera i bez flushu threadingu. Na plastrze 50k = utrata zarobionej pracy.
Klasy przejsciowe nazwane teraz jawnie w TRANSIENT_EMBED_ERRORS.
kb-retrieval:
- embed_batch(timeout_s=...) — bound per zadanie, skalowalny z batch size
- embed_batch_resilient() — retry z backoffem wykladniczym, a po ich wyczerpaniu
probe /api/tags rozstrzyga: backend zywy -> bisekcja izolujaca trujacy chunk
(jeden zly tekst kosztowal caly batch 64, bo /api/embed jest all-or-nothing);
backend martwy -> natychmiastowe gave_up bez bisekcji, ktora spalilaby 2n-1
zadan i opoznila breaker. EmbeddingDimensionError nigdy nie jest retry'owane.
- failed_indices wyprowadzane z wyniku, nie akumulowane per span — przy gave_up
w srodku bisekcji porzucone poddrzewo nigdy nie dochodzi do liscia.
mail-body-ingest:
- breaker liczy give-upy (backend padl), nie dowolne nieudane batche; porazka
czesciowa przy zywym backendzie nie przesuwa licznika, bo te chunki i tak
zlapie kolejny run przez idempotencje
- wiersze zembedowane w umierajacym batchu sa commitowane przed abortem
- parametryzacja: --batch-size/--embed-retries/--embed-backoff/--embed-timeout,
kazdy z odpowiednikiem env MAIL_INGEST_*; bledna wartosc env = glosny SystemExit
- metryka embed_ms_per_chunk (porownywalna miedzy runami, w odroznieniu od
sredniej per batch) + embed_requests_total/embed_calls jako sygnal zdrowia
mail-body-ingest-bench: nowy entry point, sweep batch size na realnych chunkach.
Read-only (SELECT + inferencja, zero sciezki zapisu), warmup przed pomiarem, ten
sam zbior chunkow dla kazdego rozmiaru. Czyni liczby z planu §1.4 odtwarzalnymi.
Fallback SOLARIA->PIHA dla backfillu SWIADOMIE nie powstaje (potwierdzone przez
operatora): 271k chunkow x 790 ms CPU ~ 60 h na 8 GB PIHA dzielonym z HA i
Paperlessem. Wlasciwa odpowiedzia na martwy backend jest exit 2 i wznowienie
plastra. Tor online (kb-query -> embed_router) zachowuje fallback — rozdzial
torow udokumentowany w docstringu embed.py i w kb/services/.
Testy: 117 zielonych (62 job + 22 klient embed + reszta pakietow), w tym
regresja na TimeoutError, bisekcja, ograniczony koszt przy martwym backendzie
i porazka czesciowa nieprzesuwajaca breakera. Bez uruchamiania backfillu.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 12:02:41 +02:00
|
|
|
mail-body-ingest-bench = "mail_body_ingest.benchmark:main"
|
feat(mail-body-ingest): new job to chunk+embed gmail body content (faza mailowa Krok 2)
Second full pass over the gmail .eml archive (gmail-bulk-import's first pass
skipped inline text/plain and text/html on purpose). Per envelope: typed
parse with compat32 fallback -> body extraction (inline text/plain preferred,
HTML->text via a small stdlib HTMLParser otherwise) -> quote-strip (reply
markers + `>`-quoted lines, EN/PL/Outlook patterns) -> newsletter
classification (List-Unsubscribe/List-Id/Precedence, chunked but not
embedded, excluded_reason='newsletter') -> Temat/Od/Data prefix from the
already-backfilled entities[type=headers] (zero header re-parse) -> chunk via
kb_mail.chunking -> batched embed_batch (64) -> INSERT document_chunk.
In-Reply-To/References are appended as entities[type=threading] during the
same read (idempotent WHERE NOT EXISTS append, 1:1 with
gmail-header-backfill) -- the only DB writes are document_chunk INSERTs and
an additive envelope.entities UPDATE; the .eml archive stays read-only.
--dsn/KB_DSN, --archive-root, --since/--limit/--offset, --batch-size,
--apply (dry-run default). Idempotency keys on (envelope_id, chunk_index)
pre-fetched scoped to --model, built correctly from the start per the
plan's flagged chunk_embed.py precedent. A failed embed batch is isolated
(chunks_errors, no abort) for Ollama's documented instability; a wrong
embedding dimension aborts the whole run.
48 tests, DoD smoke run against live kb-postgres@PIHA confirmed wiring
(archive not yet rsync'd to SOLARIA, so all 5 rows correctly reported
missing_file). docs/kb/modules/05-faza-mailowa-plan.md, §5.
2026-07-22 19:01:14 +02:00
|
|
|
|
|
|
|
|
[tool.setuptools.packages.find]
|
|
|
|
|
where = ["src"]
|
|
|
|
|
|
|
|
|
|
[tool.pytest.ini_options]
|
|
|
|
|
asyncio_mode = "auto"
|
|
|
|
|
testpaths = ["tests"]
|