---
okf: "0.1"
type: runbook
visibility: private
status: active
updated: 2026-08-05
links:
  - ../services/job-mail-body-ingest.md
---

# mail-body-ingest — uruchomienie i testy

## Usage

```bash
# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero
# DB writes (including the threading UPDATE):
mail-body-ingest --dsn postgresql://kb:<pw>@piha:5433/kb --archive-root /home/oskar/kb/mail/archive

# Etap A pilot — last 12 months only (plan Decyzja 9):
mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1

# Etap B — full archive in 50k slices (plan §9; ORDER BY id is stable, so slices are
# reproducible, and idempotency covers their boundaries):
nice -n 10 ionice -c2 -n7 mail-body-ingest --dsn ... --apply \
    --limit 50000 --offset 0 > mail-ingest-etapB-0.log 2>&1

# Smoke-test slice:
mail-body-ingest --dsn ... --apply --limit 10
```

DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default
`http://localhost:11434` — this job is meant to run where Ollama lives).

## Embed tuning

| Flag | Env | Default | Notes |
|---|---|---|---|
| `--batch-size` | `MAIL_INGEST_BATCH_SIZE` | 64 | `/api/embed` batch. 64 measured best across the corpus's length mix; short-text-heavy slices favour 128 (plan §1.4) |
| `--embed-retries` | `MAIL_INGEST_EMBED_RETRIES` | 2 | Retries per batch on transport errors; `0` disables |
| `--embed-backoff` | `MAIL_INGEST_EMBED_BACKOFF` | 1.0 s | Base backoff, doubled each attempt |
| `--embed-timeout` | `MAIL_INGEST_EMBED_TIMEOUT` | 120 s | Per-request hard timeout — scale it with `--batch-size` |
| `--max-embed-failures` | — | 5 | Consecutive give-ups before aborting with exit 2; `0` disables |

A malformed env value is a startup failure, not a silent fallback — a typo'd
`MAIL_INGEST_BATCH_SIZE` must not quietly produce a multi-hour run at the wrong batch size.

## Batch-size benchmark

Read-only (`SELECT`s + inference, no write path at all), so it is safe against the live DB:

```bash
mail-body-ingest-bench --dsn postgresql://kb:<pw>@piha:5433/kb \
    --archive-root /home/oskar/kb/mail/archive --sample-envelopes 200

# Wider sweep, capped so batch=1 doesn't dominate the wall clock:
mail-body-ingest-bench --dsn ... --sizes 1,8,32,64,128 --max-chunks 300
```

Prints ms/chunk, chunks/s and a projected full-corpus wall clock per batch size. Re-run it
after an Ollama upgrade or on a different GPU before trusting the default `--batch-size`.

## Tests

```bash
pip install -e "jobs/mail-body-ingest[dev]"
cd jobs/mail-body-ingest && pytest
```

Pure unit tests (62), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect`
and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers:
quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/
gmail_quote skipping), newsletter classification, threading extraction, prefix building,
body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse
fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks
never reaching Ollama, dimension-mismatch abort, and the circuit breaker (trips on N
consecutive give-ups, resets on a success, disabled by `0`, flushes pending threading on
abort). Batch-specific: a poison chunk isolated out of a full batch, a partial failure not
advancing the breaker, and an Ollama hang degrading to the breaker instead of crashing the run.

The batch client itself is tested in `packages/kb-retrieval/tests/test_embed.py` (22) — retry
with backoff, bisection, dead-backend give-up bounds, and the `TimeoutError`-is-not-a-
`ClientError` regression. The benchmark's pure logic (batch splitting, derived metrics, table
formatting, failure counting) is in `jobs/mail-body-ingest/tests/test_benchmark.py`.

In a worktree without a preinstalled venv:

```bash
python3 -m venv /tmp/kbvenv && /tmp/kbvenv/bin/pip install -q pytest pytest-asyncio \
    -e packages/kb-mail/ -e packages/kb-retrieval/ -e jobs/mail-body-ingest/
/tmp/kbvenv/bin/python -m pytest jobs/mail-body-ingest/tests/ packages/kb-retrieval/tests/ -q
```