6b85c7ef68
Wzorzec mechaniczny: sekcje deploy/verify/install/testy wycinane do kb/runbooks/<serwis>-*.md, reszta zostaje dokumentem type: service. Wzajemne `links` w obie strony. Tresc sekcji nietknieta — przenoszone doslownie, dodany wylacznie naglowek H1 nowego runbooka. kb-query, paperless-worker, planner-agent, ha-diag-agent, ollama-piha, narty27, home-assistant, ha-mcp, job-gmail-header-backfill, job-mail-body-ingest. Weryfikacja: dla kazdego pliku multizbior niepustych linii (main + runbook) == oryginal z HEAD. Zero zgubionych, zero dodanych. Recon szacowal 13 splitow service+runbook; faktycznie 2-typowych jest 10, pozostale 5 (paperless, nextcloud, gokapi, fleet-prometheus, deploy-runner) sa 3-typowe i ida osobno jako splity wielotypowe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
52 lines
1.9 KiB
Markdown
52 lines
1.9 KiB
Markdown
---
|
|
okf: "0.1"
|
|
type: runbook
|
|
visibility: private
|
|
status: active
|
|
updated: 2026-07-22
|
|
links:
|
|
- ../services/job-mail-body-ingest.md
|
|
---
|
|
|
|
# mail-body-ingest — uruchomienie i testy
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Dry run (default) — parse, quote-strip, classify, chunk, count. Zero Ollama calls, zero
|
|
# DB writes (including the threading UPDATE):
|
|
mail-body-ingest --dsn postgresql://kb:<pw>@piha:5433/kb --archive-root /home/oskar/kb/mail/archive
|
|
|
|
# Etap A pilot — last 12 months only (plan Decyzja 9):
|
|
mail-body-ingest --dsn ... --since 2025-07-01 --apply > mail-ingest-etapA.log 2>&1
|
|
|
|
# Etap B — full archive in 50k slices (plan §9; ORDER BY id is stable, so slices are
|
|
# reproducible, and idempotency covers their boundaries):
|
|
nice -n 10 ionice -c2 -n7 mail-body-ingest --dsn ... --apply \
|
|
--limit 50000 --offset 0 > mail-ingest-etapB-0.log 2>&1
|
|
|
|
# Smoke-test slice:
|
|
mail-body-ingest --dsn ... --apply --limit 10
|
|
```
|
|
|
|
DSN can also come from `KB_DSN`, Ollama URL from `OLLAMA_URL` (default
|
|
`http://localhost:11434` — this job is meant to run where Ollama lives).
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pip install -e "jobs/mail-body-ingest[dev]"
|
|
cd jobs/mail-body-ingest && pytest
|
|
```
|
|
|
|
Pure unit tests (55), no DB/Ollama — `run()` is tested by monkeypatching `asyncpg.connect`
|
|
and `aiohttp.ClientSession` with in-memory fakes, `.eml` bytes written to `tmp_path`. Covers:
|
|
quote-strip (EN/PL/Outlook markers, bare `>` lines), HTML->text (style/script/blockquote/
|
|
gmail_quote skipping), newsletter classification, threading extraction, prefix building,
|
|
body extraction (plain-preferred, HTML fallback, attachment-only), the typed/compat32 parse
|
|
fallback, stats balance, idempotency (second run inserts nothing new), newsletter chunks
|
|
never reaching Ollama, Ollama-offline batch isolation, dimension-mismatch abort, and the
|
|
circuit breaker (trips on N consecutive failures, resets on a success, disabled by `0`,
|
|
flushes pending threading on abort).
|
|
|