homelab-codex-ws/jobs/gmail-header-backfill/pyproject.toml

23 lines
436 B
TOML
Raw Normal View History

feat(gmail-header-backfill): one-shot job — backfill headers into envelope.entities Module 5 phase 2 (docs/kb/modules/05-faza2-plan.md, §5). 225,030 existing source='gmail' envelope rows carry only an attachment manifest — no from/to/cc/delivered_to/subject anywhere in the DB, which blocks answering "is this on me or my wife" / distinguishing aliases (plan §1.8). Separate job from gmail-bulk-import per plan §2 decision 7: UPDATE semantics against production data is a different risk profile than the historical INSERT job. Parses headers only (no MIME-walk of attachments) from the archived .eml files, appends {"type": "headers", ...} (plan §4.1) via the idempotent UPDATE ... WHERE NOT EXISTS from §5.2, batched via executemany. --limit/ --offset (ORDER BY id) give stable, deterministic partitioning so the full backfill can run and be verified in slices instead of one unattended pass. Plain CLI (pip install -e), no Dockerfile — same convention as gmail-bulk-import/documents-ingest, which run directly on PIHA for local filesystem access to the .eml archive. Verified against kb-postgres@PIHA (100-row dry-run + apply, re-run proved idempotent no-op, 1000-row timed slice): 1096/225030 rows backfilled, 996/1000 succeeded on the timed slice (4 parse_errors — a 2014 spam message with an RFC 2047 encoded-word decoding to an embedded newline in the From display name, correctly caught and skipped rather than crashing the batch). Extrapolated full-run time ~16 minutes. Full 225,030-row run is out of scope for this change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-14 15:34:07 +02:00
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "gmail-header-backfill"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
"asyncpg>=0.29",
"structlog>=24.1",
]
[project.scripts]
gmail-header-backfill = "gmail_header_backfill.backfill:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]