homelab-codex-ws/jobs/gmail-header-backfill

gmail-header-backfill

One-shot job, module 5 phase 2 backfill (docs/kb/modules/05-faza2-plan.md, §5). Backfills {"type": "headers", ...} onto the 225 030 existing source='gmail' envelope rows in kb-postgres, which today carry only an attachment manifest ({"type": "attachment", ...}) — no from/to/cc/delivered_to/subject anywhere in the DB (plan §1.8). This job does not touch document_chunk or any non-entities data, and never rewrites existing entities elements — it only appends one new element per envelope.

Why a separate job, not an extension of gmail-bulk-import

gmail-bulk-import has INSERT semantics (new envelopes from mbox/Takeout), already run once, historical. This job has UPDATE semantics against the 225 030 rows already in production — different risk profile, different lifecycle. See plan §2 decision 7.

Where it runs

Locally on PIHA, as a plain CLI (not a container) — same reasoning as documents-ingest: it needs simultaneous local filesystem access to the mail archive (/home/oskar/kb/mail/archive) and a DB connection to kb-postgres.

Install (from repo root, on PIHA):

pip install -e jobs/gmail-header-backfill/

Usage

# Dry run (default) — parse and count only, no DB writes:
gmail-header-backfill --dsn postgresql://kb:<pw>@localhost:5433/kb --limit 100

# Real run — apply the UPDATE for this slice:
gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply

# Next slice — offset is stable/deterministic (ORDER BY id), independent of
# how many rows in earlier slices were already backfilled:
gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply

DSN can also come from the KB_DSN env var instead of --dsn.

--limit/--offset exist so the full 225 030-row backfill can be run in verifiable partitions instead of one long unattended run (plan §5.3) — start small (--limit 100), check the result in the DB, then widen.

Source of headers: the archived .eml, not the original mbox

Same access pattern as documents-ingest: archive_root / row["raw_ref"] -> read_bytes() -> email.message_from_bytes(raw, policy=email.policy.default). The .eml archive is the immutable raw layer (kb-00 rule #1) and carries the full original headers — no need to go back to the source Takeout mbox.

Only headers are parsed — the job does not MIME-walk into attachment parts (plan §5.1); that keeps per-message cost low relative to gmail-bulk-import's full import (which does walk attachments).

entities[type=headers] shape (plan §4.1)

{
  "type": "headers",
  "from":         {"name": "WARTA", "address": "no-reply@warta.pl"},
  "to":           [{"name": "...", "address": "oskar@gmail.com"}],
  "cc":           [],
  "delivered_to": ["oskar+alias@gmail.com", "oskar@gmail.com"],
  "subject":      "Twoja polisa OC/AC",
  "date_raw":     "Mon, 9 Jun 2026 12:34:56 +0200"
}

• from/to/cc are parsed to {name, address} via email.policy.default (RFC 2047 encoded-word decoding) + email.utils.getaddresses on the decoded text. to/cc are lists (comma-separated multi-address headers are split); from is a single object or null — if a message carries more than one From: header (malformed but seen in the wild), the first is used and a headers.multiple_from warning is logged.
• delivered_to is a list of raw, unparsed strings — Delivered-To can repeat per hop, and every occurrence is kept (this is what identifies which of Oskar's aliases received the message; motivation in plan §1.8).
• date_raw is the literal original Date: header text, taken from a separate email.policy.compat32 parse — policy.default's structured DateHeader reformats the value (corrects the weekday name, zero-pads the day) rather than preserving what was actually in the file, and date_raw exists specifically for byte-for-byte comparison/debug against envelope.ts (already parsed at import time — this is not a duplicate source of truth).
• Malformed/undecodable headers never raise — they degrade to best-effort text or are skipped, logged, and counted; the row is left for a future run rather than half-updated.

Fallback parse (parse_headers_fallback)

The full 225 030-row run (2026-07) left 9 envelopes whose headers the typed policy.default parse rejects outright (diagnosis 2026-07-14):

• 7× ValueError: address parts cannot contain CR or LF — an RFC 2047 encoded-word decoding to text with a newline (=0A) inside a display name, e.g. Rekrutacja z =?utf-8?Q?ExampleCorp=0A?= <mailing@example.pl>
• 1× AttributeError: 'Group' object has no attribute 'local_part' — RFC 5322 group syntax: To: unlisted-recipients:; (no To-header on input)
• 1× AttributeError: 'str' object has no attribute 'token_type' — a malformed display name (CPython parser bug, fixed in newer versions but present on PIHA's 3.11)

When the typed parse raises, the job retries with parse_headers_fallback: a pure policy.compat32 parse where address fields are split into {name, address} by email.utils.getaddresses over the raw header text (no RFC 2047 decoding — names may keep literal =?...?= encoded-words), and subject/delivered_to/date_raw stay raw strings. The written entity has exactly the same §4.1 shape — only parse quality degrades, never the schema. String values are sanitized so no lone surrogates (which postgres jsonb rejects) can reach the DB.

Fallback successes are counted separately as parsed_fallback (a labeled subset of updated) and logged per row (headers.parsed_fallback with the original typed error) — never silently mixed into ordinary successes. Only when the fallback also fails does the row count as parse_errors.

Stats must balance — no silent skips

Every scanned row lands in exactly one bucket, and run_complete / summary must satisfy:

scanned = updated + already_has_headers + parse_errors + read_errors
          + missing_file

A missing .eml is its own counter (missing_file) with a per-row skip.missing_file info log carrying the envelope id and the expected path — a full run can no longer lose rows without a trace. Any non-zero parse_errors/read_errors/missing_file makes the CLI exit 1.

Idempotency and resumability (plan §5.2)

UPDATE envelope
SET entities = entities || $2::jsonb
WHERE id = $1
  AND NOT EXISTS (
      SELECT 1 FROM jsonb_array_elements(entities) e WHERE e->>'type' = 'headers'
  );

Rows that already carry a headers entity are skipped (checked client-side before building the batch, and enforced again at the SQL level as defense-in-depth). Re-running any slice — including after a crash mid-batch — is always safe: already-backfilled rows are no-ops, not double-appended.

Writes are batched: rows are queued in memory and flushed via conn.executemany every 500 rows (one round-trip per batch, not one transaction per row) — mirrors gmail-bulk-import's _insert_batch pattern.

--limit/--offset partition by ORDER BY id, not by "rows still missing headers" — this keeps a given --offset naming the same slice of the table across repeated runs, so progress is easy to reason about (e.g. "slices 0, 1000, 2000, ... cover the whole table") independent of how much of it is already done.

Performance estimate (not measured — job didn't exist before this change)

Per plan §5.3: ~225 030 files, ~124 KB average. Each row costs an open + read + header-only parse (no MIME-walk of attachments) + a batched UPDATE. The full gmail-bulk-import run (parsing the whole mbox, including attachment MIME-walk and inserts) took ~29 minutes. This job does less per-message work but pays for opening 225k small files individually instead of streaming one mbox — the plan's estimate is "same order of magnitude, likely tens of minutes." Not measured directly in this change — see the DoD note below.

Tests

pip install -e jobs/gmail-header-backfill/
cd jobs/gmail-header-backfill && pytest

Pure unit tests, no DB or filesystem outside tmp_path/synthetic .eml bytes — run() is tested by monkeypatching asyncpg.connect with an in-memory fake connection. Covers: header parsing (multi-address To/Cc, quoted display names with commas, multiple Delivered-To occurrences, RFC 2047 encoded-words including Polish diacritics, malformed encoded-words that must not raise, missing/multiple From, date_raw preserving literal text vs. Date header reformatting), idempotency (rows already carrying a headers entity are skipped and never re-appended), batch flushing, and --limit/--offset query shape.

Definition of Done

Per CLAUDE.md: this job's smoke run is gmail-header-backfill --dsn ... --limit 100 (dry-run first, then --apply against a small slice) — run against kb-postgres@PIHA over SSH, not executed as part of this change without operator confirmation (UPDATE against production data). pytest passes locally (33/33) before this commit; full 225 030-row backfill is out of scope for this change — see the plan for the rollout sequence.