homelab-codex-ws/jobs/gmail-header-backfill/README.md

# gmail-header-backfill

One-shot job, module 5 phase 2 backfill (`docs/kb/modules/05-faza2-plan.md`, §5).
Backfills `{"type": "headers", ...}` onto the 225 030 existing `source='gmail'`
envelope rows in kb-postgres, which today carry only an attachment manifest
(`{"type": "attachment", ...}`) — no `from`/`to`/`cc`/`delivered_to`/`subject`
anywhere in the DB (plan §1.8). This job does **not** touch `document_chunk` or
any non-`entities` data, and never rewrites existing `entities` elements — it
only appends one new element per envelope.

## Why a separate job, not an extension of gmail-bulk-import

`gmail-bulk-import` has INSERT semantics (new envelopes from mbox/Takeout),
already run once, historical. This job has UPDATE semantics against the
225 030 rows already in production — different risk profile, different
lifecycle. See plan §2 decision 7.

## Where it runs

**Locally on PIHA**, as a plain CLI (not a container) — same reasoning as
`documents-ingest`: it needs simultaneous local filesystem access to the mail
archive (`/home/oskar/kb/mail/archive`) and a DB connection to kb-postgres.

Install (from repo root, on PIHA):

```bash
pip install -e jobs/gmail-header-backfill/
```

## Usage

```bash
# Dry run (default) — parse and count only, no DB writes:
gmail-header-backfill --dsn postgresql://kb:<pw>@localhost:5433/kb --limit 100

# Real run — apply the UPDATE for this slice:
gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply

# Next slice — offset is stable/deterministic (ORDER BY id), independent of
# how many rows in earlier slices were already backfilled:
gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply
```

DSN can also come from the `KB_DSN` env var instead of `--dsn`.

`--limit`/`--offset` exist so the full 225 030-row backfill can be run in
verifiable partitions instead of one long unattended run (plan §5.3) — start
small (`--limit 100`), check the result in the DB, then widen.

## Source of headers: the archived `.eml`, not the original mbox

Same access pattern as `documents-ingest`: `archive_root / row["raw_ref"]` ->
`read_bytes()` -> `email.message_from_bytes(raw, policy=email.policy.default)`.
The `.eml` archive is the immutable raw layer (kb-00 rule #1) and carries the
full original headers — no need to go back to the source Takeout mbox.

Only headers are parsed — the job does **not** MIME-walk into attachment
parts (plan §5.1); that keeps per-message cost low relative to
`gmail-bulk-import`'s full import (which does walk attachments).

## `entities[type=headers]` shape (plan §4.1)

```json
{
  "type": "headers",
  "from":         {"name": "WARTA", "address": "no-reply@warta.pl"},
  "to":           [{"name": "...", "address": "oskar@gmail.com"}],
  "cc":           [],
  "delivered_to": ["oskar+alias@gmail.com", "oskar@gmail.com"],
  "subject":      "Twoja polisa OC/AC",
  "date_raw":     "Mon, 9 Jun 2026 12:34:56 +0200"
}
```

- `from`/`to`/`cc` are parsed to `{name, address}` via `email.policy.default`
  (RFC 2047 encoded-word decoding) + `email.utils.getaddresses` on the
  decoded text. `to`/`cc` are lists (comma-separated multi-address headers
  are split); `from` is a single object or `null` — if a message carries more
  than one `From:` header (malformed but seen in the wild), the first is used
  and a `headers.multiple_from` warning is logged.
- `delivered_to` is a list of **raw, unparsed strings** — `Delivered-To` can
  repeat per hop, and every occurrence is kept (this is what identifies which
  of Oskar's aliases received the message; motivation in plan §1.8).
- `date_raw` is the literal original `Date:` header text, taken from a
  separate `email.policy.compat32` parse — `policy.default`'s structured
  `DateHeader` reformats the value (corrects the weekday name, zero-pads the
  day) rather than preserving what was actually in the file, and `date_raw`
  exists specifically for byte-for-byte comparison/debug against `envelope.ts`
  (already parsed at import time — this is not a duplicate source of truth).
- Malformed/undecodable headers never raise — they degrade to best-effort
  text or are skipped, logged, and counted; the row is left for a future run
  rather than half-updated.

## Idempotency and resumability (plan §5.2)

```sql
UPDATE envelope
SET entities = entities || $2::jsonb
WHERE id = $1
  AND NOT EXISTS (
      SELECT 1 FROM jsonb_array_elements(entities) e WHERE e->>'type' = 'headers'
  );
```

Rows that already carry a `headers` entity are skipped (checked client-side
before building the batch, and enforced again at the SQL level as
defense-in-depth). Re-running any slice — including after a crash mid-batch —
is always safe: already-backfilled rows are no-ops, not double-appended.

Writes are batched: rows are queued in memory and flushed via
`conn.executemany` every 500 rows (one round-trip per batch, not one
transaction per row) — mirrors `gmail-bulk-import`'s `_insert_batch` pattern.

`--limit`/`--offset` partition by `ORDER BY id`, not by "rows still missing
headers" — this keeps a given `--offset` naming the same slice of the table
across repeated runs, so progress is easy to reason about (e.g. "slices 0,
1000, 2000, ... cover the whole table") independent of how much of it is
already done.

## Performance estimate (not measured — job didn't exist before this change)

Per plan §5.3: ~225 030 files, ~124 KB average. Each row costs an `open` +
`read` + header-only parse (no MIME-walk of attachments) + a batched UPDATE.
The full `gmail-bulk-import` run (parsing the whole mbox, including
attachment MIME-walk and inserts) took ~29 minutes. This job does less
per-message work but pays for opening 225k small files individually instead
of streaming one mbox — the plan's estimate is "same order of magnitude,
likely tens of minutes." Not measured directly in this change — see the
DoD note below.

## Tests

```bash
pip install -e jobs/gmail-header-backfill/
cd jobs/gmail-header-backfill && pytest
```

Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml`
bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an
in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`,
quoted display names with commas, multiple `Delivered-To` occurrences, RFC
2047 encoded-words including Polish diacritics, malformed encoded-words that
must not raise, missing/multiple `From`, `date_raw` preserving literal text
vs. `Date` header reformatting), idempotency (rows already carrying a
`headers` entity are skipped and never re-appended), batch flushing, and
`--limit`/`--offset` query shape.

## Definition of Done

Per `CLAUDE.md`: this job's smoke run is `gmail-header-backfill --dsn ...
--limit 100` (dry-run first, then `--apply` against a small slice) — run
against kb-postgres@PIHA over SSH, **not executed as part of this change**
without operator confirmation (UPDATE against production data). `pytest`
passes locally (33/33) before this commit; full 225 030-row backfill is out
of scope for this change — see the plan for the rollout sequence.