f14ad410a0
Diagnosis of the 5008 header-less envelopes after the full 225 030-row run
(read-only, on PIHA, 2026-07-14):
- 4999 = one contiguous block at ORDER BY id positions 70001-74999: the
--offset 70000 slice died mid-run. Root cause reproduced: a Date: header
with raw 8-bit bytes makes compat32 .get() return email.header.Header
(not str), and json.dumps([headers]) — which sat OUTSIDE the per-row
try/except — raised TypeError and killed the process, silently losing
the rest of the slice.
- 9 = genuine typed-parse failures: 7x RFC 2047 encoded-word decoding to
CR/LF inside a display name (ValueError in headerregistry), 1x RFC 5322
group syntax in To: ("unlisted-recipients:;"), 1x CPython
_header_value_parser bug on a malformed display name (fixed upstream,
present on PIHA's 3.11).
- 0 missing .eml files.
Fixes:
- date_raw: str() + surrogate sanitization on the compat32 value — the
crash cause, now also covered by a regression test.
- json.dumps moved inside the per-row try: a non-serializable value counts
as that row's parse_error instead of crashing the slice.
- parse_headers_fallback(): on typed-parse failure retry with a pure
compat32 parse — getaddresses over raw header text, raw-string values,
same §4.1 entity shape. Counted separately as parsed_fallback (labeled
subset of updated), logged per row with the original typed error.
- missing_file counter + skip.missing_file info log (id, expected path);
run_complete now balances: scanned = updated + already_has_headers +
parse_errors + read_errors + missing_file. Non-zero missing_file also
fails the exit code.
Verified: 43/43 pytest locally (3.13) and on PIHA (3.11); read-only dry
runs on PIHA — all 9 parse failures recover via fallback, the lost slice
completes scanned=5000 updated=4999 already_has_headers=1 with zero errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
199 lines
9 KiB
Markdown
199 lines
9 KiB
Markdown
# gmail-header-backfill
|
||
|
||
One-shot job, module 5 phase 2 backfill (`docs/kb/modules/05-faza2-plan.md`, §5).
|
||
Backfills `{"type": "headers", ...}` onto the 225 030 existing `source='gmail'`
|
||
envelope rows in kb-postgres, which today carry only an attachment manifest
|
||
(`{"type": "attachment", ...}`) — no `from`/`to`/`cc`/`delivered_to`/`subject`
|
||
anywhere in the DB (plan §1.8). This job does **not** touch `document_chunk` or
|
||
any non-`entities` data, and never rewrites existing `entities` elements — it
|
||
only appends one new element per envelope.
|
||
|
||
## Why a separate job, not an extension of gmail-bulk-import
|
||
|
||
`gmail-bulk-import` has INSERT semantics (new envelopes from mbox/Takeout),
|
||
already run once, historical. This job has UPDATE semantics against the
|
||
225 030 rows already in production — different risk profile, different
|
||
lifecycle. See plan §2 decision 7.
|
||
|
||
## Where it runs
|
||
|
||
**Locally on PIHA**, as a plain CLI (not a container) — same reasoning as
|
||
`documents-ingest`: it needs simultaneous local filesystem access to the mail
|
||
archive (`/home/oskar/kb/mail/archive`) and a DB connection to kb-postgres.
|
||
|
||
Install (from repo root, on PIHA):
|
||
|
||
```bash
|
||
pip install -e jobs/gmail-header-backfill/
|
||
```
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# Dry run (default) — parse and count only, no DB writes:
|
||
gmail-header-backfill --dsn postgresql://kb:<pw>@localhost:5433/kb --limit 100
|
||
|
||
# Real run — apply the UPDATE for this slice:
|
||
gmail-header-backfill --dsn ... --limit 1000 --offset 0 --apply
|
||
|
||
# Next slice — offset is stable/deterministic (ORDER BY id), independent of
|
||
# how many rows in earlier slices were already backfilled:
|
||
gmail-header-backfill --dsn ... --limit 1000 --offset 1000 --apply
|
||
```
|
||
|
||
DSN can also come from the `KB_DSN` env var instead of `--dsn`.
|
||
|
||
`--limit`/`--offset` exist so the full 225 030-row backfill can be run in
|
||
verifiable partitions instead of one long unattended run (plan §5.3) — start
|
||
small (`--limit 100`), check the result in the DB, then widen.
|
||
|
||
## Source of headers: the archived `.eml`, not the original mbox
|
||
|
||
Same access pattern as `documents-ingest`: `archive_root / row["raw_ref"]` ->
|
||
`read_bytes()` -> `email.message_from_bytes(raw, policy=email.policy.default)`.
|
||
The `.eml` archive is the immutable raw layer (kb-00 rule #1) and carries the
|
||
full original headers — no need to go back to the source Takeout mbox.
|
||
|
||
Only headers are parsed — the job does **not** MIME-walk into attachment
|
||
parts (plan §5.1); that keeps per-message cost low relative to
|
||
`gmail-bulk-import`'s full import (which does walk attachments).
|
||
|
||
## `entities[type=headers]` shape (plan §4.1)
|
||
|
||
```json
|
||
{
|
||
"type": "headers",
|
||
"from": {"name": "WARTA", "address": "no-reply@warta.pl"},
|
||
"to": [{"name": "...", "address": "oskar@gmail.com"}],
|
||
"cc": [],
|
||
"delivered_to": ["oskar+alias@gmail.com", "oskar@gmail.com"],
|
||
"subject": "Twoja polisa OC/AC",
|
||
"date_raw": "Mon, 9 Jun 2026 12:34:56 +0200"
|
||
}
|
||
```
|
||
|
||
- `from`/`to`/`cc` are parsed to `{name, address}` via `email.policy.default`
|
||
(RFC 2047 encoded-word decoding) + `email.utils.getaddresses` on the
|
||
decoded text. `to`/`cc` are lists (comma-separated multi-address headers
|
||
are split); `from` is a single object or `null` — if a message carries more
|
||
than one `From:` header (malformed but seen in the wild), the first is used
|
||
and a `headers.multiple_from` warning is logged.
|
||
- `delivered_to` is a list of **raw, unparsed strings** — `Delivered-To` can
|
||
repeat per hop, and every occurrence is kept (this is what identifies which
|
||
of Oskar's aliases received the message; motivation in plan §1.8).
|
||
- `date_raw` is the literal original `Date:` header text, taken from a
|
||
separate `email.policy.compat32` parse — `policy.default`'s structured
|
||
`DateHeader` reformats the value (corrects the weekday name, zero-pads the
|
||
day) rather than preserving what was actually in the file, and `date_raw`
|
||
exists specifically for byte-for-byte comparison/debug against `envelope.ts`
|
||
(already parsed at import time — this is not a duplicate source of truth).
|
||
- Malformed/undecodable headers never raise — they degrade to best-effort
|
||
text or are skipped, logged, and counted; the row is left for a future run
|
||
rather than half-updated.
|
||
|
||
## Fallback parse (`parse_headers_fallback`)
|
||
|
||
The full 225 030-row run (2026-07) left 9 envelopes whose headers the typed
|
||
`policy.default` parse rejects outright (diagnosis 2026-07-14):
|
||
|
||
- 7× `ValueError: address parts cannot contain CR or LF` — an RFC 2047
|
||
encoded-word decoding to text with a newline (`=0A`) inside a display
|
||
name, e.g. `Rekrutacja z =?utf-8?Q?ExampleCorp=0A?= <mailing@example.pl>`
|
||
- 1× `AttributeError: 'Group' object has no attribute 'local_part'` — RFC
|
||
5322 group syntax: `To: unlisted-recipients:; (no To-header on input)`
|
||
- 1× `AttributeError: 'str' object has no attribute 'token_type'` — a
|
||
malformed display name (CPython parser bug, fixed in newer versions but
|
||
present on PIHA's 3.11)
|
||
|
||
When the typed parse raises, the job retries with `parse_headers_fallback`:
|
||
a pure `policy.compat32` parse where address fields are split into
|
||
`{name, address}` by `email.utils.getaddresses` over the **raw** header text
|
||
(no RFC 2047 decoding — names may keep literal `=?...?=` encoded-words), and
|
||
subject/`delivered_to`/`date_raw` stay raw strings. The written entity has
|
||
exactly the same §4.1 shape — only parse **quality** degrades, never the
|
||
schema. String values are sanitized so no lone surrogates (which postgres
|
||
jsonb rejects) can reach the DB.
|
||
|
||
Fallback successes are counted separately as `parsed_fallback` (a labeled
|
||
subset of `updated`) and logged per row (`headers.parsed_fallback` with the
|
||
original typed error) — never silently mixed into ordinary successes. Only
|
||
when the fallback **also** fails does the row count as `parse_errors`.
|
||
|
||
## Stats must balance — no silent skips
|
||
|
||
Every scanned row lands in exactly one bucket, and `run_complete` /
|
||
`summary` must satisfy:
|
||
|
||
```
|
||
scanned = updated + already_has_headers + parse_errors + read_errors
|
||
+ missing_file
|
||
```
|
||
|
||
A missing `.eml` is its own counter (`missing_file`) with a per-row
|
||
`skip.missing_file` info log carrying the envelope id and the expected
|
||
path — a full run can no longer lose rows without a trace. Any non-zero
|
||
`parse_errors`/`read_errors`/`missing_file` makes the CLI exit 1.
|
||
|
||
## Idempotency and resumability (plan §5.2)
|
||
|
||
```sql
|
||
UPDATE envelope
|
||
SET entities = entities || $2::jsonb
|
||
WHERE id = $1
|
||
AND NOT EXISTS (
|
||
SELECT 1 FROM jsonb_array_elements(entities) e WHERE e->>'type' = 'headers'
|
||
);
|
||
```
|
||
|
||
Rows that already carry a `headers` entity are skipped (checked client-side
|
||
before building the batch, and enforced again at the SQL level as
|
||
defense-in-depth). Re-running any slice — including after a crash mid-batch —
|
||
is always safe: already-backfilled rows are no-ops, not double-appended.
|
||
|
||
Writes are batched: rows are queued in memory and flushed via
|
||
`conn.executemany` every 500 rows (one round-trip per batch, not one
|
||
transaction per row) — mirrors `gmail-bulk-import`'s `_insert_batch` pattern.
|
||
|
||
`--limit`/`--offset` partition by `ORDER BY id`, not by "rows still missing
|
||
headers" — this keeps a given `--offset` naming the same slice of the table
|
||
across repeated runs, so progress is easy to reason about (e.g. "slices 0,
|
||
1000, 2000, ... cover the whole table") independent of how much of it is
|
||
already done.
|
||
|
||
## Performance estimate (not measured — job didn't exist before this change)
|
||
|
||
Per plan §5.3: ~225 030 files, ~124 KB average. Each row costs an `open` +
|
||
`read` + header-only parse (no MIME-walk of attachments) + a batched UPDATE.
|
||
The full `gmail-bulk-import` run (parsing the whole mbox, including
|
||
attachment MIME-walk and inserts) took ~29 minutes. This job does less
|
||
per-message work but pays for opening 225k small files individually instead
|
||
of streaming one mbox — the plan's estimate is "same order of magnitude,
|
||
likely tens of minutes." Not measured directly in this change — see the
|
||
DoD note below.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
pip install -e jobs/gmail-header-backfill/
|
||
cd jobs/gmail-header-backfill && pytest
|
||
```
|
||
|
||
Pure unit tests, no DB or filesystem outside `tmp_path`/synthetic `.eml`
|
||
bytes — `run()` is tested by monkeypatching `asyncpg.connect` with an
|
||
in-memory fake connection. Covers: header parsing (multi-address `To`/`Cc`,
|
||
quoted display names with commas, multiple `Delivered-To` occurrences, RFC
|
||
2047 encoded-words including Polish diacritics, malformed encoded-words that
|
||
must not raise, missing/multiple `From`, `date_raw` preserving literal text
|
||
vs. `Date` header reformatting), idempotency (rows already carrying a
|
||
`headers` entity are skipped and never re-appended), batch flushing, and
|
||
`--limit`/`--offset` query shape.
|
||
|
||
## Definition of Done
|
||
|
||
Per `CLAUDE.md`: this job's smoke run is `gmail-header-backfill --dsn ...
|
||
--limit 100` (dry-run first, then `--apply` against a small slice) — run
|
||
against kb-postgres@PIHA over SSH, **not executed as part of this change**
|
||
without operator confirmation (UPDATE against production data). `pytest`
|
||
passes locally (33/33) before this commit; full 225 030-row backfill is out
|
||
of scope for this change — see the plan for the rollout sequence.
|