Diagnosis of the 5008 header-less envelopes after the full 225 030-row run
(read-only, on PIHA, 2026-07-14):
- 4999 = one contiguous block at ORDER BY id positions 70001-74999: the
--offset 70000 slice died mid-run. Root cause reproduced: a Date: header
with raw 8-bit bytes makes compat32 .get() return email.header.Header
(not str), and json.dumps([headers]) — which sat OUTSIDE the per-row
try/except — raised TypeError and killed the process, silently losing
the rest of the slice.
- 9 = genuine typed-parse failures: 7x RFC 2047 encoded-word decoding to
CR/LF inside a display name (ValueError in headerregistry), 1x RFC 5322
group syntax in To: ("unlisted-recipients:;"), 1x CPython
_header_value_parser bug on a malformed display name (fixed upstream,
present on PIHA's 3.11).
- 0 missing .eml files.
Fixes:
- date_raw: str() + surrogate sanitization on the compat32 value — the
crash cause, now also covered by a regression test.
- json.dumps moved inside the per-row try: a non-serializable value counts
as that row's parse_error instead of crashing the slice.
- parse_headers_fallback(): on typed-parse failure retry with a pure
compat32 parse — getaddresses over raw header text, raw-string values,
same §4.1 entity shape. Counted separately as parsed_fallback (labeled
subset of updated), logged per row with the original typed error.
- missing_file counter + skip.missing_file info log (id, expected path);
run_complete now balances: scanned = updated + already_has_headers +
parse_errors + read_errors + missing_file. Non-zero missing_file also
fails the exit code.
Verified: 43/43 pytest locally (3.13) and on PIHA (3.11); read-only dry
runs on PIHA — all 9 parse failures recover via fallback, the lost slice
completes scanned=5000 updated=4999 already_has_headers=1 with zero errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Module 5 phase 2 (docs/kb/modules/05-faza2-plan.md, §5). 225,030 existing
source='gmail' envelope rows carry only an attachment manifest — no
from/to/cc/delivered_to/subject anywhere in the DB, which blocks answering
"is this on me or my wife" / distinguishing aliases (plan §1.8). Separate
job from gmail-bulk-import per plan §2 decision 7: UPDATE semantics against
production data is a different risk profile than the historical INSERT job.
Parses headers only (no MIME-walk of attachments) from the archived .eml
files, appends {"type": "headers", ...} (plan §4.1) via the idempotent
UPDATE ... WHERE NOT EXISTS from §5.2, batched via executemany. --limit/
--offset (ORDER BY id) give stable, deterministic partitioning so the full
backfill can run and be verified in slices instead of one unattended pass.
Plain CLI (pip install -e), no Dockerfile — same convention as
gmail-bulk-import/documents-ingest, which run directly on PIHA for local
filesystem access to the .eml archive.
Verified against kb-postgres@PIHA (100-row dry-run + apply, re-run proved
idempotent no-op, 1000-row timed slice): 1096/225030 rows backfilled,
996/1000 succeeded on the timed slice (4 parse_errors — a 2014 spam message
with an RFC 2047 encoded-word decoding to an embedded newline in the From
display name, correctly caught and skipped rather than crashing the batch).
Extrapolated full-run time ~16 minutes. Full 225,030-row run is out of
scope for this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of
PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive
and drops them into Paperless consume/ for OCR, so the RAG layer has real
documents to work with before the full Paperless/Nextcloud envelope adapter
is built. sha256-first attachment matching (manifest filenames can carry raw
RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default
decodes today — confirmed against live data, ~10% of candidates were affected).
Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write.
Verified end-to-end on PIHA: dry-run + --apply both run against live
kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run
duplicates correctly deduped), Paperless picked them up and started OCR
immediately.
- Usuwa Dockerfile i docker-compose.yml; job odpalany lokalnie na PIHA (pip install -e)
- _parse_attachments: manifest MIME → entities[]{type,filename,content_type,size,sha256};
bajty zostają w .eml, wyciąganie/OCR = faza 2
- Batch inserty co 500 wpisów (executemany + ON CONFLICT DO NOTHING); idempotentny
na skipped przez _eml_ref (mirrors archive._UNSAFE); pełna wznawialność
- --limit N: ucina pętlę po N wiadomościach do testów na próbce
- epoch_fallback: licznik + WARNING gdy Date nieparsowalne/brak
- Nowe stats: msgs_with_attachments, total_attachments, total_attachment_bytes
- DSN w docstringu: localhost:5433/kb i piha:5433/kb; usunięto solaria:5433
- 9 nowych testów (24 razem), wszystkie zielone
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One-shot job `jobs/gmail-bulk-import/` wczytuje plik .mbox z Google Takeout
i importuje każdą wiadomość do archiwum .eml + opcjonalnie do koperty w DB.
Idempotentny (FileExistsError → skip; ON CONFLICT DO NOTHING w DB).
15 testów jednostkowych (bez DB, bez zewnętrznych serwisów) — wszystkie zielone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
