3755d7d6bb
Four audit findings (2026-07-14), each reproduced on crafted mboxes; the Takeout corpus has proven 8-bit header bytes, so all are real re-import risks. 1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an email.header.Header (not str) for raw 8-bit bytes; the old Header.strip() raised AttributeError. _message_id ran BEFORE the per-message try, so one bad header killed the entire import. Fix: str() + sanitize_surrogates() before strip; and move _message_id/_parse_date/_parse_attachments INSIDE the per-message try — a broken message is now errors += 1, never run death. 2. Poison batch. pending.clear() ran only AFTER a successful insert, so a failed flush (DB down / bad row) left pending intact and every later message re-flushed the doomed batch; the final flush sat in try/finally with no except and propagated out, losing all stats. Fix: _flush always clears pending and counts a failed insert as db_insert_failed; the run always reaches import_complete. 3. Stats didn't reconcile with the DB. imported counts archive writes, not DB rows, so a partial-insert drift was invisible. Fix: separate db_inserted/db_insert_failed counters; main() exits non-zero on any error, DB drift, or a processed = imported + skipped + errors imbalance. 4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header) raised even when str(header) parses fine. Fix: str() before the epoch fallback. Shared helper: _sanitize moved from gmail-header-backfill into packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs; gmail-header-backfill now depends on kb-mail. Tests: regression coverage for all four findings in gmail-bulk-import (8-bit id/date, per-message guard, failed-insert non-poisoning, stats balance) plus kb_mail.text unit tests. Full suites green: kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
13 lines
262 B
Python
13 lines
262 B
Python
from .archive import save_eml
|
|
from .db import get_envelope, insert_envelope
|
|
from .envelope import Envelope
|
|
from .text import sanitize_surrogates
|
|
|
|
__all__ = [
|
|
"Envelope",
|
|
"insert_envelope",
|
|
"get_envelope",
|
|
"save_eml",
|
|
"sanitize_surrogates",
|
|
]
|