3755d7d6bb
Four audit findings (2026-07-14), each reproduced on crafted mboxes; the Takeout corpus has proven 8-bit header bytes, so all are real re-import risks. 1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an email.header.Header (not str) for raw 8-bit bytes; the old Header.strip() raised AttributeError. _message_id ran BEFORE the per-message try, so one bad header killed the entire import. Fix: str() + sanitize_surrogates() before strip; and move _message_id/_parse_date/_parse_attachments INSIDE the per-message try — a broken message is now errors += 1, never run death. 2. Poison batch. pending.clear() ran only AFTER a successful insert, so a failed flush (DB down / bad row) left pending intact and every later message re-flushed the doomed batch; the final flush sat in try/finally with no except and propagated out, losing all stats. Fix: _flush always clears pending and counts a failed insert as db_insert_failed; the run always reaches import_complete. 3. Stats didn't reconcile with the DB. imported counts archive writes, not DB rows, so a partial-insert drift was invisible. Fix: separate db_inserted/db_insert_failed counters; main() exits non-zero on any error, DB drift, or a processed = imported + skipped + errors imbalance. 4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header) raised even when str(header) parses fine. Fix: str() before the epoch fallback. Shared helper: _sanitize moved from gmail-header-backfill into packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs; gmail-header-backfill now depends on kb-mail. Tests: regression coverage for all four findings in gmail-bulk-import (8-bit id/date, per-message guard, failed-insert non-poisoning, stats balance) plus kb_mail.text unit tests. Full suites green: kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
33 lines
990 B
Python
33 lines
990 B
Python
"""Unit tests for kb_mail.text.sanitize_surrogates."""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
|
||
from kb_mail.text import sanitize_surrogates
|
||
|
||
|
||
def test_none_stays_none():
|
||
assert sanitize_surrogates(None) is None
|
||
|
||
|
||
def test_plain_ascii_unchanged():
|
||
assert sanitize_surrogates("hello") == "hello"
|
||
|
||
|
||
def test_real_unicode_passes_through():
|
||
assert sanitize_surrogates("Kąpała") == "Kąpała"
|
||
|
||
|
||
def test_lone_surrogate_degraded_and_json_safe():
|
||
# A compat32 bytes-parse can leave a lone surrogate (undecodable byte);
|
||
# postgres jsonb and json.dumps().encode('utf-8') both reject it.
|
||
dirty = "Pr\udce9sent" # 0xe9 smuggled in as a surrogate escape
|
||
cleaned = sanitize_surrogates(dirty)
|
||
assert cleaned == "Pr?sent"
|
||
json.dumps(cleaned, ensure_ascii=False).encode("utf-8") # must not raise
|
||
|
||
|
||
def test_replacement_char_preserved():
|
||
# U+FFFD is already valid UTF-8 and must survive untouched.
|
||
assert sanitize_surrogates("bad<EFBFBD>id") == "bad<EFBFBD>id"
|