oskar/homelab-codex-ws
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
homelab-codex-ws/packages/kb-mail/src/kb_mail/__init__.py

13 lines
262 B
Python
Raw Normal View History

feat(kb-mail): fundament — pgvector spine, koperta, archiwum, pakiet domeny - services/kb-postgres: pgvector/pgvector:pg16 na SOLARIA (:5433), named volume, init/001_envelope.sql (CREATE EXTENSION vector + zamrożona tabela envelope: id/source/ts/geo/raw_ref/entities), service.yaml, healthcheck, README z poprawnym mechanizmem deploy (deploy-node.sh składa dwa -f) - hosts/solaria/runtime/kb-postgres/docker-compose.override.yml: mem_limit 4g - inventory/topology.yaml + hosts/solaria/services.yaml: kb-postgres wpisany - packages/kb-mail: nowa konwencja shared lib (pip install /repo/packages/<lib>/) envelope.py — @dataclass Envelope, walidacja tz-aware ts db.py — insert_envelope / get_envelope (asyncpg, ON CONFLICT DO NOTHING) archive.py — save_eml append-only (asyncio.to_thread, FileExistsError na dup) tests: 15 unit pass + 5 integration (@pytest.mark.integration, wymaga KB_TEST_DSN) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-17 21:50:49 +02:00
from .archive import save_eml
from .db import get_envelope, insert_envelope
from .envelope import Envelope
fix(gmail-bulk-import): harden re-import against 8-bit headers + poison batches Four audit findings (2026-07-14), each reproduced on crafted mboxes; the Takeout corpus has proven 8-bit header bytes, so all are real re-import risks. 1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an email.header.Header (not str) for raw 8-bit bytes; the old Header.strip() raised AttributeError. _message_id ran BEFORE the per-message try, so one bad header killed the entire import. Fix: str() + sanitize_surrogates() before strip; and move _message_id/_parse_date/_parse_attachments INSIDE the per-message try — a broken message is now errors += 1, never run death. 2. Poison batch. pending.clear() ran only AFTER a successful insert, so a failed flush (DB down / bad row) left pending intact and every later message re-flushed the doomed batch; the final flush sat in try/finally with no except and propagated out, losing all stats. Fix: _flush always clears pending and counts a failed insert as db_insert_failed; the run always reaches import_complete. 3. Stats didn't reconcile with the DB. imported counts archive writes, not DB rows, so a partial-insert drift was invisible. Fix: separate db_inserted/db_insert_failed counters; main() exits non-zero on any error, DB drift, or a processed = imported + skipped + errors imbalance. 4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header) raised even when str(header) parses fine. Fix: str() before the epoch fallback. Shared helper: _sanitize moved from gmail-header-backfill into packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs; gmail-header-backfill now depends on kb-mail. Tests: regression coverage for all four findings in gmail-bulk-import (8-bit id/date, per-message guard, failed-insert non-poisoning, stats balance) plus kb_mail.text unit tests. Full suites green: kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 19:51:35 +02:00
from .text import sanitize_surrogates
feat(kb-mail): fundament — pgvector spine, koperta, archiwum, pakiet domeny - services/kb-postgres: pgvector/pgvector:pg16 na SOLARIA (:5433), named volume, init/001_envelope.sql (CREATE EXTENSION vector + zamrożona tabela envelope: id/source/ts/geo/raw_ref/entities), service.yaml, healthcheck, README z poprawnym mechanizmem deploy (deploy-node.sh składa dwa -f) - hosts/solaria/runtime/kb-postgres/docker-compose.override.yml: mem_limit 4g - inventory/topology.yaml + hosts/solaria/services.yaml: kb-postgres wpisany - packages/kb-mail: nowa konwencja shared lib (pip install /repo/packages/<lib>/) envelope.py — @dataclass Envelope, walidacja tz-aware ts db.py — insert_envelope / get_envelope (asyncpg, ON CONFLICT DO NOTHING) archive.py — save_eml append-only (asyncio.to_thread, FileExistsError na dup) tests: 15 unit pass + 5 integration (@pytest.mark.integration, wymaga KB_TEST_DSN) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-17 21:50:49 +02:00
fix(gmail-bulk-import): harden re-import against 8-bit headers + poison batches Four audit findings (2026-07-14), each reproduced on crafted mboxes; the Takeout corpus has proven 8-bit header bytes, so all are real re-import risks. 1. Whole-run crash on 8-bit Message-ID. compat32 .get() returns an email.header.Header (not str) for raw 8-bit bytes; the old Header.strip() raised AttributeError. _message_id ran BEFORE the per-message try, so one bad header killed the entire import. Fix: str() + sanitize_surrogates() before strip; and move _message_id/_parse_date/_parse_attachments INSIDE the per-message try — a broken message is now errors += 1, never run death. 2. Poison batch. pending.clear() ran only AFTER a successful insert, so a failed flush (DB down / bad row) left pending intact and every later message re-flushed the doomed batch; the final flush sat in try/finally with no except and propagated out, losing all stats. Fix: _flush always clears pending and counts a failed insert as db_insert_failed; the run always reaches import_complete. 3. Stats didn't reconcile with the DB. imported counts archive writes, not DB rows, so a partial-insert drift was invisible. Fix: separate db_inserted/db_insert_failed counters; main() exits non-zero on any error, DB drift, or a processed = imported + skipped + errors imbalance. 4. 8-bit Date → needless epoch_fallback. parsedate_to_datetime(Header) raised even when str(header) parses fine. Fix: str() before the epoch fallback. Shared helper: _sanitize moved from gmail-header-backfill into packages/kb-mail (kb_mail.text.sanitize_surrogates) and used by both jobs; gmail-header-backfill now depends on kb-mail. Tests: regression coverage for all four findings in gmail-bulk-import (8-bit id/date, per-message guard, failed-insert non-poisoning, stats balance) plus kb_mail.text unit tests. Full suites green: kb-mail 27, gmail-bulk-import 33, gmail-header-backfill 43. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 19:51:35 +02:00
__all__ = [
"Envelope",
"insert_envelope",
"get_envelope",
"save_eml",
"sanitize_surrogates",
]
Reference in a new issue Copy permalink