Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of
PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive
and drops them into Paperless consume/ for OCR, so the RAG layer has real
documents to work with before the full Paperless/Nextcloud envelope adapter
is built. sha256-first attachment matching (manifest filenames can carry raw
RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default
decodes today — confirmed against live data, ~10% of candidates were affected).
Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write.
Verified end-to-end on PIHA: dry-run + --apply both run against live
kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run
duplicates correctly deduped), Paperless picked them up and started OCR
immediately.
- Usuwa Dockerfile i docker-compose.yml; job odpalany lokalnie na PIHA (pip install -e)
- _parse_attachments: manifest MIME → entities[]{type,filename,content_type,size,sha256};
bajty zostają w .eml, wyciąganie/OCR = faza 2
- Batch inserty co 500 wpisów (executemany + ON CONFLICT DO NOTHING); idempotentny
na skipped przez _eml_ref (mirrors archive._UNSAFE); pełna wznawialność
- --limit N: ucina pętlę po N wiadomościach do testów na próbce
- epoch_fallback: licznik + WARNING gdy Date nieparsowalne/brak
- Nowe stats: msgs_with_attachments, total_attachments, total_attachment_bytes
- DSN w docstringu: localhost:5433/kb i piha:5433/kb; usunięto solaria:5433
- 9 nowych testów (24 razem), wszystkie zielone
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One-shot job `jobs/gmail-bulk-import/` wczytuje plik .mbox z Google Takeout
i importuje każdą wiadomość do archiwum .eml + opcjonalnie do koperty w DB.
Idempotentny (FileExistsError → skip; ON CONFLICT DO NOTHING w DB).
15 testów jednostkowych (bez DB, bez zewnętrznych serwisów) — wszystkie zielone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
