homelab-codex-ws/jobs/documents-ingest/README.md

documents-ingest

One-shot job CLI, Phase 1 of module 5 (docs/kb/modules/05-documents-ingest.md, "Domkniecie dlugu z maili"). Extracts a sample of PDF attachments from the Gmail .eml archive (already indexed in the envelope table of kb-postgres) and drops them into Paperless' consume/ directory so Paperless does the OCR and correspondent-detection. This job does not write to the envelope table — the Paperless/Nextcloud envelope adapter is a later phase of module 5.

Why a sample, not a bulk import

The Gmail import left ~70k attachments referenced in envelope.entities manifests (bytes live inside the archived .eml files, never extracted). Dumping all of them into Paperless at once would swamp the OCR worker and the RAG layer isn't built yet to make use of that volume. This job pulls a small, recent, size-filtered sample (default: 150 envelopes, PDFs >50KB, from the last year) as a testbed — mass import is a deliberate later decision.

Where it runs

Locally on PIHA, as a plain CLI (not a container). It needs simultaneous filesystem access to three things that all live on PIHA:

• the mail archive (/home/oskar/kb/mail/archive)
• the Paperless consume/ directory (/opt/homelab/data/paperless/consume)
• kb-postgres (localhost:5433 from PIHA; reachable from elsewhere over Tailscale, but the archive and consume dir are not — those are local paths)

Install (from repo root, on PIHA):

pip install -e jobs/documents-ingest/

(Or reuse the venv already set up for gmail-bulk-import, e.g. /home/oskar/kb/venv/ — it already has asyncpg + structlog.)

Usage

# Dry run (default) — preview only, no writes:
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb

# Or via env var instead of --dsn:
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
documents-ingest

# Real run — write files into consume/ and update the registry:
documents-ingest --apply

# Smaller/larger sample, different window/threshold:
documents-ingest --limit 50 --since-days 180 --min-size 100000

Dry-run is the default and does not require --consume-dir to exist yet; --apply does (Paperless must already be deployed with its consume dir in place). See documents-ingest --help for all flags.

Candidate selection

SELECT id, raw_ref, ts, entities FROM envelope
WHERE source = 'gmail'
  AND ts > now() - interval '1 year'
  AND EXISTS (
      SELECT 1 FROM jsonb_array_elements(entities) AS att
      WHERE att->>'content_type' = 'application/pdf'
        AND (att->>'size')::numeric > 50000
  )
ORDER BY ts DESC
LIMIT 150

For each matching envelope, every attachment manifest entry that passes the filter is a separate candidate (one envelope can yield several PDFs).

Matching an attachment inside the .eml

The manifest (entities[]) only has metadata — the attachment bytes live inside the .eml (MIME multipart), so each candidate is resolved against the freshly parsed message:

1. Parse the .eml with email.policy.default and collect every application/pdf MIME part (filename + decoded payload).
2. sha256 is the proof of identity, not the filename. The manifest was built by a different parser at import time (gmail-bulk-import, using mailbox + compat32 policy) and can still hold the raw RFC 2047 encoded-word form of a filename (e.g. =?UTF-8?b?...?=, sometimes with header-folding whitespace baked in), while email.policy.default decodes it to real Unicode today. Comparing those byte-for-byte skipped ~10% of otherwise-good attachments in testing — see TestFindPdfParts / TestProcessCandidate in the test suite for the regression case. So: match by sha256 across all PDF parts in the message; if none match, use a filename match only to tell "found the named part but its bytes changed" (sha_mismatch, reported and skipped) apart from "not present at all" (parse_error, skipped).
3. The consume/ filename is built from the decoded filename (from the MIME part), not the possibly-garbled manifest one.

Mismatches and parse errors are never guessed past — they're logged and skipped.

consume/ filenames

<YYYY-MM-DD>_<sanitized-filename>.pdf, date = envelope ts. On collision (same date + sanitized name already used in this run or already present in consume/), an 8-hex sha256 prefix is appended: <YYYY-MM-DD>_<sanitized-filename>_<hash8>.pdf.

Files are written with a best-effort chown to uid:gid 1000:1000 (the Paperless container's USERMAP_UID/GID, see services/paperless/README.md) so Paperless can read them. If the chown fails (e.g. the job isn't running as root/uid 1000), a warning is logged but the run continues — the write itself already succeeded; fix ownership/perms on consume/ separately if needed. PIHA's uid/gid convention across the fleet is tracked as its own tech-debt item (see docs/backlog/), not solved here.

Idempotency — registry

A JSON file at /opt/homelab/data/documents-ingest/registry.json (default, override with --registry), keyed by attachment sha256:

{
  "<sha256>": {
    "envelope_id": "...",
    "filename": "...",
    "consume_name": "2026-06-09_invoice.pdf",
    "size": 123456,
    "ingested_at": "2026-07-13T19:35:16+00:00"
  }
}

Why a JSON file and not a kb-postgres table: this is a one-shot sampling tool for a bootstrapping phase, not a long-running service — a new table would formalize infrastructure for something temporary. A flat file needs no migration, is trivial to inspect (jq) or reset, and sits under /opt/homelab/data/ alongside other node-local state per the repo's runtime path convention. If/when module 5's real Paperless/Nextcloud adapter phase starts writing envelope rows for source=paperless, that's the natural point to fold this into a proper DB-backed ingest log — re-litigate then, not now.

Re-running the job only ever adds to the registry (on --apply); it's never consulted or mutated in dry-run mode beyond being read for the preview.

Dry-run output

Logs one line per skip (skip.duplicate / skip.sha_mismatch / skip.parse_error, with reason), a summary line with full counts (envelopes_scanned, pdf_candidates, extracted, skipped_duplicate, skipped_sha_mismatch, skipped_parse_error, errors), and up to 20 example (target_name, size, envelope_id) rows so you can sanity-check filenames before running --apply.

Verifying the result in Paperless

After --apply:

1. Paperless' consumer picks files up from consume/ automatically (polling or inotify, per its own config) — no action needed on this job's side.
2. Watch progress: Paperless UI → Documents (new items appear as OCR finishes), or docker logs -f paperless on PIHA for consumer/OCR activity.
3. Cross-check count: number of new documents in Paperless should equal stats["extracted"] from the --apply run's summary line.
4. Confirm idempotency: re-running --apply immediately after should report extracted: 0 and skipped_duplicate equal to the previous run's extracted count — nothing new lands in consume/.

Tests

pip install -e jobs/documents-ingest/
cd jobs/documents-ingest && pytest

Pure unit tests, no DB or filesystem outside tmp_path required — run() is tested by monkeypatching asyncpg.connect with an in-memory fake connection. Covers: filename sanitization, consume-name collision handling, manifest filtering, MIME PDF-part extraction (including the RFC 2047 decoding mismatch), sha256 match/mismatch, duplicate detection, dry-run vs --apply behavior, and multi-attachment envelopes.

---

Phase 2 — documents-ingest-paperless (Paperless -> envelope adapter)

Module 5, phase 2 (docs/kb/modules/05-faza2-plan.md, §4.2-4.3, §6 step 5). Reads documents from the Paperless REST API (read-only — GET only, never writes to Paperless) and inserts them as source='paperless' rows into the envelope table on kb-postgres, reusing kb_mail.envelope.Envelope / kb_mail.db.insert_envelope from packages/kb-mail (untouched by this change — see plan §1.6). Existing source='gmail' rows and document_chunk are never touched; this job only ever INSERTs new paperless rows.

Cross-source link (source_mail)

Per plan §1.9/§4.2, the deterministic join uses no heuristics: a document's original_file_name (from the Paperless API) is matched against consume_name in this job's phase-1 registry (/opt/homelab/data/documents-ingest/registry.json, produced by extractor.py — see above). A match appends a source_mail entity pointing back at the originating mail envelope; no match means the document was added outside the faktury-1 pipeline, and the entity is simply omitted — not an error.

Install

pip install -e packages/kb-mail/
pip install -e jobs/documents-ingest/

Usage

# Dry run (default) — fetch from Paperless, map, count; no DB writes:
documents-ingest-paperless --dsn postgresql://kb:<pw>@localhost:5433/kb \
    --paperless-token <token>

# Real run — insert new envelope rows:
documents-ingest-paperless --dsn ... --paperless-token ... --apply

# Smoke-test slice:
documents-ingest-paperless --dsn ... --paperless-token ... --limit 5

--dsn can come from KB_DSN, --paperless-token from PAPERLESS_API_TOKEN, --paperless-url from PAPERLESS_URL (defaults to Paperless' fixed LAN address, http://192.168.31.5:8210). No --offset: unlike the 225 030-row header backfill, a full re-scan of Paperless' ~186 documents is cheap and already idempotent, so there is no need for resumable partitioning — --limit exists only to cap a run for smoke-testing.

Mapping (plan §4.3)

id       = f"paperless:{document_id}"   -- prefixed: Paperless doc-ids are small
                                         -- sequential ints that would otherwise
                                         -- collide with any future source's ids
ts       = documents_document.created   -- Paperless-detected date (content/filename),
                                         -- not filesystem mtime
geo      = NULL
raw_ref  = str(document_id)             -- REFERENCE — Paperless is the source of truth,
                                         -- no bytes are copied
entities = content, correspondent, tag(s), filename, content_type,
           and source_mail when the registry join hits (plan §4.2)

correspondent/tag are resolved from Paperless' /api/correspondents/ and /api/tags/ (fetched once, cached in memory for the run) and kept purely as informational metadata — nothing in this pipeline depends on them being non-null (plan decision 4). A document with empty OCR content (Paperless OCR sometimes produces none) still gets a normal envelope with "text": "" — not skipped, not an error, just counted (empty_content).

Idempotency

A pre-fetched set of existing source='paperless' envelope ids (one query at the start of each run) skips documents already inserted; insert_envelope's own ON CONFLICT (id) DO NOTHING is the second line of defense. Re-running --apply immediately after a successful run reports inserted: 0 and already_in_db equal to the previous run's inserted count.

Stats must balance

fetched = already_in_db + inserted + errors

source_mail_linked and empty_content are informational subsets of fetched, not separate outcome buckets. A per-document mapping failure (e.g. an unparseable created date) is isolated, logged, and counted as errors — it never aborts the run. main() exits 1 on non-zero errors or if the balance invariant above doesn't hold (mirrors gmail-bulk-import's exit-code convention) — a clean run always exits 0.

Tests

pip install -e packages/kb-mail/
pip install -e jobs/documents-ingest/
cd jobs/documents-ingest && pytest

Pure unit tests, no DB or real HTTP — run() is tested by monkeypatching asyncpg.connect (fake connection) and aiohttp.ClientSession (fake session serving canned JSON pages). Covers: mapping shape (content, correspondent, tag(s), filename, content_type, source_mail), the registry join (hit and miss), pagination (both the documents list and the correspondents/tags lookup tables), --limit, idempotency (pre-existing ids skipped, a second --apply run inserts nothing new), isolated per-document mapping errors, and the stats-balance invariant.

Definition of Done

Per CLAUDE.md: smoke run is documents-ingest-paperless --dsn ... --paperless-token ... --limit 5 (dry-run first) against kb-postgres@PIHA and the live Paperless API, over SSH — not executed as part of this change without operator confirmation (this job reads production Paperless data and writes production envelope rows on --apply). pytest passes locally before this commit.