homelab-codex-ws/kb/services/job-documents-ingest.md

---
okf: "0.1"
type: service
visibility: private
status: active
updated: 2026-07-30
links:
  - ../phases/kb-m5-documents-ingest-fazy.md
  - ../runbooks/documents-ingest-run.md
---

# documents-ingest

One-shot job CLI, Phase 1 of module 5 (`kb/phases/kb-m5-documents-ingest.md`,
"Domkniecie dlugu z maili"). Extracts a **sample** of PDF attachments from the
Gmail `.eml` archive (already indexed in the `envelope` table of kb-postgres)
and drops them into Paperless' `consume/` directory so Paperless does the OCR
and correspondent-detection. This job does **not** write to the `envelope`
table — the Paperless/Nextcloud envelope adapter is a later phase of module 5.

## Why a sample, not a bulk import

The Gmail import left ~70k attachments referenced in `envelope.entities`
manifests (bytes live inside the archived `.eml` files, never extracted).
Dumping all of them into Paperless at once would swamp the OCR worker and the
RAG layer isn't built yet to make use of that volume. This job pulls a small,
recent, size-filtered sample (default: 150 envelopes, PDFs >50KB, from the
last year) as a testbed — mass import is a deliberate later decision.

## Where it runs

**Locally on PIHA**, as a plain CLI (not a container). It needs simultaneous
filesystem access to three things that all live on PIHA:

- the mail archive (`/home/oskar/kb/mail/archive`)
- the Paperless `consume/` directory (`/opt/homelab/data/paperless/consume`)
- kb-postgres (`localhost:5433` from PIHA; reachable from elsewhere over
  Tailscale, but the archive and consume dir are not — those are local paths)

Install (from repo root, on PIHA):

```bash
pip install -e jobs/documents-ingest/
```

(Or reuse the venv already set up for `gmail-bulk-import`, e.g.
`/home/oskar/kb/venv/` — it already has `asyncpg` + `structlog`.)

## Candidate selection

```sql
SELECT id, raw_ref, ts, entities FROM envelope
WHERE source = 'gmail'
  AND ts > now() - interval '1 year'
  AND EXISTS (
      SELECT 1 FROM jsonb_array_elements(entities) AS att
      WHERE att->>'content_type' = 'application/pdf'
        AND (att->>'size')::numeric > 50000
  )
ORDER BY ts DESC
LIMIT 150
```

For each matching envelope, every attachment manifest entry that passes the
filter is a separate candidate (one envelope can yield several PDFs).

## Matching an attachment inside the .eml

The manifest (`entities[]`) only has metadata — the attachment bytes live
inside the `.eml` (MIME multipart), so each candidate is resolved against the
freshly parsed message:

1. Parse the `.eml` with `email.policy.default` and collect every
   `application/pdf` MIME part (filename + decoded payload).
2. **sha256 is the proof of identity**, not the filename. The manifest was
   built by a different parser at import time (`gmail-bulk-import`, using
   `mailbox` + compat32 policy) and can still hold the raw RFC 2047
   encoded-word form of a filename (e.g. `=?UTF-8?b?...?=`, sometimes with
   header-folding whitespace baked in), while `email.policy.default` decodes
   it to real Unicode today. Comparing those byte-for-byte skipped ~10% of
   otherwise-good attachments in testing — see `TestFindPdfParts` /
   `TestProcessCandidate` in the test suite for the regression case. So:
   match by sha256 across all PDF parts in the message; if none match, use a
   filename match only to tell "found the named part but its bytes changed"
   (`sha_mismatch`, reported and skipped) apart from "not present at all"
   (`parse_error`, skipped).
3. The consume/ filename is built from the *decoded* filename (from the MIME
   part), not the possibly-garbled manifest one.

Mismatches and parse errors are never guessed past — they're logged and
skipped.

## consume/ filenames

`<YYYY-MM-DD>_<sanitized-filename>.pdf`, date = envelope `ts`. On collision
(same date + sanitized name already used in this run or already present in
`consume/`), an 8-hex sha256 prefix is appended:
`<YYYY-MM-DD>_<sanitized-filename>_<hash8>.pdf`.

Files are written with a best-effort `chown` to uid:gid `1000:1000` (the
Paperless container's `USERMAP_UID/GID`, see `kb/services/paperless.md`)
so Paperless can read them. If the chown fails (e.g. the job isn't running as
root/uid 1000), a warning is logged but the run continues — the write itself
already succeeded; fix ownership/perms on `consume/` separately if needed.
PIHA's uid/gid convention across the fleet is tracked as its own tech-debt
item (see `docs/backlog/`), not solved here.

## Idempotency — registry

A JSON file at `/opt/homelab/data/documents-ingest/registry.json` (default,
override with `--registry`), keyed by attachment sha256:

```json
{
  "<sha256>": {
    "envelope_id": "...",
    "filename": "...",
    "consume_name": "2026-06-09_invoice.pdf",
    "size": 123456,
    "ingested_at": "2026-07-13T19:35:16+00:00"
  }
}
```

**Why a JSON file and not a kb-postgres table:** this is a one-shot sampling
tool for a bootstrapping phase, not a long-running service — a new table
would formalize infrastructure for something temporary. A flat file needs no
migration, is trivial to inspect (`jq`) or reset, and sits under
`/opt/homelab/data/` alongside other node-local state per the repo's runtime
path convention. If/when module 5's real Paperless/Nextcloud adapter phase
starts writing `envelope` rows for `source=paperless`, that's the natural
point to fold this into a proper DB-backed ingest log — re-litigate then, not
now.

Re-running the job only ever *adds* to the registry (on `--apply`); it's
never consulted or mutated in dry-run mode beyond being read for the preview.

## Dry-run output

Logs one line per skip (`skip.duplicate` / `skip.sha_mismatch` /
`skip.parse_error`, with reason), a `summary` line with full counts
(`envelopes_scanned`, `pdf_candidates`, `extracted`, `skipped_duplicate`,
`skipped_sha_mismatch`, `skipped_parse_error`, `errors`), and up to 20 example
`(target_name, size, envelope_id)` rows so you can sanity-check filenames
before running `--apply`.