feat(kb): SPLIT documents-ingest (30 KB) -> service + phase + runbook
kb/services/job-documents-ingest.md — opis jobu i mechanizmow (candidate selection, matching, consume/, idempotency, dry-run) kb/phases/kb-m5-documents-ingest-fazy.md — faza 2, faza 2 krok 6, faza 3 krok 4, faza 3 krok 5 (4 sekcje fazowe wtopione w README) kb/runbooks/documents-ingest-run.md — Usage, Verifying in Paperless, Tests Najwiekszy README w repo. Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
oskar
parent
5439c00def
commit
ccb738cddf
|
|
@ -1,188 +1,15 @@
|
||||||
# documents-ingest
|
|
||||||
|
|
||||||
One-shot job CLI, Phase 1 of module 5 (`docs/kb/modules/05-documents-ingest.md`,
|
|
||||||
"Domkniecie dlugu z maili"). Extracts a **sample** of PDF attachments from the
|
|
||||||
Gmail `.eml` archive (already indexed in the `envelope` table of kb-postgres)
|
|
||||||
and drops them into Paperless' `consume/` directory so Paperless does the OCR
|
|
||||||
and correspondent-detection. This job does **not** write to the `envelope`
|
|
||||||
table — the Paperless/Nextcloud envelope adapter is a later phase of module 5.
|
|
||||||
|
|
||||||
## Why a sample, not a bulk import
|
|
||||||
|
|
||||||
The Gmail import left ~70k attachments referenced in `envelope.entities`
|
|
||||||
manifests (bytes live inside the archived `.eml` files, never extracted).
|
|
||||||
Dumping all of them into Paperless at once would swamp the OCR worker and the
|
|
||||||
RAG layer isn't built yet to make use of that volume. This job pulls a small,
|
|
||||||
recent, size-filtered sample (default: 150 envelopes, PDFs >50KB, from the
|
|
||||||
last year) as a testbed — mass import is a deliberate later decision.
|
|
||||||
|
|
||||||
## Where it runs
|
|
||||||
|
|
||||||
**Locally on PIHA**, as a plain CLI (not a container). It needs simultaneous
|
|
||||||
filesystem access to three things that all live on PIHA:
|
|
||||||
|
|
||||||
- the mail archive (`/home/oskar/kb/mail/archive`)
|
|
||||||
- the Paperless `consume/` directory (`/opt/homelab/data/paperless/consume`)
|
|
||||||
- kb-postgres (`localhost:5433` from PIHA; reachable from elsewhere over
|
|
||||||
Tailscale, but the archive and consume dir are not — those are local paths)
|
|
||||||
|
|
||||||
Install (from repo root, on PIHA):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -e jobs/documents-ingest/
|
|
||||||
```
|
|
||||||
|
|
||||||
(Or reuse the venv already set up for `gmail-bulk-import`, e.g.
|
|
||||||
`/home/oskar/kb/venv/` — it already has `asyncpg` + `structlog`.)
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Dry run (default) — preview only, no writes:
|
|
||||||
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb
|
|
||||||
|
|
||||||
# Or via env var instead of --dsn:
|
|
||||||
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
|
|
||||||
documents-ingest
|
|
||||||
|
|
||||||
# Real run — write files into consume/ and update the registry:
|
|
||||||
documents-ingest --apply
|
|
||||||
|
|
||||||
# Smaller/larger sample, different window/threshold:
|
|
||||||
documents-ingest --limit 50 --since-days 180 --min-size 100000
|
|
||||||
```
|
|
||||||
|
|
||||||
Dry-run is the default and does not require `--consume-dir` to exist yet;
|
|
||||||
`--apply` does (Paperless must already be deployed with its consume dir in
|
|
||||||
place). See `documents-ingest --help` for all flags.
|
|
||||||
|
|
||||||
## Candidate selection
|
|
||||||
|
|
||||||
```sql
|
|
||||||
SELECT id, raw_ref, ts, entities FROM envelope
|
|
||||||
WHERE source = 'gmail'
|
|
||||||
AND ts > now() - interval '1 year'
|
|
||||||
AND EXISTS (
|
|
||||||
SELECT 1 FROM jsonb_array_elements(entities) AS att
|
|
||||||
WHERE att->>'content_type' = 'application/pdf'
|
|
||||||
AND (att->>'size')::numeric > 50000
|
|
||||||
)
|
|
||||||
ORDER BY ts DESC
|
|
||||||
LIMIT 150
|
|
||||||
```
|
|
||||||
|
|
||||||
For each matching envelope, every attachment manifest entry that passes the
|
|
||||||
filter is a separate candidate (one envelope can yield several PDFs).
|
|
||||||
|
|
||||||
## Matching an attachment inside the .eml
|
|
||||||
|
|
||||||
The manifest (`entities[]`) only has metadata — the attachment bytes live
|
|
||||||
inside the `.eml` (MIME multipart), so each candidate is resolved against the
|
|
||||||
freshly parsed message:
|
|
||||||
|
|
||||||
1. Parse the `.eml` with `email.policy.default` and collect every
|
|
||||||
`application/pdf` MIME part (filename + decoded payload).
|
|
||||||
2. **sha256 is the proof of identity**, not the filename. The manifest was
|
|
||||||
built by a different parser at import time (`gmail-bulk-import`, using
|
|
||||||
`mailbox` + compat32 policy) and can still hold the raw RFC 2047
|
|
||||||
encoded-word form of a filename (e.g. `=?UTF-8?b?...?=`, sometimes with
|
|
||||||
header-folding whitespace baked in), while `email.policy.default` decodes
|
|
||||||
it to real Unicode today. Comparing those byte-for-byte skipped ~10% of
|
|
||||||
otherwise-good attachments in testing — see `TestFindPdfParts` /
|
|
||||||
`TestProcessCandidate` in the test suite for the regression case. So:
|
|
||||||
match by sha256 across all PDF parts in the message; if none match, use a
|
|
||||||
filename match only to tell "found the named part but its bytes changed"
|
|
||||||
(`sha_mismatch`, reported and skipped) apart from "not present at all"
|
|
||||||
(`parse_error`, skipped).
|
|
||||||
3. The consume/ filename is built from the *decoded* filename (from the MIME
|
|
||||||
part), not the possibly-garbled manifest one.
|
|
||||||
|
|
||||||
Mismatches and parse errors are never guessed past — they're logged and
|
|
||||||
skipped.
|
|
||||||
|
|
||||||
## consume/ filenames
|
|
||||||
|
|
||||||
`<YYYY-MM-DD>_<sanitized-filename>.pdf`, date = envelope `ts`. On collision
|
|
||||||
(same date + sanitized name already used in this run or already present in
|
|
||||||
`consume/`), an 8-hex sha256 prefix is appended:
|
|
||||||
`<YYYY-MM-DD>_<sanitized-filename>_<hash8>.pdf`.
|
|
||||||
|
|
||||||
Files are written with a best-effort `chown` to uid:gid `1000:1000` (the
|
|
||||||
Paperless container's `USERMAP_UID/GID`, see `services/paperless/README.md`)
|
|
||||||
so Paperless can read them. If the chown fails (e.g. the job isn't running as
|
|
||||||
root/uid 1000), a warning is logged but the run continues — the write itself
|
|
||||||
already succeeded; fix ownership/perms on `consume/` separately if needed.
|
|
||||||
PIHA's uid/gid convention across the fleet is tracked as its own tech-debt
|
|
||||||
item (see `docs/backlog/`), not solved here.
|
|
||||||
|
|
||||||
## Idempotency — registry
|
|
||||||
|
|
||||||
A JSON file at `/opt/homelab/data/documents-ingest/registry.json` (default,
|
|
||||||
override with `--registry`), keyed by attachment sha256:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"<sha256>": {
|
|
||||||
"envelope_id": "...",
|
|
||||||
"filename": "...",
|
|
||||||
"consume_name": "2026-06-09_invoice.pdf",
|
|
||||||
"size": 123456,
|
|
||||||
"ingested_at": "2026-07-13T19:35:16+00:00"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why a JSON file and not a kb-postgres table:** this is a one-shot sampling
|
|
||||||
tool for a bootstrapping phase, not a long-running service — a new table
|
|
||||||
would formalize infrastructure for something temporary. A flat file needs no
|
|
||||||
migration, is trivial to inspect (`jq`) or reset, and sits under
|
|
||||||
`/opt/homelab/data/` alongside other node-local state per the repo's runtime
|
|
||||||
path convention. If/when module 5's real Paperless/Nextcloud adapter phase
|
|
||||||
starts writing `envelope` rows for `source=paperless`, that's the natural
|
|
||||||
point to fold this into a proper DB-backed ingest log — re-litigate then, not
|
|
||||||
now.
|
|
||||||
|
|
||||||
Re-running the job only ever *adds* to the registry (on `--apply`); it's
|
|
||||||
never consulted or mutated in dry-run mode beyond being read for the preview.
|
|
||||||
|
|
||||||
## Dry-run output
|
|
||||||
|
|
||||||
Logs one line per skip (`skip.duplicate` / `skip.sha_mismatch` /
|
|
||||||
`skip.parse_error`, with reason), a `summary` line with full counts
|
|
||||||
(`envelopes_scanned`, `pdf_candidates`, `extracted`, `skipped_duplicate`,
|
|
||||||
`skipped_sha_mismatch`, `skipped_parse_error`, `errors`), and up to 20 example
|
|
||||||
`(target_name, size, envelope_id)` rows so you can sanity-check filenames
|
|
||||||
before running `--apply`.
|
|
||||||
|
|
||||||
## Verifying the result in Paperless
|
|
||||||
|
|
||||||
After `--apply`:
|
|
||||||
|
|
||||||
1. Paperless' consumer picks files up from `consume/` automatically (polling
|
|
||||||
or inotify, per its own config) — no action needed on this job's side.
|
|
||||||
2. Watch progress: Paperless UI → Documents (new items appear as OCR
|
|
||||||
finishes), or `docker logs -f paperless` on PIHA for consumer/OCR activity.
|
|
||||||
3. Cross-check count: number of new documents in Paperless should equal
|
|
||||||
`stats["extracted"]` from the `--apply` run's summary line.
|
|
||||||
4. Confirm idempotency: re-running `--apply` immediately after should report
|
|
||||||
`extracted: 0` and `skipped_duplicate` equal to the previous run's
|
|
||||||
`extracted` count — nothing new lands in `consume/`.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -e jobs/documents-ingest/
|
|
||||||
cd jobs/documents-ingest && pytest
|
|
||||||
```
|
|
||||||
|
|
||||||
Pure unit tests, no DB or filesystem outside `tmp_path` required — `run()` is
|
|
||||||
tested by monkeypatching `asyncpg.connect` with an in-memory fake connection.
|
|
||||||
Covers: filename sanitization, consume-name collision handling, manifest
|
|
||||||
filtering, MIME PDF-part extraction (including the RFC 2047 decoding
|
|
||||||
mismatch), sha256 match/mismatch, duplicate detection, dry-run vs `--apply`
|
|
||||||
behavior, and multi-attachment envelopes.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
okf: "0.1"
|
||||||
|
type: phase
|
||||||
|
visibility: private
|
||||||
|
status: active
|
||||||
|
updated: 2026-07-30
|
||||||
|
links:
|
||||||
|
- ../services/job-documents-ingest.md
|
||||||
|
- ../runbooks/documents-ingest-run.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# documents-ingest — fazy 2 i 3
|
||||||
|
|
||||||
## Phase 2 — `documents-ingest-paperless` (Paperless -> envelope adapter)
|
## Phase 2 — `documents-ingest-paperless` (Paperless -> envelope adapter)
|
||||||
|
|
||||||
64
kb/runbooks/documents-ingest-run.md
Normal file
64
kb/runbooks/documents-ingest-run.md
Normal file
|
|
@ -0,0 +1,64 @@
|
||||||
|
---
|
||||||
|
okf: "0.1"
|
||||||
|
type: runbook
|
||||||
|
visibility: private
|
||||||
|
status: active
|
||||||
|
updated: 2026-07-30
|
||||||
|
links:
|
||||||
|
- ../services/job-documents-ingest.md
|
||||||
|
- ../phases/kb-m5-documents-ingest-fazy.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# documents-ingest — uruchomienie i weryfikacja
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Dry run (default) — preview only, no writes:
|
||||||
|
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb
|
||||||
|
|
||||||
|
# Or via env var instead of --dsn:
|
||||||
|
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
|
||||||
|
documents-ingest
|
||||||
|
|
||||||
|
# Real run — write files into consume/ and update the registry:
|
||||||
|
documents-ingest --apply
|
||||||
|
|
||||||
|
# Smaller/larger sample, different window/threshold:
|
||||||
|
documents-ingest --limit 50 --since-days 180 --min-size 100000
|
||||||
|
```
|
||||||
|
|
||||||
|
Dry-run is the default and does not require `--consume-dir` to exist yet;
|
||||||
|
`--apply` does (Paperless must already be deployed with its consume dir in
|
||||||
|
place). See `documents-ingest --help` for all flags.
|
||||||
|
|
||||||
|
## Verifying the result in Paperless
|
||||||
|
|
||||||
|
After `--apply`:
|
||||||
|
|
||||||
|
1. Paperless' consumer picks files up from `consume/` automatically (polling
|
||||||
|
or inotify, per its own config) — no action needed on this job's side.
|
||||||
|
2. Watch progress: Paperless UI → Documents (new items appear as OCR
|
||||||
|
finishes), or `docker logs -f paperless` on PIHA for consumer/OCR activity.
|
||||||
|
3. Cross-check count: number of new documents in Paperless should equal
|
||||||
|
`stats["extracted"]` from the `--apply` run's summary line.
|
||||||
|
4. Confirm idempotency: re-running `--apply` immediately after should report
|
||||||
|
`extracted: 0` and `skipped_duplicate` equal to the previous run's
|
||||||
|
`extracted` count — nothing new lands in `consume/`.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e jobs/documents-ingest/
|
||||||
|
cd jobs/documents-ingest && pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
Pure unit tests, no DB or filesystem outside `tmp_path` required — `run()` is
|
||||||
|
tested by monkeypatching `asyncpg.connect` with an in-memory fake connection.
|
||||||
|
Covers: filename sanitization, consume-name collision handling, manifest
|
||||||
|
filtering, MIME PDF-part extraction (including the RFC 2047 decoding
|
||||||
|
mismatch), sha256 match/mismatch, duplicate detection, dry-run vs `--apply`
|
||||||
|
behavior, and multi-attachment envelopes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
146
kb/services/job-documents-ingest.md
Normal file
146
kb/services/job-documents-ingest.md
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
---
|
||||||
|
okf: "0.1"
|
||||||
|
type: service
|
||||||
|
visibility: private
|
||||||
|
status: active
|
||||||
|
updated: 2026-07-30
|
||||||
|
links:
|
||||||
|
- ../phases/kb-m5-documents-ingest-fazy.md
|
||||||
|
- ../runbooks/documents-ingest-run.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# documents-ingest
|
||||||
|
|
||||||
|
One-shot job CLI, Phase 1 of module 5 (`docs/kb/modules/05-documents-ingest.md`,
|
||||||
|
"Domkniecie dlugu z maili"). Extracts a **sample** of PDF attachments from the
|
||||||
|
Gmail `.eml` archive (already indexed in the `envelope` table of kb-postgres)
|
||||||
|
and drops them into Paperless' `consume/` directory so Paperless does the OCR
|
||||||
|
and correspondent-detection. This job does **not** write to the `envelope`
|
||||||
|
table — the Paperless/Nextcloud envelope adapter is a later phase of module 5.
|
||||||
|
|
||||||
|
## Why a sample, not a bulk import
|
||||||
|
|
||||||
|
The Gmail import left ~70k attachments referenced in `envelope.entities`
|
||||||
|
manifests (bytes live inside the archived `.eml` files, never extracted).
|
||||||
|
Dumping all of them into Paperless at once would swamp the OCR worker and the
|
||||||
|
RAG layer isn't built yet to make use of that volume. This job pulls a small,
|
||||||
|
recent, size-filtered sample (default: 150 envelopes, PDFs >50KB, from the
|
||||||
|
last year) as a testbed — mass import is a deliberate later decision.
|
||||||
|
|
||||||
|
## Where it runs
|
||||||
|
|
||||||
|
**Locally on PIHA**, as a plain CLI (not a container). It needs simultaneous
|
||||||
|
filesystem access to three things that all live on PIHA:
|
||||||
|
|
||||||
|
- the mail archive (`/home/oskar/kb/mail/archive`)
|
||||||
|
- the Paperless `consume/` directory (`/opt/homelab/data/paperless/consume`)
|
||||||
|
- kb-postgres (`localhost:5433` from PIHA; reachable from elsewhere over
|
||||||
|
Tailscale, but the archive and consume dir are not — those are local paths)
|
||||||
|
|
||||||
|
Install (from repo root, on PIHA):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -e jobs/documents-ingest/
|
||||||
|
```
|
||||||
|
|
||||||
|
(Or reuse the venv already set up for `gmail-bulk-import`, e.g.
|
||||||
|
`/home/oskar/kb/venv/` — it already has `asyncpg` + `structlog`.)
|
||||||
|
|
||||||
|
## Candidate selection
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT id, raw_ref, ts, entities FROM envelope
|
||||||
|
WHERE source = 'gmail'
|
||||||
|
AND ts > now() - interval '1 year'
|
||||||
|
AND EXISTS (
|
||||||
|
SELECT 1 FROM jsonb_array_elements(entities) AS att
|
||||||
|
WHERE att->>'content_type' = 'application/pdf'
|
||||||
|
AND (att->>'size')::numeric > 50000
|
||||||
|
)
|
||||||
|
ORDER BY ts DESC
|
||||||
|
LIMIT 150
|
||||||
|
```
|
||||||
|
|
||||||
|
For each matching envelope, every attachment manifest entry that passes the
|
||||||
|
filter is a separate candidate (one envelope can yield several PDFs).
|
||||||
|
|
||||||
|
## Matching an attachment inside the .eml
|
||||||
|
|
||||||
|
The manifest (`entities[]`) only has metadata — the attachment bytes live
|
||||||
|
inside the `.eml` (MIME multipart), so each candidate is resolved against the
|
||||||
|
freshly parsed message:
|
||||||
|
|
||||||
|
1. Parse the `.eml` with `email.policy.default` and collect every
|
||||||
|
`application/pdf` MIME part (filename + decoded payload).
|
||||||
|
2. **sha256 is the proof of identity**, not the filename. The manifest was
|
||||||
|
built by a different parser at import time (`gmail-bulk-import`, using
|
||||||
|
`mailbox` + compat32 policy) and can still hold the raw RFC 2047
|
||||||
|
encoded-word form of a filename (e.g. `=?UTF-8?b?...?=`, sometimes with
|
||||||
|
header-folding whitespace baked in), while `email.policy.default` decodes
|
||||||
|
it to real Unicode today. Comparing those byte-for-byte skipped ~10% of
|
||||||
|
otherwise-good attachments in testing — see `TestFindPdfParts` /
|
||||||
|
`TestProcessCandidate` in the test suite for the regression case. So:
|
||||||
|
match by sha256 across all PDF parts in the message; if none match, use a
|
||||||
|
filename match only to tell "found the named part but its bytes changed"
|
||||||
|
(`sha_mismatch`, reported and skipped) apart from "not present at all"
|
||||||
|
(`parse_error`, skipped).
|
||||||
|
3. The consume/ filename is built from the *decoded* filename (from the MIME
|
||||||
|
part), not the possibly-garbled manifest one.
|
||||||
|
|
||||||
|
Mismatches and parse errors are never guessed past — they're logged and
|
||||||
|
skipped.
|
||||||
|
|
||||||
|
## consume/ filenames
|
||||||
|
|
||||||
|
`<YYYY-MM-DD>_<sanitized-filename>.pdf`, date = envelope `ts`. On collision
|
||||||
|
(same date + sanitized name already used in this run or already present in
|
||||||
|
`consume/`), an 8-hex sha256 prefix is appended:
|
||||||
|
`<YYYY-MM-DD>_<sanitized-filename>_<hash8>.pdf`.
|
||||||
|
|
||||||
|
Files are written with a best-effort `chown` to uid:gid `1000:1000` (the
|
||||||
|
Paperless container's `USERMAP_UID/GID`, see `services/paperless/README.md`)
|
||||||
|
so Paperless can read them. If the chown fails (e.g. the job isn't running as
|
||||||
|
root/uid 1000), a warning is logged but the run continues — the write itself
|
||||||
|
already succeeded; fix ownership/perms on `consume/` separately if needed.
|
||||||
|
PIHA's uid/gid convention across the fleet is tracked as its own tech-debt
|
||||||
|
item (see `docs/backlog/`), not solved here.
|
||||||
|
|
||||||
|
## Idempotency — registry
|
||||||
|
|
||||||
|
A JSON file at `/opt/homelab/data/documents-ingest/registry.json` (default,
|
||||||
|
override with `--registry`), keyed by attachment sha256:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"<sha256>": {
|
||||||
|
"envelope_id": "...",
|
||||||
|
"filename": "...",
|
||||||
|
"consume_name": "2026-06-09_invoice.pdf",
|
||||||
|
"size": 123456,
|
||||||
|
"ingested_at": "2026-07-13T19:35:16+00:00"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why a JSON file and not a kb-postgres table:** this is a one-shot sampling
|
||||||
|
tool for a bootstrapping phase, not a long-running service — a new table
|
||||||
|
would formalize infrastructure for something temporary. A flat file needs no
|
||||||
|
migration, is trivial to inspect (`jq`) or reset, and sits under
|
||||||
|
`/opt/homelab/data/` alongside other node-local state per the repo's runtime
|
||||||
|
path convention. If/when module 5's real Paperless/Nextcloud adapter phase
|
||||||
|
starts writing `envelope` rows for `source=paperless`, that's the natural
|
||||||
|
point to fold this into a proper DB-backed ingest log — re-litigate then, not
|
||||||
|
now.
|
||||||
|
|
||||||
|
Re-running the job only ever *adds* to the registry (on `--apply`); it's
|
||||||
|
never consulted or mutated in dry-run mode beyond being read for the preview.
|
||||||
|
|
||||||
|
## Dry-run output
|
||||||
|
|
||||||
|
Logs one line per skip (`skip.duplicate` / `skip.sha_mismatch` /
|
||||||
|
`skip.parse_error`, with reason), a `summary` line with full counts
|
||||||
|
(`envelopes_scanned`, `pdf_candidates`, `extracted`, `skipped_duplicate`,
|
||||||
|
`skipped_sha_mismatch`, `skipped_parse_error`, `errors`), and up to 20 example
|
||||||
|
`(target_name, size, envelope_id)` rows so you can sanity-check filenames
|
||||||
|
before running `--apply`.
|
||||||
|
|
||||||
Loading…
Reference in a new issue