oskar/homelab-codex-ws
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
homelab-codex-ws/kb/runbooks/documents-ingest-run.md
oskar 93485be1ae feat(kb): SPLIT documents-ingest (30 KB) -> service + phase + runbook 
kb/services/job-documents-ingest.md — opis jobu i mechanizmow
  (candidate selection, matching, consume/, idempotency, dry-run)
kb/phases/kb-m5-documents-ingest-fazy.md — faza 2, faza 2 krok 6,
  faza 3 krok 4, faza 3 krok 5 (4 sekcje fazowe wtopione w README)
kb/runbooks/documents-ingest-run.md — Usage, Verifying in Paperless, Tests

Najwiekszy README w repo. Tresc sekcji nietknieta; kontrola multizbioru
linii == oryginal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 16:19:31 +02:00

2.1 KiB
Raw Blame History

okf type visibility status updated links
0.1 runbook private active 2026-07-30
../services/job-documents-ingest.md
../phases/kb-m5-documents-ingest-fazy.md

documents-ingest — uruchomienie i weryfikacja

Usage

# Dry run (default) — preview only, no writes:
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb

# Or via env var instead of --dsn:
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
documents-ingest

# Real run — write files into consume/ and update the registry:
documents-ingest --apply

# Smaller/larger sample, different window/threshold:
documents-ingest --limit 50 --since-days 180 --min-size 100000

Dry-run is the default and does not require --consume-dir to exist yet; --apply does (Paperless must already be deployed with its consume dir in place). See documents-ingest --help for all flags.

Verifying the result in Paperless

After --apply:

  1. Paperless' consumer picks files up from consume/ automatically (polling or inotify, per its own config) — no action needed on this job's side.
  2. Watch progress: Paperless UI → Documents (new items appear as OCR finishes), or docker logs -f paperless on PIHA for consumer/OCR activity.
  3. Cross-check count: number of new documents in Paperless should equal stats["extracted"] from the --apply run's summary line.
  4. Confirm idempotency: re-running --apply immediately after should report extracted: 0 and skipped_duplicate equal to the previous run's extracted count — nothing new lands in consume/.

Tests

pip install -e jobs/documents-ingest/
cd jobs/documents-ingest && pytest

Pure unit tests, no DB or filesystem outside tmp_path required — run() is tested by monkeypatching asyncpg.connect with an in-memory fake connection. Covers: filename sanitization, consume-name collision handling, manifest filtering, MIME PDF-part extraction (including the RFC 2047 decoding mismatch), sha256 match/mismatch, duplicate detection, dry-run vs --apply behavior, and multi-attachment envelopes.