ccb738cddf
kb/services/job-documents-ingest.md — opis jobu i mechanizmow (candidate selection, matching, consume/, idempotency, dry-run) kb/phases/kb-m5-documents-ingest-fazy.md — faza 2, faza 2 krok 6, faza 3 krok 4, faza 3 krok 5 (4 sekcje fazowe wtopione w README) kb/runbooks/documents-ingest-run.md — Usage, Verifying in Paperless, Tests Najwiekszy README w repo. Tresc sekcji nietknieta; kontrola multizbioru linii == oryginal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
65 lines
2.1 KiB
Markdown
65 lines
2.1 KiB
Markdown
---
|
|
okf: "0.1"
|
|
type: runbook
|
|
visibility: private
|
|
status: active
|
|
updated: 2026-07-30
|
|
links:
|
|
- ../services/job-documents-ingest.md
|
|
- ../phases/kb-m5-documents-ingest-fazy.md
|
|
---
|
|
|
|
# documents-ingest — uruchomienie i weryfikacja
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
# Dry run (default) — preview only, no writes:
|
|
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb
|
|
|
|
# Or via env var instead of --dsn:
|
|
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
|
|
documents-ingest
|
|
|
|
# Real run — write files into consume/ and update the registry:
|
|
documents-ingest --apply
|
|
|
|
# Smaller/larger sample, different window/threshold:
|
|
documents-ingest --limit 50 --since-days 180 --min-size 100000
|
|
```
|
|
|
|
Dry-run is the default and does not require `--consume-dir` to exist yet;
|
|
`--apply` does (Paperless must already be deployed with its consume dir in
|
|
place). See `documents-ingest --help` for all flags.
|
|
|
|
## Verifying the result in Paperless
|
|
|
|
After `--apply`:
|
|
|
|
1. Paperless' consumer picks files up from `consume/` automatically (polling
|
|
or inotify, per its own config) — no action needed on this job's side.
|
|
2. Watch progress: Paperless UI → Documents (new items appear as OCR
|
|
finishes), or `docker logs -f paperless` on PIHA for consumer/OCR activity.
|
|
3. Cross-check count: number of new documents in Paperless should equal
|
|
`stats["extracted"]` from the `--apply` run's summary line.
|
|
4. Confirm idempotency: re-running `--apply` immediately after should report
|
|
`extracted: 0` and `skipped_duplicate` equal to the previous run's
|
|
`extracted` count — nothing new lands in `consume/`.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
pip install -e jobs/documents-ingest/
|
|
cd jobs/documents-ingest && pytest
|
|
```
|
|
|
|
Pure unit tests, no DB or filesystem outside `tmp_path` required — `run()` is
|
|
tested by monkeypatching `asyncpg.connect` with an in-memory fake connection.
|
|
Covers: filename sanitization, consume-name collision handling, manifest
|
|
filtering, MIME PDF-part extraction (including the RFC 2047 decoding
|
|
mismatch), sha256 match/mismatch, duplicate detection, dry-run vs `--apply`
|
|
behavior, and multi-attachment envelopes.
|
|
|
|
---
|
|
|