65 lines
2.1 KiB
Markdown
65 lines
2.1 KiB
Markdown
|
---
|
||
|
|
okf: "0.1"
|
||
|
|
type: runbook
|
||
|
|
visibility: private
|
||
|
|
status: active
|
||
|
|
updated: 2026-07-30
|
||
|
|
links:
|
||
|
|
- ../services/job-documents-ingest.md
|
||
|
|
- ../phases/kb-m5-documents-ingest-fazy.md
|
||
|
|
---
|
||
|
|
|
||
|
|
# documents-ingest — uruchomienie i weryfikacja
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Dry run (default) — preview only, no writes:
|
||
|
|
documents-ingest --dsn postgresql://kb:<pw>@localhost:5433/kb
|
||
|
|
|
||
|
|
# Or via env var instead of --dsn:
|
||
|
|
export KB_DSN=postgresql://kb:<pw>@localhost:5433/kb
|
||
|
|
documents-ingest
|
||
|
|
|
||
|
|
# Real run — write files into consume/ and update the registry:
|
||
|
|
documents-ingest --apply
|
||
|
|
|
||
|
|
# Smaller/larger sample, different window/threshold:
|
||
|
|
documents-ingest --limit 50 --since-days 180 --min-size 100000
|
||
|
|
```
|
||
|
|
|
||
|
|
Dry-run is the default and does not require `--consume-dir` to exist yet;
|
||
|
|
`--apply` does (Paperless must already be deployed with its consume dir in
|
||
|
|
place). See `documents-ingest --help` for all flags.
|
||
|
|
|
||
|
|
## Verifying the result in Paperless
|
||
|
|
|
||
|
|
After `--apply`:
|
||
|
|
|
||
|
|
1. Paperless' consumer picks files up from `consume/` automatically (polling
|
||
|
|
or inotify, per its own config) — no action needed on this job's side.
|
||
|
|
2. Watch progress: Paperless UI → Documents (new items appear as OCR
|
||
|
|
finishes), or `docker logs -f paperless` on PIHA for consumer/OCR activity.
|
||
|
|
3. Cross-check count: number of new documents in Paperless should equal
|
||
|
|
`stats["extracted"]` from the `--apply` run's summary line.
|
||
|
|
4. Confirm idempotency: re-running `--apply` immediately after should report
|
||
|
|
`extracted: 0` and `skipped_duplicate` equal to the previous run's
|
||
|
|
`extracted` count — nothing new lands in `consume/`.
|
||
|
|
|
||
|
|
## Tests
|
||
|
|
|
||
|
|
```bash
|
||
|
|
pip install -e jobs/documents-ingest/
|
||
|
|
cd jobs/documents-ingest && pytest
|
||
|
|
```
|
||
|
|
|
||
|
|
Pure unit tests, no DB or filesystem outside `tmp_path` required — `run()` is
|
||
|
|
tested by monkeypatching `asyncpg.connect` with an in-memory fake connection.
|
||
|
|
Covers: filename sanitization, consume-name collision handling, manifest
|
||
|
|
filtering, MIME PDF-part extraction (including the RFC 2047 decoding
|
||
|
|
mismatch), sha256 match/mismatch, duplicate detection, dry-run vs `--apply`
|
||
|
|
behavior, and multi-attachment envelopes.
|
||
|
|
|
||
|
|
---
|
||
|
|
|