oskar/homelab-codex-ws
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
homelab-codex-ws/jobs/documents-ingest/pyproject.toml

26 lines
528 B
TOML
Raw Normal View History

feat(documents-ingest): PDF attachment extractor — mail archive -> Paperless consume/ Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive and drops them into Paperless consume/ for OCR, so the RAG layer has real documents to work with before the full Paperless/Nextcloud envelope adapter is built. sha256-first attachment matching (manifest filenames can carry raw RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default decodes today — confirmed against live data, ~10% of candidates were affected). Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write. Verified end-to-end on PIHA: dry-run + --apply both run against live kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run duplicates correctly deduped), Paperless picked them up and started OCR immediately.
2026-07-13 20:00:00 +02:00
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "documents-ingest"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
"asyncpg>=0.29",
"structlog>=24.1",
feat(documents-ingest): Paperless -> envelope adapter (module 5 phase 2 step 5) Adds documents-ingest-paperless: paginated GET /api/documents/, maps each doc to a source='paperless' envelope per plan §4.2-4.3, reusing kb_mail.Envelope/insert_envelope unchanged (packages/kb-mail not touched). Cross-source link (source_mail entity) is a deterministic join of original_file_name against the phase-1 registry.json consume_name index — no heuristics, no correspondent guessing (plan decision 4). Stats always balance (fetched = already_in_db + inserted + errors) and main() now also exits non-zero on imbalance, not just on errors>0, matching the exit-code convention already established in gmail-bulk-import. Verified live on PIHA (rsync to /tmp, ~/kb/venv, PIHA checkout untouched): dry-run then --apply inserted 186/186 paperless envelopes (0 errors, 180 source_mail links), a second --apply reported inserted=0/already_in_db=186 (idempotent), gmail rows stayed at 225030 and document_chunk stayed empty. Rotated the kb-ingest Paperless API token after it was accidentally partially echoed during recon (old token now dead). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:06:56 +02:00
"aiohttp>=3.9",
"kb-mail",
feat(documents-ingest): PDF attachment extractor — mail archive -> Paperless consume/ Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive and drops them into Paperless consume/ for OCR, so the RAG layer has real documents to work with before the full Paperless/Nextcloud envelope adapter is built. sha256-first attachment matching (manifest filenames can carry raw RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default decodes today — confirmed against live data, ~10% of candidates were affected). Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write. Verified end-to-end on PIHA: dry-run + --apply both run against live kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run duplicates correctly deduped), Paperless picked them up and started OCR immediately.
2026-07-13 20:00:00 +02:00
]
[project.scripts]
documents-ingest = "documents_ingest.extractor:main"
feat(documents-ingest): Paperless -> envelope adapter (module 5 phase 2 step 5) Adds documents-ingest-paperless: paginated GET /api/documents/, maps each doc to a source='paperless' envelope per plan §4.2-4.3, reusing kb_mail.Envelope/insert_envelope unchanged (packages/kb-mail not touched). Cross-source link (source_mail entity) is a deterministic join of original_file_name against the phase-1 registry.json consume_name index — no heuristics, no correspondent guessing (plan decision 4). Stats always balance (fetched = already_in_db + inserted + errors) and main() now also exits non-zero on imbalance, not just on errors>0, matching the exit-code convention already established in gmail-bulk-import. Verified live on PIHA (rsync to /tmp, ~/kb/venv, PIHA checkout untouched): dry-run then --apply inserted 186/186 paperless envelopes (0 errors, 180 source_mail links), a second --apply reported inserted=0/already_in_db=186 (idempotent), gmail rows stayed at 225030 and document_chunk stayed empty. Rotated the kb-ingest Paperless API token after it was accidentally partially echoed during recon (old token now dead). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-15 15:06:56 +02:00
documents-ingest-paperless = "documents_ingest.paperless_adapter:main"
feat(documents-ingest): PDF attachment extractor — mail archive -> Paperless consume/ Module 5 phase 1 (docs/kb/modules/05-documents-ingest.md): pulls a sample of PDF attachments (>50KB, last year, LIMIT 150) out of the Gmail .eml archive and drops them into Paperless consume/ for OCR, so the RAG layer has real documents to work with before the full Paperless/Nextcloud envelope adapter is built. sha256-first attachment matching (manifest filenames can carry raw RFC 2047 encoded-word artifacts that don't byte-match what email.policy.default decodes today — confirmed against live data, ~10% of candidates were affected). Idempotent via a sha256-keyed JSON registry; dry-run by default, --apply to write. Verified end-to-end on PIHA: dry-run + --apply both run against live kb-postgres/archive, 185/222 candidate PDFs written to consume/ (37 in-run duplicates correctly deduped), Paperless picked them up and started OCR immediately.
2026-07-13 20:00:00 +02:00
[tool.setuptools.packages.find]
where = ["src"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
Reference in a new issue Copy permalink