homelab-codex-ws/jobs/mail-imap-sync/env.example

72 lines
4 KiB
Plaintext
Raw Normal View History

feat(mail-imap-sync): job przyrostowki + wpiecie w tor body-ingest Nowy job jobs/mail-imap-sync — jedyny wlasciwy nowy kod przyrostowki (recon kb/audits/mail-sync-2026-08-06.md §6 poz. 1). Jeden tick, per konto i folder: EXAMINE -> plan -> UID SEARCH -> FETCH BODY.PEEK[] -> save_eml -> insert_envelope(entities=[headers, attachment...]) -> UPDATE mail_sync_state. Wlasciwosci, ktore latwo zgubic po cichu: - Job NIE embeduje i NIE chunkuje (recon §3.2). Pobieranie jest sieciowe i chodzi na PIHA 24/7; chunk+embed potrzebuje Ollamy na SOLARII, wylaczanej ~16 h/dobe. Spoiwem jest kolejka wynikajaca z danych: koperta bez chunkow JEST elementem kolejki, ktora drenuje mail-body-ingest --only-unchunked. - Koperta dostaje entities[type=headers] juz przy INSERCIE. Bez tego kazdy nowy mail mialby prefiks "Temat: (brak tematu) | Od: ?" — bez bledu, tylko z gorszym retrievalem (recon §2.5 i). - Kolizja Message-ID miedzy kontami jest liczona (envelopes_conflict_other_source), nie ukryta w zwyklych duplikatach (recon §2.4). - Kursor przesuwa sie tylko po nieprzerwanym ciagu w pelni trwalych wiadomosci. Bledna wiadomosc jest ponawiana (dedup czyni to darmowym), nie przeskakiwana; trwale zatrucie widac jako niezerowy licznik bledow i stojacy kursor. - Poswiadczenia wylacznie ze srodowiska — brak flagi --password/--user (Decyzja (c)). - Tryb --measure (STATUS MESSAGES/UIDNEXT/UIDVALIDITY): pomiar, na ktorym operator oprze decyzje o historii Fastmaila (Decyzja (e), celowo nieodgadywana). - Metryki .prom per konto; last_success_timestamp przenoszony przez nieudany run. Wpiecie w istniejacy tor (recon §6 poz. 4-6): - mail_body_ingest.fetch_envelopes: --sources (domyslnie gmail,fastmail) + --only-unchunked. - fetch_existing_chunk_keys zawezone do zbioru roboczego — bez tego kazdy tick czyta wszystkie 389 012 kluczy chunkow (~26 MB, ~1,0 s) na nodzie z 2,4 GB available (recon §2.5 iv). - DEFAULT_SUMMARYLESS_SOURCES += "fastmail" (Decyzja (g)) w TYM SAMYM commicie, ktory wprowadza zrodlo: bez tego koperty fastmail zaindeksowalyby sie poprawnie i byly niewidoczne w /search, bez zadnego bledu. Testy: 80 dla nowego joba (mock IMAP na poziomie imaplib, wiec testowane jest prawdziwe parsowanie protokolu) — nowe wiadomosci, uniewaznienie UIDVALIDITY, dedup, wznowienie po przerwaniu, izolacja kont, dry-run bez sieci, metryki; + 47 mail-body-ingest/kb-retrieval. Smoke: --help, blad konfiguracji -> exit 2. Zero polaczen z zywymi kontami — pierwszy zywy sync robi operator wg runbooka. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 14:37:32 +02:00
# mail-imap-sync configuration — placeholders only. NEVER commit real values.
#
# On PIHA these keys are APPENDED to the existing /opt/homelab/kb/.env, which is already
# root-owned 0600 and already holds KB_DSN, PAPERLESS_API_TOKEN and ANTHROPIC_API_KEY
# (recon Decyzja (c)). systemd reads EnvironmentFile as root BEFORE dropping to
# User=oskar, so the secrets reach the process without being readable by `oskar` at rest —
# a better property than a 0600 file owned by the job's own user. Keep it that way.
#
# Rules, all learned the hard way in this repo:
# 1. Never on the command line. This job has no --user/--password flag on purpose;
# argv shows up in `ps` and in shell history.
# 2. Never echoed to a terminal. Two secrets have already had to be rotated after
# leaking into a session transcript (docs/sessions/2026-07-15.md,
# docs/sessions/2026-07-21.md). Install them with an editor, not `echo >>`.
# 3. App passwords, not account passwords — revocable one at a time, no access to
# account settings, no bypassing 2FA.
#
# Full setup, including how to mint the app passwords: kb/runbooks/mail-sync-run.md
# --- which mailboxes to sync -------------------------------------------------------
# Required. Comma-separated; each name is also the envelope.source value and the
# mail_sync_state.account key. Start with one account if the other is not set up yet.
MAIL_ACCOUNTS=gmail,fastmail
# --- gmail -------------------------------------------------------------------------
# App password requires 2FA on the account. Paste it WITHOUT the spaces Google displays.
MAIL_GMAIL_USER=you@gmail.com
MAIL_GMAIL_APP_PASSWORD=change-me-16-char-app-password
# Optional, defaults shown. The folder is chosen by SPECIAL-USE attribute, never by name:
# with a Polish UI the mailbox is called "[Gmail]/Wszystkie", and a hardcoded
# "[Gmail]/All Mail" would sync nothing while reporting success every hour.
# MAIL_GMAIL_HOST=imap.gmail.com
# MAIL_GMAIL_PORT=993
# MAIL_GMAIL_SPECIAL_USE=\All
# MAIL_GMAIL_FOLDERS=
#
# What the FIRST tick on a folder does — new-only | since | full. Ignored once a
# mail_sync_state row exists for that folder.
# new-only : start the corpus here, fetch nothing older.
# since : fetch everything the server RECEIVED on/after MAIL_*_INITIAL_SINCE.
# full : sweep the whole folder (dedup absorbs what is already in the DB).
# For gmail, 'since' is the point of the exercise: the corpus stops at 2026-06-19, so a
# 'new-only' first tick would leave that gap unfilled forever. Pick a date a few days
# BEFORE the last envelope in the DB — dedup makes the overlap free.
MAIL_GMAIL_INITIAL_MODE=since
MAIL_GMAIL_INITIAL_SINCE=2026-06-15
# --- fastmail ----------------------------------------------------------------------
# Fastmail app passwords can be scoped to IMAP only — use that, not a full-access one.
MAIL_FASTMAIL_USER=you@fastmail.com
MAIL_FASTMAIL_APP_PASSWORD=change-me-app-password
# Optional, defaults shown. Fastmail has no \All equivalent, so the scope is a literal
# list. Spam/Trash/Drafts are excluded on purpose (defined noise; drafts have no stable
# Message-ID and mutate).
# MAIL_FASTMAIL_HOST=imap.fastmail.com
# MAIL_FASTMAIL_PORT=993
# MAIL_FASTMAIL_FOLDERS=INBOX,Archive,Sent
#
# Fastmail is greenfield — zero rows, zero archive files. Whether to pull its history is
# a decision the recon deliberately leaves open until measured:
# mail-imap-sync --measure
# prints MESSAGES per folder. Decide 'new-only' vs 'full' from that number, then set it
# here BEFORE the first --apply run (after the first run the cursor exists and this key
# no longer has any effect).
MAIL_FASTMAIL_INITIAL_MODE=new-only
# --- shared ------------------------------------------------------------------------
# Already present in /opt/homelab/kb/.env on PIHA — listed here for completeness.
# KB_DSN=postgresql://kb:CHANGE-ME@localhost:5433/kb
# MAIL_ARCHIVE_ROOT=/home/oskar/kb/mail/archive
# KB_MAIL_SYNC_PROM_PATH=/opt/homelab/state/node-exporter/kb-mail-sync.prom