f056b08574
Nowy job jobs/mail-imap-sync — jedyny wlasciwy nowy kod przyrostowki (recon kb/audits/mail-sync-2026-08-06.md §6 poz. 1). Jeden tick, per konto i folder: EXAMINE -> plan -> UID SEARCH -> FETCH BODY.PEEK[] -> save_eml -> insert_envelope(entities=[headers, attachment...]) -> UPDATE mail_sync_state. Wlasciwosci, ktore latwo zgubic po cichu: - Job NIE embeduje i NIE chunkuje (recon §3.2). Pobieranie jest sieciowe i chodzi na PIHA 24/7; chunk+embed potrzebuje Ollamy na SOLARII, wylaczanej ~16 h/dobe. Spoiwem jest kolejka wynikajaca z danych: koperta bez chunkow JEST elementem kolejki, ktora drenuje mail-body-ingest --only-unchunked. - Koperta dostaje entities[type=headers] juz przy INSERCIE. Bez tego kazdy nowy mail mialby prefiks "Temat: (brak tematu) | Od: ?" — bez bledu, tylko z gorszym retrievalem (recon §2.5 i). - Kolizja Message-ID miedzy kontami jest liczona (envelopes_conflict_other_source), nie ukryta w zwyklych duplikatach (recon §2.4). - Kursor przesuwa sie tylko po nieprzerwanym ciagu w pelni trwalych wiadomosci. Bledna wiadomosc jest ponawiana (dedup czyni to darmowym), nie przeskakiwana; trwale zatrucie widac jako niezerowy licznik bledow i stojacy kursor. - Poswiadczenia wylacznie ze srodowiska — brak flagi --password/--user (Decyzja (c)). - Tryb --measure (STATUS MESSAGES/UIDNEXT/UIDVALIDITY): pomiar, na ktorym operator oprze decyzje o historii Fastmaila (Decyzja (e), celowo nieodgadywana). - Metryki .prom per konto; last_success_timestamp przenoszony przez nieudany run. Wpiecie w istniejacy tor (recon §6 poz. 4-6): - mail_body_ingest.fetch_envelopes: --sources (domyslnie gmail,fastmail) + --only-unchunked. - fetch_existing_chunk_keys zawezone do zbioru roboczego — bez tego kazdy tick czyta wszystkie 389 012 kluczy chunkow (~26 MB, ~1,0 s) na nodzie z 2,4 GB available (recon §2.5 iv). - DEFAULT_SUMMARYLESS_SOURCES += "fastmail" (Decyzja (g)) w TYM SAMYM commicie, ktory wprowadza zrodlo: bez tego koperty fastmail zaindeksowalyby sie poprawnie i byly niewidoczne w /search, bez zadnego bledu. Testy: 80 dla nowego joba (mock IMAP na poziomie imaplib, wiec testowane jest prawdziwe parsowanie protokolu) — nowe wiadomosci, uniewaznienie UIDVALIDITY, dedup, wznowienie po przerwaniu, izolacja kont, dry-run bez sieci, metryki; + 47 mail-body-ingest/kb-retrieval. Smoke: --help, blad konfiguracji -> exit 2. Zero polaczen z zywymi kontami — pierwszy zywy sync robi operator wg runbooka. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72 lines
4 KiB
Plaintext
72 lines
4 KiB
Plaintext
# mail-imap-sync configuration — placeholders only. NEVER commit real values.
|
|
#
|
|
# On PIHA these keys are APPENDED to the existing /opt/homelab/kb/.env, which is already
|
|
# root-owned 0600 and already holds KB_DSN, PAPERLESS_API_TOKEN and ANTHROPIC_API_KEY
|
|
# (recon Decyzja (c)). systemd reads EnvironmentFile as root BEFORE dropping to
|
|
# User=oskar, so the secrets reach the process without being readable by `oskar` at rest —
|
|
# a better property than a 0600 file owned by the job's own user. Keep it that way.
|
|
#
|
|
# Rules, all learned the hard way in this repo:
|
|
# 1. Never on the command line. This job has no --user/--password flag on purpose;
|
|
# argv shows up in `ps` and in shell history.
|
|
# 2. Never echoed to a terminal. Two secrets have already had to be rotated after
|
|
# leaking into a session transcript (docs/sessions/2026-07-15.md,
|
|
# docs/sessions/2026-07-21.md). Install them with an editor, not `echo >>`.
|
|
# 3. App passwords, not account passwords — revocable one at a time, no access to
|
|
# account settings, no bypassing 2FA.
|
|
#
|
|
# Full setup, including how to mint the app passwords: kb/runbooks/mail-sync-run.md
|
|
|
|
# --- which mailboxes to sync -------------------------------------------------------
|
|
# Required. Comma-separated; each name is also the envelope.source value and the
|
|
# mail_sync_state.account key. Start with one account if the other is not set up yet.
|
|
MAIL_ACCOUNTS=gmail,fastmail
|
|
|
|
# --- gmail -------------------------------------------------------------------------
|
|
# App password requires 2FA on the account. Paste it WITHOUT the spaces Google displays.
|
|
MAIL_GMAIL_USER=you@gmail.com
|
|
MAIL_GMAIL_APP_PASSWORD=change-me-16-char-app-password
|
|
# Optional, defaults shown. The folder is chosen by SPECIAL-USE attribute, never by name:
|
|
# with a Polish UI the mailbox is called "[Gmail]/Wszystkie", and a hardcoded
|
|
# "[Gmail]/All Mail" would sync nothing while reporting success every hour.
|
|
# MAIL_GMAIL_HOST=imap.gmail.com
|
|
# MAIL_GMAIL_PORT=993
|
|
# MAIL_GMAIL_SPECIAL_USE=\All
|
|
# MAIL_GMAIL_FOLDERS=
|
|
#
|
|
# What the FIRST tick on a folder does — new-only | since | full. Ignored once a
|
|
# mail_sync_state row exists for that folder.
|
|
# new-only : start the corpus here, fetch nothing older.
|
|
# since : fetch everything the server RECEIVED on/after MAIL_*_INITIAL_SINCE.
|
|
# full : sweep the whole folder (dedup absorbs what is already in the DB).
|
|
# For gmail, 'since' is the point of the exercise: the corpus stops at 2026-06-19, so a
|
|
# 'new-only' first tick would leave that gap unfilled forever. Pick a date a few days
|
|
# BEFORE the last envelope in the DB — dedup makes the overlap free.
|
|
MAIL_GMAIL_INITIAL_MODE=since
|
|
MAIL_GMAIL_INITIAL_SINCE=2026-06-15
|
|
|
|
# --- fastmail ----------------------------------------------------------------------
|
|
# Fastmail app passwords can be scoped to IMAP only — use that, not a full-access one.
|
|
MAIL_FASTMAIL_USER=you@fastmail.com
|
|
MAIL_FASTMAIL_APP_PASSWORD=change-me-app-password
|
|
# Optional, defaults shown. Fastmail has no \All equivalent, so the scope is a literal
|
|
# list. Spam/Trash/Drafts are excluded on purpose (defined noise; drafts have no stable
|
|
# Message-ID and mutate).
|
|
# MAIL_FASTMAIL_HOST=imap.fastmail.com
|
|
# MAIL_FASTMAIL_PORT=993
|
|
# MAIL_FASTMAIL_FOLDERS=INBOX,Archive,Sent
|
|
#
|
|
# Fastmail is greenfield — zero rows, zero archive files. Whether to pull its history is
|
|
# a decision the recon deliberately leaves open until measured:
|
|
# mail-imap-sync --measure
|
|
# prints MESSAGES per folder. Decide 'new-only' vs 'full' from that number, then set it
|
|
# here BEFORE the first --apply run (after the first run the cursor exists and this key
|
|
# no longer has any effect).
|
|
MAIL_FASTMAIL_INITIAL_MODE=new-only
|
|
|
|
# --- shared ------------------------------------------------------------------------
|
|
# Already present in /opt/homelab/kb/.env on PIHA — listed here for completeness.
|
|
# KB_DSN=postgresql://kb:CHANGE-ME@localhost:5433/kb
|
|
# MAIL_ARCHIVE_ROOT=/home/oskar/kb/mail/archive
|
|
# KB_MAIL_SYNC_PROM_PATH=/opt/homelab/state/node-exporter/kb-mail-sync.prom
|