--- okf: "0.1" type: service visibility: private status: active updated: 2026-08-06 links: - ../runbooks/mail-sync-run.md - ../audits/mail-sync-2026-08-06.md - ../phases/kb-m5-faza-mailowa.md --- # mail-imap-sync Module 5, faza mailowa, Krok 7 (`kb/phases/kb-m5-faza-mailowa.md` §10), built to the recon `kb/audits/mail-sync-2026-08-06.md` after the operator approved decisions (a)-(g) on 2026-08-06. It is the first code in this repo that talks to a mail server. Before it, the mail corpus was a photograph: 225 030 envelopes ending at **2026-06-19**, imported once from a Gmail Takeout, with a gap that grew by roughly 37 messages a day. This job closes the gap and keeps it closed. ## Where it runs **On PIHA**, hourly, as a host-level systemd oneshot (`jobs/mail-imap-sync/systemd/`). The choice is not close (recon §3.1): PIHA is up 24/7, holds the canonical `.eml` archive, and runs kb-postgres locally. SOLARIA is powered off ~16 h a day by design, so a poller living there would read the mailbox only when the desktop happened to be on. Install (from repo root, on PIHA): ```bash pip install -e packages/kb-mail/ pip install -e jobs/mail-imap-sync/ ``` ## What one tick does Per account, per folder: 1. **EXAMINE** the folder — read-only. Never `SELECT`, and bodies come via `BODY.PEEK[]`, so the job never sets `\Seen` on the operator's own mail. 2. **Plan** the tick from `mail_sync_state` and the server's UIDVALIDITY/UIDNEXT — first contact, ordinary increment, or a UIDVALIDITY reset (see below). 3. **UID SEARCH** for the UIDs this tick owns. 4. Per UID, ascending: **FETCH** → `save_eml()` (append-only; `FileExistsError` just means "already have it") → `insert_envelope()` with `entities = [headers, attachment…]`. 5. **UPDATE `mail_sync_state`** — but only over the contiguous prefix of messages that are fully durable in both the archive and the DB. ## Folder scope (Decyzja (e)) | Account | Scope | Why | |---|---|---| | gmail | SPECIAL-USE `\All` | one fetch per message despite many labels; the same population as the Takeout the existing 225 030 envelopes came from, so the corpus stays continuous | | fastmail | `INBOX` + `Archive` + `Sent` | a classic IMAP layout has no `\All` equivalent | Spam, Trash and Drafts are excluded from both. Spam and Trash are defined noise and were not in the Takeout either; drafts are not correspondence, have no stable Message-ID, and mutate. The gmail folder is resolved **by SPECIAL-USE attribute, never by name**. Gmail localizes the mailbox — with a Polish UI it is `[Gmail]/Wszystkie` — so a hardcoded `[Gmail]/All Mail` would sync nothing while reporting a clean run every hour. Scope is per-account configuration (`MAIL__FOLDERS` / `_SPECIAL_USE`), so adding a folder later is a config change, not a code change; Message-ID dedup absorbs any overlap. ## The sync cursor `mail_sync_state` (migration `005`), keyed `(account, folder)`, holding `uidvalidity`, `last_uid` and `last_sync_ts`. A table rather than a file in `/opt/homelab/state/` (Decyzja (f)) for one decisive reason: **the cursor and the envelopes it describes must restore together or not at all.** A state file surviving a DB restore would make the poller silently skip everything between the restored rows and the file's `last_uid` — a failure with no symptom, noticed months later as missing mail. Deriving the cursor from `max(envelope.ts)` instead was rejected outright: `ts` is the sender's `Date:` header, not delivery time. 2 559 envelopes in the live corpus already sit at epoch 1970. ### UIDVALIDITY When the server changes UIDVALIDITY, every stored UID becomes meaningless. The job sweeps the folder (`UID SEARCH ALL`), leans on Message-ID dedup to make that cheap, and stores the new value. This is rare — and it is the failure that loses mail with no error anywhere if it is not handled. ### Why the cursor only crosses a contiguous prefix A message that fails is **not** stepped over. The cursor stops at the message before it, and the next tick refetches from there — free, because the archive write is append-only and the insert is `ON CONFLICT DO NOTHING`. Messages *after* the failure are still stored in the same tick; they simply do not move the cursor. The consequence is deliberate: a message that fails permanently **stalls its folder**. It does so visibly — a non-zero error counter every tick, a cursor that stops moving, the offending UID in the log — which is strictly better than the alternative of dropping mail while reporting success. The runbook documents the manual `UPDATE mail_sync_state` escape hatch for that case. ### The `n:*` trap `UID SEARCH UID n:*` is a **range**, and when `n` exceeds the highest existing UID the server resolves it as `highest:n` and returns the last message anyway. Without the client-side filter, "new mails this tick" would never read zero on an idle mailbox, and every observability claim built on that counter would be a small permanent lie. ## First contact with a folder `MAIL__INITIAL_MODE` decides what the first-ever tick on a folder fetches. It has no effect once a `mail_sync_state` row exists. | Mode | Behaviour | When | |---|---|---| | `new-only` | record `UIDNEXT-1`, fetch nothing | start the corpus from now | | `since` | `UID SEARCH SINCE ` (server INTERNALDATE, not the `Date:` header) | close a known gap | | `full` | `UID SEARCH ALL` | pull a mailbox's whole history | **gmail wants `since`.** A `new-only` first tick would leave the 48-day gap the recon measured unfilled forever — and that gap is the entire reason this job exists. Point `MAIL_GMAIL_INITIAL_SINCE` a few days before the newest envelope in the DB; dedup makes the overlap free. **fastmail is undecided by design.** The account is greenfield (zero rows, zero archive files), and whether to pull its history depends on how big it is — a number nobody has yet. Run `mail-imap-sync --measure` first (recon Decyzja (e) explicitly refuses to guess it). ## Message-ID collisions across accounts `envelope.id` is a bare Message-ID and therefore **globally unique in the table**, with no source prefix (unlike paperless, which uses `paperless:N`). A mail present in both mailboxes — a mailing list, a forward, a CC to both addresses — has the same Message-ID in both, so whoever inserts first sets `source` and the second is silently skipped by `ON CONFLICT DO NOTHING`. `save_eml` still writes both `.eml` copies, because the archive path contains the source; `raw_ref` points at one of them. This is not corruption and it is the desired index behaviour: the content is indexed once. It has two measurable effects worth knowing about **before** someone starts wondering about the numbers — "new fastmail mails" reads systematically low by the shared part, and `source` filtering in retrieval attributes such a mail to whichever account won the race. The recon's resolution (§2.4) is to keep the behaviour and make the phenomenon visible: `envelopes_conflict_other_source`, one extra query per suppressed insert. Re-keying to `(source, message_id)` was rejected — it breaks the frozen envelope contract, requires rewriting 225 030 ids and every `raw_ref`, and buys duplicates in the index. ## What this job does NOT do **It does not chunk and does not embed.** That split is the architecture, not an omission (recon §3.2): fetching is network-bound and belongs on the 24/7 node; chunking and embedding need Ollama on SOLARIA. Coupling them would mean mail only arrives when the desktop is on. The handoff needs no queue of its own — **an envelope with no `document_chunk` rows *is* the queue**, drained by `mail-body-ingest --only-unchunked` from the `kb-ingest` timer. That is self-healing in a way a timestamp cursor is not: an interrupted run, a mail fetched while the GPU slept, a message inserted with a backdated header — each stays pending until it has chunks, with nothing to reconcile. `--since` was explicitly rejected as the glue: it filters on `envelope.ts`, so a mail delivered today with a month-old header date would fall outside the window and never be chunked at all. ## Headers at INSERT time New envelopes carry `entities[type=headers]` immediately, via `kb_mail.headers.parse_headers_resilient` (typed parse, compat32 fallback). Historical envelopes only have headers because a separate backfill job walked all 225 030 of them later. Without this, `mail_body_ingest.build_prefix` would produce `Temat: (brak tematu) | Od: ? | Data: …` for every new mail — with no error, just permanently worse retrieval (recon §2.5 i). ## Secrets Environment only (Decyzja (c)). There is no `--password` or `--user` flag: `--dsn ` lands in `ps` output and shell history. Values go into the existing `/opt/homelab/kb/.env` (root-owned `0600`), which systemd reads **as root before dropping to `User=oskar`** — so the app passwords reach the process without being readable by `oskar` at rest. The repo ships only `jobs/mail-imap-sync/env.example` with placeholders. ## Stats must balance ``` uids_seen = processed + vanished + errors processed = archived + archive_exists processed = envelopes_inserted + envelopes_skipped_dup + envelopes_conflict_other_source ``` `headers_fallback`, `uidvalidity_resets`, `folder_errors` and `account_errors` are labels and diagnostics, deliberately outside the equations — a folder that failed on EXAMINE never produced a UID to account for. An unbalanced run is a failure regardless of what else it reports. ## Exit codes | Code | Meaning | |---|---| | 0 | Balanced, zero errors at any level | | 1 | Any message/folder/account error, or an unbalanced sum | | 2 | Configuration error — missing/malformed env, no DSN, no archive directory | Failures are isolated per account: a Gmail app password that stopped working must not also stop Fastmail from syncing (the same stage isolation `cyclic_ingest` uses). ## Modes | Invocation | Effect | |---|---| | *(default)* | Dry run: EXAMINE + SEARCH, report what a real tick would pull. No FETCH, no writes, no `.prom`. | | `--apply` | The real thing, and the only mode that publishes metrics. | | `--measure` | `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, then exit. Read-only, no DB. | | `--limit N` | At most N messages per folder per tick — a brake for a first big sweep; the rest follows next tick. | ## Metrics `/opt/homelab/state/node-exporter/kb-mail-sync.prom`, written atomically (tmp + rename) and scraped by node_exporter's textfile collector on PIHA. | Metric | Notes | |---|---| | `kb_mail_sync_last_run_timestamp` | every run | | `kb_mail_sync_last_success_timestamp` | carried forward across a failed run, so a blip does not reset the staleness clock | | `kb_mail_sync_last_exit_code` | | | `kb_mail_sync_envelopes_inserted{account}` | | | `kb_mail_sync_envelopes_skipped_dup{account}` | | | `kb_mail_sync_conflict_other_source{account}` | the cross-account overlap above | | `kb_mail_sync_uidvalidity_resets{account}` | | | `kb_mail_sync_errors{account}` | message + folder + account levels | | `kb_mail_sync_last_message_ts{account}` | `max(envelope.ts)`; omitted for an account with no rows | One alert, `KbMailSyncStale` (`services/fleet-prometheus/rules/kb-mail-sync.yml`): 6 h without a successful tick. **"No new mail for X days" was considered and rejected** (recon §3.4) — zero new mail is a legal state of a mailbox, an alert that fires on a healthy system gets muted, and a muted alert is not there on the day it matters. A per-account warning on `kb_mail_sync_last_message_ts` may be worth adding after a month of observation, with a threshold taken from measured behaviour rather than a guess. ## Definition of Done Per `CLAUDE.md`: `pytest` passes (80 for this job, 111 for `kb-mail` including the adapter and the cursor planner) plus a CLI smoke run. The tests drive the real `ImapClient` over a fake `imaplib` connection, so what is covered is the adapter's actual protocol parsing: new messages, UIDVALIDITY invalidation, dedup, resumption after an interrupted run, cross-account collisions, account isolation, expunged UIDs, a dry run that touches nothing, and the metric carry-forward. **Nothing here has ever connected to a live mailbox.** The first real sync is an operator step, supervised, per `kb/runbooks/mail-sync-run.md`. ## Follow-ups - `mail_imap_sync.prom` and `documents_ingest.cyclic_ingest`'s renderer are two copies of the same idea (this one supports labels, that one does not). Worth folding into one shared helper the next time either changes. - `kb-mail-sync.timer` joins the "shadow-deploy family" — units installed outside GitOps drift detection — listed in `hosts/piha/jobs.yaml`. That is open question 5 of `kb/subsystems/recon-multiagent.md`; the list is now one item longer, knowingly.