homelab-codex-ws/kb/services/job-mail-imap-sync.md

okf	type	visibility	status	updated	links
0.1	service	private	active	2026-08-06	../runbooks/mail-sync-run.md ../audits/mail-sync-2026-08-06.md ../phases/kb-m5-faza-mailowa.md

mail-imap-sync

Module 5, faza mailowa, Krok 7 (kb/phases/kb-m5-faza-mailowa.md §10), built to the recon kb/audits/mail-sync-2026-08-06.md after the operator approved decisions (a)-(g) on 2026-08-06. It is the first code in this repo that talks to a mail server.

Before it, the mail corpus was a photograph: 225 030 envelopes ending at 2026-06-19, imported once from a Gmail Takeout, with a gap that grew by roughly 37 messages a day. This job closes the gap and keeps it closed.

Where it runs

On PIHA, hourly, as a host-level systemd oneshot (jobs/mail-imap-sync/systemd/). The choice is not close (recon §3.1): PIHA is up 24/7, holds the canonical .eml archive, and runs kb-postgres locally. SOLARIA is powered off ~16 h a day by design, so a poller living there would read the mailbox only when the desktop happened to be on.

Install (from repo root, on PIHA):

pip install -e packages/kb-mail/
pip install -e jobs/mail-imap-sync/

What one tick does

Per account, per folder:

1. EXAMINE the folder — read-only. Never SELECT, and bodies come via BODY.PEEK[], so the job never sets \Seen on the operator's own mail.
2. Plan the tick from mail_sync_state and the server's UIDVALIDITY/UIDNEXT — first contact, ordinary increment, or a UIDVALIDITY reset (see below).
3. UID SEARCH for the UIDs this tick owns.
4. Per UID, ascending: FETCH → save_eml() (append-only; FileExistsError just means "already have it") → insert_envelope() with entities = [headers, attachment…].
5. UPDATE mail_sync_state — but only over the contiguous prefix of messages that are fully durable in both the archive and the DB.

Folder scope (Decyzja (e))

Account	Scope	Why
gmail	SPECIAL-USE \All	one fetch per message despite many labels; the same population as the Takeout the existing 225 030 envelopes came from, so the corpus stays continuous
fastmail	INBOX + Archive + Sent	a classic IMAP layout has no \All equivalent

Spam, Trash and Drafts are excluded from both. Spam and Trash are defined noise and were not in the Takeout either; drafts are not correspondence, have no stable Message-ID, and mutate.

The gmail folder is resolved by SPECIAL-USE attribute, never by name. Gmail localizes the mailbox — with a Polish UI it is [Gmail]/Wszystkie — so a hardcoded [Gmail]/All Mail would sync nothing while reporting a clean run every hour.

Scope is per-account configuration (MAIL_<ACCOUNT>_FOLDERS / _SPECIAL_USE), so adding a folder later is a config change, not a code change; Message-ID dedup absorbs any overlap.

The sync cursor

mail_sync_state (migration 005), keyed (account, folder), holding uidvalidity, last_uid and last_sync_ts. A table rather than a file in /opt/homelab/state/ (Decyzja (f)) for one decisive reason: the cursor and the envelopes it describes must restore together or not at all. A state file surviving a DB restore would make the poller silently skip everything between the restored rows and the file's last_uid — a failure with no symptom, noticed months later as missing mail.

Deriving the cursor from max(envelope.ts) instead was rejected outright: ts is the sender's Date: header, not delivery time. 2 559 envelopes in the live corpus already sit at epoch 1970.

UIDVALIDITY

When the server changes UIDVALIDITY, every stored UID becomes meaningless. The job sweeps the folder (UID SEARCH ALL), leans on Message-ID dedup to make that cheap, and stores the new value. This is rare — and it is the failure that loses mail with no error anywhere if it is not handled.

Why the cursor only crosses a contiguous prefix

A message that fails is not stepped over. The cursor stops at the message before it, and the next tick refetches from there — free, because the archive write is append-only and the insert is ON CONFLICT DO NOTHING. Messages after the failure are still stored in the same tick; they simply do not move the cursor.

The consequence is deliberate: a message that fails permanently stalls its folder. It does so visibly — a non-zero error counter every tick, a cursor that stops moving, the offending UID in the log — which is strictly better than the alternative of dropping mail while reporting success. The runbook documents the manual UPDATE mail_sync_state escape hatch for that case.

The n:* trap

UID SEARCH UID n:* is a range, and when n exceeds the highest existing UID the server resolves it as highest:n and returns the last message anyway. Without the client-side filter, "new mails this tick" would never read zero on an idle mailbox, and every observability claim built on that counter would be a small permanent lie.

First contact with a folder

MAIL_<ACCOUNT>_INITIAL_MODE decides what the first-ever tick on a folder fetches. It has no effect once a mail_sync_state row exists.

Mode	Behaviour	When
new-only	record UIDNEXT-1, fetch nothing	start the corpus from now
since	UID SEARCH SINCE <date> (server INTERNALDATE, not the Date: header)	close a known gap
full	UID SEARCH ALL	pull a mailbox's whole history

gmail wants since. A new-only first tick would leave the 48-day gap the recon measured unfilled forever — and that gap is the entire reason this job exists. Point MAIL_GMAIL_INITIAL_SINCE a few days before the newest envelope in the DB; dedup makes the overlap free.

fastmail is undecided by design. The account is greenfield (zero rows, zero archive files), and whether to pull its history depends on how big it is — a number nobody has yet. Run mail-imap-sync --measure first (recon Decyzja (e) explicitly refuses to guess it).

Message-ID collisions across accounts

envelope.id is a bare Message-ID and therefore globally unique in the table, with no source prefix (unlike paperless, which uses paperless:N). A mail present in both mailboxes — a mailing list, a forward, a CC to both addresses — has the same Message-ID in both, so whoever inserts first sets source and the second is silently skipped by ON CONFLICT DO NOTHING. save_eml still writes both .eml copies, because the archive path contains the source; raw_ref points at one of them.

This is not corruption and it is the desired index behaviour: the content is indexed once. It has two measurable effects worth knowing about before someone starts wondering about the numbers — "new fastmail mails" reads systematically low by the shared part, and source filtering in retrieval attributes such a mail to whichever account won the race.

The recon's resolution (§2.4) is to keep the behaviour and make the phenomenon visible: envelopes_conflict_other_source, one extra query per suppressed insert. Re-keying to (source, message_id) was rejected — it breaks the frozen envelope contract, requires rewriting 225 030 ids and every raw_ref, and buys duplicates in the index.

What this job does NOT do

It does not chunk and does not embed. That split is the architecture, not an omission (recon §3.2): fetching is network-bound and belongs on the 24/7 node; chunking and embedding need Ollama on SOLARIA. Coupling them would mean mail only arrives when the desktop is on.

The handoff needs no queue of its own — an envelope with no document_chunk rows is the queue, drained by mail-body-ingest --only-unchunked from the kb-ingest timer. That is self-healing in a way a timestamp cursor is not: an interrupted run, a mail fetched while the GPU slept, a message inserted with a backdated header — each stays pending until it has chunks, with nothing to reconcile.

--since was explicitly rejected as the glue: it filters on envelope.ts, so a mail delivered today with a month-old header date would fall outside the window and never be chunked at all.

Headers at INSERT time

New envelopes carry entities[type=headers] immediately, via kb_mail.headers.parse_headers_resilient (typed parse, compat32 fallback). Historical envelopes only have headers because a separate backfill job walked all 225 030 of them later. Without this, mail_body_ingest.build_prefix would produce Temat: (brak tematu) | Od: ? | Data: … for every new mail — with no error, just permanently worse retrieval (recon §2.5 i).

Secrets

Environment only (Decyzja (c)). There is no --password or --user flag: --dsn <secret> lands in ps output and shell history. Values go into the existing /opt/homelab/kb/.env (root-owned 0600), which systemd reads as root before dropping to User=oskar — so the app passwords reach the process without being readable by oskar at rest. The repo ships only jobs/mail-imap-sync/env.example with placeholders.

Stats must balance

uids_seen = processed + vanished + errors
processed = archived + archive_exists
processed = envelopes_inserted + envelopes_skipped_dup + envelopes_conflict_other_source

headers_fallback, uidvalidity_resets, folder_errors and account_errors are labels and diagnostics, deliberately outside the equations — a folder that failed on EXAMINE never produced a UID to account for. An unbalanced run is a failure regardless of what else it reports.

Exit codes

Code	Meaning
0	Balanced, zero errors at any level
1	Any message/folder/account error, or an unbalanced sum
2	Configuration error — missing/malformed env, no DSN, no archive directory

Failures are isolated per account: a Gmail app password that stopped working must not also stop Fastmail from syncing (the same stage isolation cyclic_ingest uses).

Modes

Invocation	Effect
(default)	Dry run: EXAMINE + SEARCH, report what a real tick would pull. No FETCH, no writes, no .prom.
--apply	The real thing, and the only mode that publishes metrics.
--measure	STATUS (MESSAGES UIDNEXT UIDVALIDITY) per folder, then exit. Read-only, no DB.
--limit N	At most N messages per folder per tick — a brake for a first big sweep; the rest follows next tick.

Metrics

/opt/homelab/state/node-exporter/kb-mail-sync.prom, written atomically (tmp + rename) and scraped by node_exporter's textfile collector on PIHA.

Metric	Notes
kb_mail_sync_last_run_timestamp	every run
kb_mail_sync_last_success_timestamp	carried forward across a failed run, so a blip does not reset the staleness clock
kb_mail_sync_last_exit_code
kb_mail_sync_envelopes_inserted{account}
kb_mail_sync_envelopes_skipped_dup{account}
kb_mail_sync_conflict_other_source{account}	the cross-account overlap above
kb_mail_sync_uidvalidity_resets{account}
kb_mail_sync_errors{account}	message + folder + account levels
kb_mail_sync_last_message_ts{account}	max(envelope.ts); omitted for an account with no rows

One alert, KbMailSyncStale (services/fleet-prometheus/rules/kb-mail-sync.yml): 6 h without a successful tick. "No new mail for X days" was considered and rejected (recon §3.4) — zero new mail is a legal state of a mailbox, an alert that fires on a healthy system gets muted, and a muted alert is not there on the day it matters. A per-account warning on kb_mail_sync_last_message_ts may be worth adding after a month of observation, with a threshold taken from measured behaviour rather than a guess.

Definition of Done

Per CLAUDE.md: pytest passes (80 for this job, 111 for kb-mail including the adapter and the cursor planner) plus a CLI smoke run. The tests drive the real ImapClient over a fake imaplib connection, so what is covered is the adapter's actual protocol parsing: new messages, UIDVALIDITY invalidation, dedup, resumption after an interrupted run, cross-account collisions, account isolation, expunged UIDs, a dry run that touches nothing, and the metric carry-forward.

Nothing here has ever connected to a live mailbox. The first real sync is an operator step, supervised, per kb/runbooks/mail-sync-run.md.

Follow-ups

• mail_imap_sync.prom and documents_ingest.cyclic_ingest's renderer are two copies of the same idea (this one supports labels, that one does not). Worth folding into one shared helper the next time either changes.
• kb-mail-sync.timer joins the "shadow-deploy family" — units installed outside GitOps drift detection — listed in hosts/piha/jobs.yaml. That is open question 5 of kb/subsystems/recon-multiagent.md; the list is now one item longer, knowingly.