ae16deb8d3
Domkniecie Kroku 7. Realizuje Decyzje (d) reconu (host schedulera + korekta
kadencji indeksowania) i doklada dokumentacje wg konwencji OKF.
Scheduler (NIEAKTYWOWANY — wlacza operator):
- jobs/mail-imap-sync/systemd/{service,timer,run.sh} — wzorzec 1:1 z kb-ingest,
OnCalendar=hourly, Persistent=true, log do pliku (nigdy sam journal).
- hosts/piha/jobs.yaml — deklaracja jednostek host-level na PIHA. Nowy plik, bo
services.yaml jest dla kontenerow (supervisor dopasowuje jego wpisy do world-state
i wpis niekontenerowy dryfowalby wiecznie jako missing_service). Nic tego pliku
nie czyta — istnieje po to, zeby "shadow-deploy family" z otwartego pytania 5
reconu multiagentowego byla spisana, a nie tylko na nodzie.
Takt indeksowania (Decyzja (d), recon §3.3):
- kb-ingest.timer: 03:30 raz na dobe -> co 2 h. O 03:30 SOLARIA prawie na pewno spi
(potwierdzone odczytem kb_ingest_embed_skipped 1 z 2026-08-06), a tick dostaje
teraz etap mailowy: ~60 nowych chunkow na dobe pomijanych kazdej nocy sprawiloby,
ze backlog rosnie monotonicznie i KbEmbedBacklogGrowing zapala sie NA STALE.
Co 2 h zamiast stalej godziny — probe Ollamy sam wybiera okno, wiec ktorys tick
w nie trafi niezaleznie od nawykow operatora.
- cyclic_ingest: etap mailowy (mail_body_ingest --only-unchunked), import miekki,
wiec venv bez tego pakietu pomija etap zamiast wywracac wrapper. Predykat bledu
JEST luzniejszy niz wlasne main() tamtego joba i to jedyne takie miejsce w tym
wrapperze: pojedynczy trwale nieparsowalny mail nie moze zamrozic
last_success_timestamp i zapalic KbIngestStale na zawsze. Bledy per-mail sa
publikowane jako kb_ingest_mail_parse_errors, nie chowane.
Obserwowalnosc: KbMailSyncStale (6 h bez udanego ticku). Alert na cisze w skrzynce
ODRZUCONY (decyzja operatora, zgodna z reconem §3.4) — zero nowych maili to legalny
stan skrzynki, a alert zapalajacy sie na zdrowym systemie zostaje wyciszony
i przestaje dzialac wtedy, gdy jest potrzebny.
Dokumentacja:
- kb/services/job-mail-imap-sync.md (OKF), kb/runbooks/mail-sync-run.md — 9 krokow
pierwszego uruchomienia, w tym checklista 4 punktow [do weryfikacji na zywo]
z reconu (polityki dostawcow — do sprawdzenia, nie do zgadniecia) oraz pomiar
STATUS (MESSAGES) na Fastmailu, na ktorym zapada ODLOZONA decyzja o historii.
- kb-mail-pillar.md: KOREKTA JMAP -> IMAP dla Fastmaila jako decyzja 2026-08-06;
stary zapis zostaje jako historia z data. Zamkniete "unifikacja adaptera"
i "sizing Gmaila"; otwarte zostaje "sizing Fastmaila" — celowo, bo rozstrzyga
je pomiar, nie dyskusja.
- kb-m5-faza-mailowa.md: Krok 7 IN PROGRESS + tabela zakresu wdrozonego,
kb-m5-faza3.md: korekta harmonogramu i sekwencji wrappera,
pkg-kb-mail.md: rozpisany ze stubu, kb-postgres.md: lista migracji + 005.
Testy: 642 passed (calosc kb-mail, kb-retrieval i jobs). systemd-analyze verify
na timerze przechodzi, OnCalendar=0/2:00:00 normalizuje sie do co 2 h.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
257 lines
13 KiB
Markdown
257 lines
13 KiB
Markdown
---
|
|
okf: "0.1"
|
|
type: service
|
|
visibility: private
|
|
status: active
|
|
updated: 2026-08-06
|
|
links:
|
|
- ../runbooks/mail-sync-run.md
|
|
- ../audits/mail-sync-2026-08-06.md
|
|
- ../phases/kb-m5-faza-mailowa.md
|
|
---
|
|
|
|
# mail-imap-sync
|
|
|
|
Module 5, faza mailowa, Krok 7 (`kb/phases/kb-m5-faza-mailowa.md` §10), built to the recon
|
|
`kb/audits/mail-sync-2026-08-06.md` after the operator approved decisions (a)-(g) on
|
|
2026-08-06. It is the first code in this repo that talks to a mail server.
|
|
|
|
Before it, the mail corpus was a photograph: 225 030 envelopes ending at **2026-06-19**,
|
|
imported once from a Gmail Takeout, with a gap that grew by roughly 37 messages a day. This
|
|
job closes the gap and keeps it closed.
|
|
|
|
## Where it runs
|
|
|
|
**On PIHA**, hourly, as a host-level systemd oneshot (`jobs/mail-imap-sync/systemd/`). The
|
|
choice is not close (recon §3.1): PIHA is up 24/7, holds the canonical `.eml` archive, and
|
|
runs kb-postgres locally. SOLARIA is powered off ~16 h a day by design, so a poller living
|
|
there would read the mailbox only when the desktop happened to be on.
|
|
|
|
Install (from repo root, on PIHA):
|
|
|
|
```bash
|
|
pip install -e packages/kb-mail/
|
|
pip install -e jobs/mail-imap-sync/
|
|
```
|
|
|
|
## What one tick does
|
|
|
|
Per account, per folder:
|
|
|
|
1. **EXAMINE** the folder — read-only. Never `SELECT`, and bodies come via `BODY.PEEK[]`, so
|
|
the job never sets `\Seen` on the operator's own mail.
|
|
2. **Plan** the tick from `mail_sync_state` and the server's UIDVALIDITY/UIDNEXT — first
|
|
contact, ordinary increment, or a UIDVALIDITY reset (see below).
|
|
3. **UID SEARCH** for the UIDs this tick owns.
|
|
4. Per UID, ascending: **FETCH** → `save_eml()` (append-only; `FileExistsError` just means
|
|
"already have it") → `insert_envelope()` with `entities = [headers, attachment…]`.
|
|
5. **UPDATE `mail_sync_state`** — but only over the contiguous prefix of messages that are
|
|
fully durable in both the archive and the DB.
|
|
|
|
## Folder scope (Decyzja (e))
|
|
|
|
| Account | Scope | Why |
|
|
|---|---|---|
|
|
| gmail | SPECIAL-USE `\All` | one fetch per message despite many labels; the same population as the Takeout the existing 225 030 envelopes came from, so the corpus stays continuous |
|
|
| fastmail | `INBOX` + `Archive` + `Sent` | a classic IMAP layout has no `\All` equivalent |
|
|
|
|
Spam, Trash and Drafts are excluded from both. Spam and Trash are defined noise and were not
|
|
in the Takeout either; drafts are not correspondence, have no stable Message-ID, and mutate.
|
|
|
|
The gmail folder is resolved **by SPECIAL-USE attribute, never by name**. Gmail localizes the
|
|
mailbox — with a Polish UI it is `[Gmail]/Wszystkie` — so a hardcoded `[Gmail]/All Mail` would
|
|
sync nothing while reporting a clean run every hour.
|
|
|
|
Scope is per-account configuration (`MAIL_<ACCOUNT>_FOLDERS` / `_SPECIAL_USE`), so adding a
|
|
folder later is a config change, not a code change; Message-ID dedup absorbs any overlap.
|
|
|
|
## The sync cursor
|
|
|
|
`mail_sync_state` (migration `005`), keyed `(account, folder)`, holding `uidvalidity`,
|
|
`last_uid` and `last_sync_ts`. A table rather than a file in `/opt/homelab/state/`
|
|
(Decyzja (f)) for one decisive reason: **the cursor and the envelopes it describes must
|
|
restore together or not at all.** A state file surviving a DB restore would make the poller
|
|
silently skip everything between the restored rows and the file's `last_uid` — a failure with
|
|
no symptom, noticed months later as missing mail.
|
|
|
|
Deriving the cursor from `max(envelope.ts)` instead was rejected outright: `ts` is the
|
|
sender's `Date:` header, not delivery time. 2 559 envelopes in the live corpus already sit at
|
|
epoch 1970.
|
|
|
|
### UIDVALIDITY
|
|
|
|
When the server changes UIDVALIDITY, every stored UID becomes meaningless. The job sweeps the
|
|
folder (`UID SEARCH ALL`), leans on Message-ID dedup to make that cheap, and stores the new
|
|
value. This is rare — and it is the failure that loses mail with no error anywhere if it is
|
|
not handled.
|
|
|
|
### Why the cursor only crosses a contiguous prefix
|
|
|
|
A message that fails is **not** stepped over. The cursor stops at the message before it, and
|
|
the next tick refetches from there — free, because the archive write is append-only and the
|
|
insert is `ON CONFLICT DO NOTHING`. Messages *after* the failure are still stored in the same
|
|
tick; they simply do not move the cursor.
|
|
|
|
The consequence is deliberate: a message that fails permanently **stalls its folder**. It
|
|
does so visibly — a non-zero error counter every tick, a cursor that stops moving, the
|
|
offending UID in the log — which is strictly better than the alternative of dropping mail
|
|
while reporting success. The runbook documents the manual `UPDATE mail_sync_state` escape
|
|
hatch for that case.
|
|
|
|
### The `n:*` trap
|
|
|
|
`UID SEARCH UID n:*` is a **range**, and when `n` exceeds the highest existing UID the server
|
|
resolves it as `highest:n` and returns the last message anyway. Without the client-side
|
|
filter, "new mails this tick" would never read zero on an idle mailbox, and every
|
|
observability claim built on that counter would be a small permanent lie.
|
|
|
|
## First contact with a folder
|
|
|
|
`MAIL_<ACCOUNT>_INITIAL_MODE` decides what the first-ever tick on a folder fetches. It has no
|
|
effect once a `mail_sync_state` row exists.
|
|
|
|
| Mode | Behaviour | When |
|
|
|---|---|---|
|
|
| `new-only` | record `UIDNEXT-1`, fetch nothing | start the corpus from now |
|
|
| `since` | `UID SEARCH SINCE <date>` (server INTERNALDATE, not the `Date:` header) | close a known gap |
|
|
| `full` | `UID SEARCH ALL` | pull a mailbox's whole history |
|
|
|
|
**gmail wants `since`.** A `new-only` first tick would leave the 48-day gap the recon measured
|
|
unfilled forever — and that gap is the entire reason this job exists. Point
|
|
`MAIL_GMAIL_INITIAL_SINCE` a few days before the newest envelope in the DB; dedup makes the
|
|
overlap free.
|
|
|
|
**fastmail is undecided by design.** The account is greenfield (zero rows, zero archive
|
|
files), and whether to pull its history depends on how big it is — a number nobody has yet.
|
|
Run `mail-imap-sync --measure` first (recon Decyzja (e) explicitly refuses to guess it).
|
|
|
|
## Message-ID collisions across accounts
|
|
|
|
`envelope.id` is a bare Message-ID and therefore **globally unique in the table**, with no
|
|
source prefix (unlike paperless, which uses `paperless:N`). A mail present in both mailboxes —
|
|
a mailing list, a forward, a CC to both addresses — has the same Message-ID in both, so
|
|
whoever inserts first sets `source` and the second is silently skipped by
|
|
`ON CONFLICT DO NOTHING`. `save_eml` still writes both `.eml` copies, because the archive path
|
|
contains the source; `raw_ref` points at one of them.
|
|
|
|
This is not corruption and it is the desired index behaviour: the content is indexed once.
|
|
It has two measurable effects worth knowing about **before** someone starts wondering about
|
|
the numbers — "new fastmail mails" reads systematically low by the shared part, and
|
|
`source` filtering in retrieval attributes such a mail to whichever account won the race.
|
|
|
|
The recon's resolution (§2.4) is to keep the behaviour and make the phenomenon visible:
|
|
`envelopes_conflict_other_source`, one extra query per suppressed insert. Re-keying to
|
|
`(source, message_id)` was rejected — it breaks the frozen envelope contract, requires
|
|
rewriting 225 030 ids and every `raw_ref`, and buys duplicates in the index.
|
|
|
|
## What this job does NOT do
|
|
|
|
**It does not chunk and does not embed.** That split is the architecture, not an omission
|
|
(recon §3.2): fetching is network-bound and belongs on the 24/7 node; chunking and embedding
|
|
need Ollama on SOLARIA. Coupling them would mean mail only arrives when the desktop is on.
|
|
|
|
The handoff needs no queue of its own — **an envelope with no `document_chunk` rows *is* the
|
|
queue**, drained by `mail-body-ingest --only-unchunked` from the `kb-ingest` timer. That is
|
|
self-healing in a way a timestamp cursor is not: an interrupted run, a mail fetched while the
|
|
GPU slept, a message inserted with a backdated header — each stays pending until it has
|
|
chunks, with nothing to reconcile.
|
|
|
|
`--since` was explicitly rejected as the glue: it filters on `envelope.ts`, so a mail
|
|
delivered today with a month-old header date would fall outside the window and never be
|
|
chunked at all.
|
|
|
|
## Headers at INSERT time
|
|
|
|
New envelopes carry `entities[type=headers]` immediately, via
|
|
`kb_mail.headers.parse_headers_resilient` (typed parse, compat32 fallback). Historical
|
|
envelopes only have headers because a separate backfill job walked all 225 030 of them later.
|
|
Without this, `mail_body_ingest.build_prefix` would produce
|
|
`Temat: (brak tematu) | Od: ? | Data: …` for every new mail — with no error, just permanently
|
|
worse retrieval (recon §2.5 i).
|
|
|
|
## Secrets
|
|
|
|
Environment only (Decyzja (c)). There is no `--password` or `--user` flag: `--dsn <secret>`
|
|
lands in `ps` output and shell history. Values go into the existing `/opt/homelab/kb/.env`
|
|
(root-owned `0600`), which systemd reads **as root before dropping to `User=oskar`** — so the
|
|
app passwords reach the process without being readable by `oskar` at rest. The repo ships only
|
|
`jobs/mail-imap-sync/env.example` with placeholders.
|
|
|
|
## Stats must balance
|
|
|
|
```
|
|
uids_seen = processed + vanished + errors
|
|
processed = archived + archive_exists
|
|
processed = envelopes_inserted + envelopes_skipped_dup + envelopes_conflict_other_source
|
|
```
|
|
|
|
`headers_fallback`, `uidvalidity_resets`, `folder_errors` and `account_errors` are labels and
|
|
diagnostics, deliberately outside the equations — a folder that failed on EXAMINE never
|
|
produced a UID to account for. An unbalanced run is a failure regardless of what else it
|
|
reports.
|
|
|
|
## Exit codes
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| 0 | Balanced, zero errors at any level |
|
|
| 1 | Any message/folder/account error, or an unbalanced sum |
|
|
| 2 | Configuration error — missing/malformed env, no DSN, no archive directory |
|
|
|
|
Failures are isolated per account: a Gmail app password that stopped working must not also
|
|
stop Fastmail from syncing (the same stage isolation `cyclic_ingest` uses).
|
|
|
|
## Modes
|
|
|
|
| Invocation | Effect |
|
|
|---|---|
|
|
| *(default)* | Dry run: EXAMINE + SEARCH, report what a real tick would pull. No FETCH, no writes, no `.prom`. |
|
|
| `--apply` | The real thing, and the only mode that publishes metrics. |
|
|
| `--measure` | `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, then exit. Read-only, no DB. |
|
|
| `--limit N` | At most N messages per folder per tick — a brake for a first big sweep; the rest follows next tick. |
|
|
|
|
## Metrics
|
|
|
|
`/opt/homelab/state/node-exporter/kb-mail-sync.prom`, written atomically (tmp + rename) and
|
|
scraped by node_exporter's textfile collector on PIHA.
|
|
|
|
| Metric | Notes |
|
|
|---|---|
|
|
| `kb_mail_sync_last_run_timestamp` | every run |
|
|
| `kb_mail_sync_last_success_timestamp` | carried forward across a failed run, so a blip does not reset the staleness clock |
|
|
| `kb_mail_sync_last_exit_code` | |
|
|
| `kb_mail_sync_envelopes_inserted{account}` | |
|
|
| `kb_mail_sync_envelopes_skipped_dup{account}` | |
|
|
| `kb_mail_sync_conflict_other_source{account}` | the cross-account overlap above |
|
|
| `kb_mail_sync_uidvalidity_resets{account}` | |
|
|
| `kb_mail_sync_errors{account}` | message + folder + account levels |
|
|
| `kb_mail_sync_last_message_ts{account}` | `max(envelope.ts)`; omitted for an account with no rows |
|
|
|
|
One alert, `KbMailSyncStale` (`services/fleet-prometheus/rules/kb-mail-sync.yml`): 6 h without
|
|
a successful tick. **"No new mail for X days" was considered and rejected** (recon §3.4) —
|
|
zero new mail is a legal state of a mailbox, an alert that fires on a healthy system gets
|
|
muted, and a muted alert is not there on the day it matters. A per-account warning on
|
|
`kb_mail_sync_last_message_ts` may be worth adding after a month of observation, with a
|
|
threshold taken from measured behaviour rather than a guess.
|
|
|
|
## Definition of Done
|
|
|
|
Per `CLAUDE.md`: `pytest` passes (80 for this job, 111 for `kb-mail` including the adapter and
|
|
the cursor planner) plus a CLI smoke run. The tests drive the real `ImapClient` over a fake
|
|
`imaplib` connection, so what is covered is the adapter's actual protocol parsing: new
|
|
messages, UIDVALIDITY invalidation, dedup, resumption after an interrupted run, cross-account
|
|
collisions, account isolation, expunged UIDs, a dry run that touches nothing, and the metric
|
|
carry-forward.
|
|
|
|
**Nothing here has ever connected to a live mailbox.** The first real sync is an operator
|
|
step, supervised, per `kb/runbooks/mail-sync-run.md`.
|
|
|
|
## Follow-ups
|
|
|
|
- `mail_imap_sync.prom` and `documents_ingest.cyclic_ingest`'s renderer are two copies of the
|
|
same idea (this one supports labels, that one does not). Worth folding into one shared
|
|
helper the next time either changes.
|
|
- `kb-mail-sync.timer` joins the "shadow-deploy family" — units installed outside GitOps drift
|
|
detection — listed in `hosts/piha/jobs.yaml`. That is open question 5 of
|
|
`kb/subsystems/recon-multiagent.md`; the list is now one item longer, knowingly.
|