From ae16deb8d3ae48c9583c37212aa2c61792f7f39f Mon Sep 17 00:00:00 2001 From: oskar Date: Thu, 6 Aug 2026 14:49:58 +0200 Subject: [PATCH] feat(mail-sync): scheduler PIHA, takt kb-ingest, runbook i dokumentacja MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Domkniecie Kroku 7. Realizuje Decyzje (d) reconu (host schedulera + korekta kadencji indeksowania) i doklada dokumentacje wg konwencji OKF. Scheduler (NIEAKTYWOWANY — wlacza operator): - jobs/mail-imap-sync/systemd/{service,timer,run.sh} — wzorzec 1:1 z kb-ingest, OnCalendar=hourly, Persistent=true, log do pliku (nigdy sam journal). - hosts/piha/jobs.yaml — deklaracja jednostek host-level na PIHA. Nowy plik, bo services.yaml jest dla kontenerow (supervisor dopasowuje jego wpisy do world-state i wpis niekontenerowy dryfowalby wiecznie jako missing_service). Nic tego pliku nie czyta — istnieje po to, zeby "shadow-deploy family" z otwartego pytania 5 reconu multiagentowego byla spisana, a nie tylko na nodzie. Takt indeksowania (Decyzja (d), recon §3.3): - kb-ingest.timer: 03:30 raz na dobe -> co 2 h. O 03:30 SOLARIA prawie na pewno spi (potwierdzone odczytem kb_ingest_embed_skipped 1 z 2026-08-06), a tick dostaje teraz etap mailowy: ~60 nowych chunkow na dobe pomijanych kazdej nocy sprawiloby, ze backlog rosnie monotonicznie i KbEmbedBacklogGrowing zapala sie NA STALE. Co 2 h zamiast stalej godziny — probe Ollamy sam wybiera okno, wiec ktorys tick w nie trafi niezaleznie od nawykow operatora. - cyclic_ingest: etap mailowy (mail_body_ingest --only-unchunked), import miekki, wiec venv bez tego pakietu pomija etap zamiast wywracac wrapper. Predykat bledu JEST luzniejszy niz wlasne main() tamtego joba i to jedyne takie miejsce w tym wrapperze: pojedynczy trwale nieparsowalny mail nie moze zamrozic last_success_timestamp i zapalic KbIngestStale na zawsze. Bledy per-mail sa publikowane jako kb_ingest_mail_parse_errors, nie chowane. Obserwowalnosc: KbMailSyncStale (6 h bez udanego ticku). Alert na cisze w skrzynce ODRZUCONY (decyzja operatora, zgodna z reconem §3.4) — zero nowych maili to legalny stan skrzynki, a alert zapalajacy sie na zdrowym systemie zostaje wyciszony i przestaje dzialac wtedy, gdy jest potrzebny. Dokumentacja: - kb/services/job-mail-imap-sync.md (OKF), kb/runbooks/mail-sync-run.md — 9 krokow pierwszego uruchomienia, w tym checklista 4 punktow [do weryfikacji na zywo] z reconu (polityki dostawcow — do sprawdzenia, nie do zgadniecia) oraz pomiar STATUS (MESSAGES) na Fastmailu, na ktorym zapada ODLOZONA decyzja o historii. - kb-mail-pillar.md: KOREKTA JMAP -> IMAP dla Fastmaila jako decyzja 2026-08-06; stary zapis zostaje jako historia z data. Zamkniete "unifikacja adaptera" i "sizing Gmaila"; otwarte zostaje "sizing Fastmaila" — celowo, bo rozstrzyga je pomiar, nie dyskusja. - kb-m5-faza-mailowa.md: Krok 7 IN PROGRESS + tabela zakresu wdrozonego, kb-m5-faza3.md: korekta harmonogramu i sekwencji wrappera, pkg-kb-mail.md: rozpisany ze stubu, kb-postgres.md: lista migracji + 005. Testy: 642 passed (calosc kb-mail, kb-retrieval i jobs). systemd-analyze verify na timerze przechodzi, OnCalendar=0/2:00:00 normalizuje sie do co 2 h. Co-Authored-By: Claude Opus 5 (1M context) --- hosts/piha/jobs.yaml | 48 +++ .../src/documents_ingest/cyclic_ingest.py | 135 ++++++- jobs/documents-ingest/systemd/kb-ingest.timer | 30 +- .../tests/test_cyclic_ingest.py | 152 +++++++- .../systemd/kb-mail-sync-run.sh | 18 + .../systemd/kb-mail-sync.service | 22 ++ .../mail-imap-sync/systemd/kb-mail-sync.timer | 22 ++ kb/phases/kb-m5-faza-mailowa.md | 51 ++- kb/phases/kb-m5-faza3.md | 15 +- kb/runbooks/mail-sync-run.md | 352 ++++++++++++++++++ kb/services/job-mail-body-ingest.md | 40 +- kb/services/job-mail-imap-sync.md | 256 +++++++++++++ kb/services/kb-postgres.md | 4 + kb/services/pkg-kb-mail.md | 55 ++- kb/subsystems/kb-mail-pillar.md | 35 +- .../fleet-prometheus/rules/kb-mail-sync.yml | 40 ++ 16 files changed, 1234 insertions(+), 41 deletions(-) create mode 100644 hosts/piha/jobs.yaml create mode 100755 jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh create mode 100644 jobs/mail-imap-sync/systemd/kb-mail-sync.service create mode 100644 jobs/mail-imap-sync/systemd/kb-mail-sync.timer create mode 100644 kb/runbooks/mail-sync-run.md create mode 100644 kb/services/job-mail-imap-sync.md create mode 100644 services/fleet-prometheus/rules/kb-mail-sync.yml diff --git a/hosts/piha/jobs.yaml b/hosts/piha/jobs.yaml new file mode 100644 index 0000000..03421bb --- /dev/null +++ b/hosts/piha/jobs.yaml @@ -0,0 +1,48 @@ +# Host-level systemd units on PIHA — declaration only. +# +# These are NOT docker-compose services, so they do not belong in services.yaml (whose +# entries the supervisor matches against world-state service keys; adding a non-container +# entry there would drift forever as missing_service). Nothing reads this file: it exists so +# the units installed outside the compose pipeline are written down in the repo rather than +# living only on the node and in a runbook. +# +# This is the "shadow-deploy family" the multiagent recon flags in its open question 5 +# (kb/subsystems/recon-multiagent.md) — units installed outside GitOps drift detection. +# kb-mail-sync joins that list knowingly, not by oversight. When question 5 is settled, this +# file is the inventory to settle it against. +# +# Installation and activation are always operator steps. A `git pull` never starts a timer. + +host: piha + +systemd_units: + kb-ingest: + unit: kb-ingest.timer + service: kb-ingest.service + source: jobs/documents-ingest/systemd/ + schedule: "0/2:00:00" # every 2 h — see the note in the .timer file + runs_as: oskar + environment_file: /opt/homelab/kb/.env + log_path: /opt/homelab/logs/kb-ingest/ + metrics: /opt/homelab/state/node-exporter/kb-ingest.prom + state: active # installed and enabled since 2026-07-30 + description: > + Cyclic KB ingest: paperless adapter, chunk+embed, summarize, embed summaries, and + (since 2026-08-06) the mail body stage that drains the unchunked-envelope queue + mail-imap-sync fills. Both embed stages are gated on an Ollama@SOLARIA probe. + + kb-mail-sync: + unit: kb-mail-sync.timer + service: kb-mail-sync.service + source: jobs/mail-imap-sync/systemd/ + schedule: hourly + runs_as: oskar + environment_file: /opt/homelab/kb/.env + log_path: /opt/homelab/logs/kb-mail-sync/ + metrics: /opt/homelab/state/node-exporter/kb-mail-sync.prom + state: declared # NOT installed, NOT enabled — operator activates + runbook: kb/runbooks/mail-sync-run.md + description: > + Incremental IMAP fetch for gmail (\All) and fastmail (INBOX, Archive, Sent) into the + .eml archive and the envelope table. Network-bound, no GPU — which is why it lives on + PIHA (24/7) and is decoupled from the embedding stages on SOLARIA (~16 h/day off). diff --git a/jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py b/jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py index c4afaa6..ccac1f1 100644 --- a/jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py +++ b/jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py @@ -3,9 +3,19 @@ paperless_adapter.run() -- new source='paperless' envelopes -> chunk_embed.run() -- new document_chunk rows (bge-m3 via Ollama@SOLARIA) + -> mail_body_ingest.run(only_unchunked=True) -- chunks+embeds mail the incremental + IMAP sync fetched (faza mailowa Krok 7) -> summarize.run_summarize(backend='anthropic') -- new document_summary rows -> summarize.run_embed_summaries() -- embeds those summaries (bge-m3) +The mail stage is the consumer end of `jobs/mail-imap-sync`, which runs on its own hourly +timer and deliberately does no GPU work (recon kb/audits/mail-sync-2026-08-06.md §3.2). The +handoff between them needs no queue of its own: an envelope with no `document_chunk` rows IS +the queue, so an interrupted run, a mail fetched while SOLARIA slept, or one inserted with a +backdated header all stay pending until they have chunks, with no state to reconcile. The +stage is soft-imported -- a venv without `mail-body-ingest` installed skips it and says so, +rather than failing to start the whole wrapper. + The two embed stages are gated on a pre-flight Ollama reachability probe (`GET /api/tags`) because SOLARIA has `availability_target: medium` (planned power-off, plan §1.3): "Ollama unreachable" is skipped, not failed -- new chunks/summaries are picked @@ -50,6 +60,11 @@ import structlog from documents_ingest import chunk_embed, paperless_adapter, summarize +try: # optional: only installed in venvs that also run the mail pipeline + from mail_body_ingest import ingest as mail_body_ingest +except ImportError: # pragma: no cover - exercised by the "not installed" test via monkeypatch + mail_body_ingest = None + _log = structlog.get_logger(__name__) # Wrapper always runs on PIHA -- Ollama lives on SOLARIA, reached over Tailscale (plan §7.1). @@ -60,6 +75,10 @@ DEFAULT_SUMMARY_MODEL = "claude-haiku-4-5" DEFAULT_EMBED_MODEL = "bge-m3" DEFAULT_OLLAMA_PROBE_TIMEOUT = 5.0 DEFAULT_PROM_PATH = Path("/opt/homelab/state/node-exporter/kb-ingest.prom") +# Mail sources whose envelopes this wrapper chunks+embeds. The archive root is PIHA's +# canonical one -- this wrapper only ever runs there. +DEFAULT_MAIL_SOURCES = ("gmail", "fastmail") +DEFAULT_MAIL_ARCHIVE_ROOT = Path("/home/oskar/kb/mail/archive") _BACKLOG_QUERY = ( "SELECT count(*) AS n FROM document_chunk WHERE excluded_reason IS NULL AND embedding IS NULL" @@ -95,6 +114,19 @@ PROM_METRIC_HELP: dict[str, tuple[str, str]] = { "gauge", "Active document_chunk rows (excluded_reason IS NULL) still missing an embedding.", ), + "kb_ingest_mail_chunks_inserted": ( + "gauge", "New document_chunk rows inserted by the mail body stage in the last run.", + ), + "kb_ingest_mail_parse_errors": ( + "gauge", + "Mails the body stage could not read or parse in the last run. Tolerated (they do " + "not fail the wrapper) but tracked -- a rising number is a real signal.", + ), + "kb_ingest_mail_skipped": ( + "gauge", + "1 if the mail body stage was skipped this run (Ollama unreachable, the " + "mail-body-ingest package not installed, or no .eml archive on this host).", + ), } _PROM_LINE_RE = re.compile(r"^(kb_ingest_\w+)\s+([0-9.eE+-]+)\s*$") @@ -139,6 +171,40 @@ def _embed_summaries_failed(stats: dict) -> bool: return stats["errors"] > 0 or not balanced +def _mail_body_failed(stats: dict) -> bool: + """Deliberately NOT a 1:1 mirror of mail_body_ingest.main() -- the one predicate in this + wrapper that is looser than its job's own. + + That CLI exits 1 on any `parse_errors`/`missing_file`/`read_errors`, which is right for a + supervised slice run: across 225k mails a handful is expected and worth a look. On a + recurring unattended tick it would be corrosive. A single permanently-unparseable mail + would mark every run failed, `kb_ingest_last_success_timestamp` would stop advancing, and + KbIngestStale would fire after 48h and stay lit -- reporting nothing but one bad mail + nobody can fix. That is the same "an alert that always fires is not an alert" failure the + schedule change in kb-ingest.timer exists to avoid. + + So per-mail failures are tolerated and published as `kb_ingest_mail_parse_errors` instead. + What still fails the wrapper is what means the stage itself is broken: unbalanced stats, + or `chunks_conflict_skipped` (an insert silently no-op'd -- a real defect, not bad input). + """ + balanced = ( + stats["mails_scanned"] == ( + stats["missing_file"] + stats["read_errors"] + stats["parse_errors"] + + stats["body_empty"] + stats["mails_chunked"] + ) + and stats["chunks_total"] == ( + stats["chunks_inserted"] + stats["chunks_newsletter_flagged"] + + stats["chunks_already_embedded"] + stats["chunks_conflict_skipped"] + + stats["chunks_errors"] + ) + ) + return stats["chunks_conflict_skipped"] > 0 or not balanced + + +def _mail_parse_errors(stats: dict) -> int: + return stats["missing_file"] + stats["read_errors"] + stats["parse_errors"] + + async def probe_ollama(ollama_url: str, timeout: float = DEFAULT_OLLAMA_PROBE_TIMEOUT) -> bool: """`GET /api/tags` reachability check (plan §7.1). Any exception or non-2xx means "treat as offline, skip the embed stages this tick" -- never raises.""" @@ -178,11 +244,14 @@ async def run_cyclic( tags_vocab_path: Path = summarize.DEFAULT_TAGS_VOCAB_PATH, apply: bool = False, ollama_probe_timeout: float = DEFAULT_OLLAMA_PROBE_TIMEOUT, + mail_sources: tuple[str, ...] = DEFAULT_MAIL_SOURCES, + mail_archive_root: Path = DEFAULT_MAIL_ARCHIVE_ROOT, ) -> dict: - """Runs all four stages, isolating failures per stage (an earlier failure never skips + """Runs all five stages, isolating failures per stage (an earlier failure never skips a later stage). Returns a dict with one entry per stage (`adapter`, `chunk_embed`, - `summarize`, `embed_summaries`, each `_stage_result()`-shaped), plus `ollama_up`, - `embed_backlog`, and the aggregate `failed` bool the caller should exit non-zero on.""" + `mail_body`, `summarize`, `embed_summaries`, each `_stage_result()`-shaped), plus + `ollama_up`, `embed_backlog`, and the aggregate `failed` bool the caller should exit + non-zero on.""" ollama_up = await probe_ollama(ollama_url, timeout=ollama_probe_timeout) adapter_stage = _stage_result() @@ -208,6 +277,39 @@ async def run_cyclic( else: _log.warning("skip.chunk_embed", reason="ollama_unreachable") + # Mail body stage -- the consumer end of the incremental sync (recon §3.2). It drains the + # queue that IS the data: envelopes with no document_chunk rows. Nothing to reconcile and + # nothing to reset, so a tick that skips it (GPU asleep) costs only latency. + mail_stage = _stage_result(skipped=True) + mail_skip_reason = None + if not mail_sources: + mail_skip_reason = "no_mail_sources_configured" + elif not ollama_up: + mail_skip_reason = "ollama_unreachable" + elif mail_body_ingest is None: + mail_skip_reason = "mail_body_ingest_not_installed" + elif not Path(mail_archive_root).is_dir(): + mail_skip_reason = "archive_root_missing" + + if mail_skip_reason is None: + try: + stats = await mail_body_ingest.run( + dsn=dsn, + archive_root=Path(mail_archive_root), + ollama_url=ollama_url, + model=embed_model, + sources=tuple(mail_sources), + only_unchunked=True, + apply=apply, + ) + mail_stage = _stage_result(stats=stats, failed=_mail_body_failed(stats)) + except Exception as exc: + _log.error("mail_body_exception", error=str(exc)) + mail_stage = _stage_result(failed=True, error=str(exc)) + else: + _log.warning("skip.mail_body", reason=mail_skip_reason) + mail_stage = _stage_result(skipped=True, error=None) + summarize_stage = _stage_result() try: stats = await summarize.run_summarize( @@ -241,13 +343,16 @@ async def run_cyclic( failed = any( stage["failed"] - for stage in (adapter_stage, chunk_stage, summarize_stage, embed_summaries_stage) + for stage in (adapter_stage, chunk_stage, mail_stage, summarize_stage, + embed_summaries_stage) ) return { "ollama_up": ollama_up, "adapter": adapter_stage, "chunk_embed": chunk_stage, + "mail_body": mail_stage, + "mail_skip_reason": mail_skip_reason, "summarize": summarize_stage, "embed_summaries": embed_summaries_stage, "embed_backlog": embed_backlog, @@ -285,6 +390,11 @@ def build_metrics(result: dict, now_ts: float, prom_path: Path) -> dict: "kb_ingest_chunks_inserted": (result["chunk_embed"]["stats"] or {}).get("chunks_inserted", 0), "kb_ingest_summaries_inserted": (result["summarize"]["stats"] or {}).get("summarized", 0), "kb_ingest_embed_skipped": 0 if result["ollama_up"] else 1, + "kb_ingest_mail_chunks_inserted": + (result["mail_body"]["stats"] or {}).get("chunks_inserted", 0), + "kb_ingest_mail_parse_errors": + _mail_parse_errors(result["mail_body"]["stats"]) if result["mail_body"]["stats"] else 0, + "kb_ingest_mail_skipped": 1 if result["mail_body"]["skipped"] else 0, } if result["embed_backlog"] is not None: metrics["kb_ingest_embed_backlog"] = result["embed_backlog"] @@ -335,6 +445,17 @@ def main() -> None: help=f"Ollama embedding model, chunks+summaries (default: {DEFAULT_EMBED_MODEL})") parser.add_argument("--tags-vocab", type=Path, default=summarize.DEFAULT_TAGS_VOCAB_PATH, help="Path to tags-vocab.yaml") + parser.add_argument("--mail-sources", + default=os.environ.get("KB_MAIL_SOURCES", ",".join(DEFAULT_MAIL_SOURCES)), + metavar="A,B", + help=f"Envelope sources the mail body stage drains " + f"(default: {','.join(DEFAULT_MAIL_SOURCES)}, or set KB_MAIL_SOURCES). " + f"Empty disables the stage.") + parser.add_argument("--mail-archive-root", type=Path, + default=Path(os.environ.get("MAIL_ARCHIVE_ROOT", + str(DEFAULT_MAIL_ARCHIVE_ROOT))), + help=f"Mail .eml archive root (default: {DEFAULT_MAIL_ARCHIVE_ROOT}). " + f"The stage is skipped if this directory does not exist.") parser.add_argument("--prom-path", type=Path, default=Path(os.environ.get("KB_INGEST_PROM_PATH", str(DEFAULT_PROM_PATH))), help=f"Textfile-collector output path (default: {DEFAULT_PROM_PATH})") @@ -365,6 +486,8 @@ def main() -> None: tags_vocab_path=args.tags_vocab, apply=args.apply, ollama_probe_timeout=args.ollama_probe_timeout, + mail_sources=tuple(s.strip() for s in args.mail_sources.split(",") if s.strip()), + mail_archive_root=args.mail_archive_root, )) now_ts = time.time() @@ -382,6 +505,10 @@ def main() -> None: chunk_embed=result["chunk_embed"]["stats"], chunk_embed_skipped=result["chunk_embed"]["skipped"], chunk_embed_error=result["chunk_embed"]["error"], + mail_body=result["mail_body"]["stats"], + mail_body_skipped=result["mail_body"]["skipped"], + mail_skip_reason=result["mail_skip_reason"], + mail_body_error=result["mail_body"]["error"], summarize=result["summarize"]["stats"], summarize_error=result["summarize"]["error"], embed_summaries=result["embed_summaries"]["stats"], diff --git a/jobs/documents-ingest/systemd/kb-ingest.timer b/jobs/documents-ingest/systemd/kb-ingest.timer index 45f9e7a..632ab69 100644 --- a/jobs/documents-ingest/systemd/kb-ingest.timer +++ b/jobs/documents-ingest/systemd/kb-ingest.timer @@ -1,14 +1,32 @@ # kb-ingest.timer — module 5 phase 3 step 5 (kb/phases/kb-m5-faza3.md §7.1). -# Daily at 03:30, not hourly: the plan fixes this schedule explicitly (low-traffic window, -# and the underlying jobs are full-corpus re-scans each tick — cheap at 186 documents -# today, but daily keeps headroom as the corpus grows). Persistent=true catches up after a -# PIHA reboot instead of silently skipping the day's run. +# +# Every 2 hours since 2026-08-06. It used to be daily at 03:30, which was right when the only +# producer was Paperless: full-corpus re-scans, 186 documents, and a low-traffic window. +# +# Two facts made that schedule wrong (recon kb/audits/mail-sync-2026-08-06.md §3.3, operator +# Decyzja (d)): +# 1. SOLARIA is powered off ~16 h/day BY DESIGN, and 03:30 falls inside that window. +# Measured on PIHA 2026-08-06: `kb_ingest_embed_skipped 1` — that morning's tick +# skipped both embed stages because the Ollama probe got no answer. +# 2. From 2026-08-06 this tick also drains the mail queue that kb-mail-sync fills: roughly +# 60 new chunks a day. Skipped every single night, the embed backlog would grow +# monotonically and KbEmbedBacklogGrowing (`> 0` for 72h) would light up permanently — +# signalling nothing but a schedule that never coincides with the GPU being on. An +# alert that is always firing has stopped being an alert. +# +# Two hours, rather than a fixed hour chosen to match when SOLARIA is usually up: the Ollama +# probe already picks the working window on its own, so some tick lands in it whatever the +# operator's habits are that week. Ticks that miss it skip the embed stages and cost nothing; +# the write stages are idempotent and the queue is derived from the data. +# +# Persistent=true catches up after a PIHA reboot instead of silently skipping a run. [Unit] -Description=Run kb-ingest.service daily (module 5 phase 3 step 5) +Description=Run kb-ingest.service every 2 hours (module 5 phase 3 step 5) [Timer] -OnCalendar=*-*-* 03:30:00 +OnCalendar=0/2:00:00 Persistent=true +RandomizedDelaySec=300 [Install] WantedBy=timers.target diff --git a/jobs/documents-ingest/tests/test_cyclic_ingest.py b/jobs/documents-ingest/tests/test_cyclic_ingest.py index 496eb3c..7861351 100644 --- a/jobs/documents-ingest/tests/test_cyclic_ingest.py +++ b/jobs/documents-ingest/tests/test_cyclic_ingest.py @@ -166,11 +166,16 @@ SUMMARIZE_OK = {"documents_fetched": 1, "duplicates_skipped": 0, "no_active_chun "already_summarized": 0, "summarized": 1, "llm_errors": 0, "documents_mapreduce": 0, "tags_truncated": 0} EMBED_SUMMARIES_OK = {"summaries_fetched": 1, "embedded": 1, "errors": 0} +MAIL_BODY_OK = {"mails_scanned": 1, "missing_file": 0, "read_errors": 0, "parse_errors": 0, + "body_empty": 0, "mails_chunked": 1, "chunks_total": 2, "chunks_inserted": 2, + "chunks_newsletter_flagged": 0, "chunks_already_embedded": 0, + "chunks_conflict_skipped": 0, "chunks_errors": 0} def _patch_stages(monkeypatch, *, ollama_up=True, adapter=None, chunk=None, summarize=None, - embed_summaries=None, backlog=0): - calls = {"adapter": 0, "chunk_embed": 0, "summarize": 0, "embed_summaries": 0} + embed_summaries=None, mail_body=None, backlog=0, mail_archive=True): + calls = {"adapter": 0, "chunk_embed": 0, "summarize": 0, "embed_summaries": 0, + "mail_body": 0} async def _fake_probe(url, timeout=5.0): return ollama_up @@ -195,9 +200,22 @@ def _patch_stages(monkeypatch, *, ollama_up=True, adapter=None, chunk=None, summ calls["embed_summaries"] += 1 return embed_summaries if embed_summaries is not None else dict(EMBED_SUMMARIES_OK) + async def _fake_mail_body_run(**kwargs): + calls["mail_body"] += 1 + if mail_body is not None and isinstance(mail_body, Exception): + raise mail_body + return mail_body if mail_body is not None else dict(MAIL_BODY_OK) + async def _fake_backlog(dsn): return backlog + class _FakeMailBodyModule: + run = staticmethod(_fake_mail_body_run) + + monkeypatch.setattr(cyclic_ingest, "mail_body_ingest", _FakeMailBodyModule) + # The archive root check is a real filesystem probe; pin it so the suite does not depend + # on whether the host running the tests happens to have a mail archive. + monkeypatch.setattr(cyclic_ingest.Path, "is_dir", lambda self: mail_archive) monkeypatch.setattr(cyclic_ingest, "probe_ollama", _fake_probe) monkeypatch.setattr(cyclic_ingest.paperless_adapter, "run", _fake_adapter_run) monkeypatch.setattr(cyclic_ingest.chunk_embed, "run", _fake_chunk_run) @@ -216,7 +234,8 @@ class TestRunCyclic: ) assert result["failed"] is False assert result["ollama_up"] is True - assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1} + assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1, + "mail_body": 1} assert result["chunk_embed"]["skipped"] is False assert result["embed_summaries"]["skipped"] is False assert result["embed_backlog"] == 0 @@ -238,6 +257,107 @@ class TestRunCyclic: assert result["chunk_embed"]["stats"] is None assert result["embed_summaries"]["skipped"] is True + async def test_mail_stage_drains_only_unchunked_envelopes(self, monkeypatch): + # The handoff from mail-imap-sync is the unchunked-envelope queue, not a timestamp. + captured = {} + + async def _capture(**kwargs): + captured.update(kwargs) + return dict(MAIL_BODY_OK) + + _patch_stages(monkeypatch, ollama_up=True) + monkeypatch.setattr(cyclic_ingest.mail_body_ingest, "run", staticmethod(_capture)) + await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert captured["only_unchunked"] is True + assert captured["sources"] == ("gmail", "fastmail") + assert captured["apply"] is True + + async def test_mail_stage_is_skipped_when_ollama_is_down(self, monkeypatch): + calls = _patch_stages(monkeypatch, ollama_up=False) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert calls["mail_body"] == 0 + assert result["mail_body"]["skipped"] is True + assert result["mail_skip_reason"] == "ollama_unreachable" + assert result["failed"] is False + + async def test_mail_stage_is_skipped_when_the_package_is_not_installed(self, monkeypatch): + # PIHA's venv may predate mail-body-ingest; that must not stop the whole wrapper. + calls = _patch_stages(monkeypatch, ollama_up=True) + monkeypatch.setattr(cyclic_ingest, "mail_body_ingest", None) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_skip_reason"] == "mail_body_ingest_not_installed" + assert result["failed"] is False + assert calls["adapter"] == 1 + + async def test_mail_stage_is_skipped_without_an_archive(self, monkeypatch): + _patch_stages(monkeypatch, ollama_up=True, mail_archive=False) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_skip_reason"] == "archive_root_missing" + assert result["failed"] is False + + async def test_mail_stage_is_skipped_when_no_sources_are_configured(self, monkeypatch): + _patch_stages(monkeypatch, ollama_up=True) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, mail_sources=(), + ) + assert result["mail_skip_reason"] == "no_mail_sources_configured" + + async def test_per_mail_parse_errors_do_not_fail_the_wrapper(self, monkeypatch): + # Looser than mail-body-ingest's own CLI predicate, on purpose: one permanently + # unparseable mail must not keep last_success frozen and light KbIngestStale forever. + flaky = dict(MAIL_BODY_OK, mails_scanned=2, parse_errors=1) + _patch_stages(monkeypatch, ollama_up=True, mail_body=flaky) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_body"]["failed"] is False + assert result["failed"] is False + + async def test_unbalanced_mail_stats_do_fail_the_wrapper(self, monkeypatch): + _patch_stages(monkeypatch, ollama_up=True, + mail_body=dict(MAIL_BODY_OK, mails_scanned=9)) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_body"]["failed"] is True + assert result["failed"] is True + + async def test_silent_chunk_conflict_fails_the_wrapper(self, monkeypatch): + # An insert that no-op'd is a defect in the code, not bad input. + broken = dict(MAIL_BODY_OK, chunks_inserted=1, chunks_conflict_skipped=1) + _patch_stages(monkeypatch, ollama_up=True, mail_body=broken) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_body"]["failed"] is True + + async def test_mail_stage_exception_is_isolated(self, monkeypatch): + calls = _patch_stages(monkeypatch, ollama_up=True, + mail_body=RuntimeError("archive vanished")) + result = await run_cyclic( + dsn="dsn", paperless_url="http://paperless", paperless_token="tok", + anthropic_api_key="key", apply=True, + ) + assert result["mail_body"]["failed"] is True + assert result["failed"] is True + assert calls["summarize"] == 1 # later stages still ran + async def test_adapter_failure_does_not_skip_later_stages(self, monkeypatch): broken_adapter = dict(ADAPTER_OK, errors=1) calls = _patch_stages(monkeypatch, ollama_up=True, adapter=broken_adapter) @@ -248,7 +368,8 @@ class TestRunCyclic: assert result["failed"] is True assert result["adapter"]["failed"] is True # every other stage still ran despite the adapter failing this tick. - assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1} + assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1, + "mail_body": 1} assert result["chunk_embed"]["failed"] is False assert result["summarize"]["failed"] is False assert result["embed_summaries"]["failed"] is False @@ -322,10 +443,33 @@ class TestBuildMetrics: "chunk_embed": {"stats": dict(CHUNK_OK), "failed": False, "skipped": False, "error": None}, "summarize": {"stats": dict(SUMMARIZE_OK), "failed": False, "skipped": False, "error": None}, "embed_summaries": {"stats": dict(EMBED_SUMMARIES_OK), "failed": False, "skipped": False, "error": None}, + "mail_body": {"stats": dict(MAIL_BODY_OK), "failed": False, "skipped": False, "error": None}, + "mail_skip_reason": None, "embed_backlog": embed_backlog, "failed": failed, } + def test_mail_metrics_are_published(self, tmp_path): + metrics = build_metrics(self._result(failed=False), now_ts=1000.0, + prom_path=tmp_path / "kb-ingest.prom") + assert metrics["kb_ingest_mail_chunks_inserted"] == 2 + assert metrics["kb_ingest_mail_parse_errors"] == 0 + assert metrics["kb_ingest_mail_skipped"] == 0 + + def test_tolerated_mail_parse_errors_are_still_published(self, tmp_path): + # Tolerated by the exit code, but not invisible — a rising count is a real signal. + result = self._result(failed=False) + result["mail_body"]["stats"] = dict(MAIL_BODY_OK, parse_errors=2, missing_file=1) + metrics = build_metrics(result, now_ts=1000.0, prom_path=tmp_path / "kb-ingest.prom") + assert metrics["kb_ingest_mail_parse_errors"] == 3 + + def test_skipped_mail_stage_reports_zero_chunks_and_a_skip_flag(self, tmp_path): + result = self._result(failed=False) + result["mail_body"] = {"stats": None, "failed": False, "skipped": True, "error": None} + metrics = build_metrics(result, now_ts=1000.0, prom_path=tmp_path / "kb-ingest.prom") + assert metrics["kb_ingest_mail_skipped"] == 1 + assert metrics["kb_ingest_mail_chunks_inserted"] == 0 + def test_success_sets_last_success_to_now(self, tmp_path): prom_path = tmp_path / "kb-ingest.prom" metrics = build_metrics(self._result(failed=False), now_ts=1000.0, prom_path=prom_path) diff --git a/jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh b/jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh new file mode 100755 index 0000000..1278dfe --- /dev/null +++ b/jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# kb-mail-sync-run.sh — thin launcher for kb-mail-sync.service (faza mailowa Krok 7). +# +# All sequencing and exit-code logic lives in the Python job (mail_imap_sync/sync.py); this +# script only computes a date-stamped log path and redirects, per the repo convention that a +# job must never log only to a terminal or the journal — the lesson from the 2026-07 backfill +# that lost 4999 rows of diagnostics to a closed tmux. +# +# Installed at /opt/homelab/kb/kb-mail-sync-run.sh (copied from the repo at install time), +# invoked by kb-mail-sync.service. `exec` preserves the job's exit code as this script's own, +# so systemd — and `kb_mail_sync_last_exit_code` — see the real result. +set -euo pipefail + +LOG_DIR="/opt/homelab/logs/kb-mail-sync" +mkdir -p "$LOG_DIR" +LOG_FILE="${LOG_DIR}/run-$(date +%Y%m%d).log" + +exec /opt/homelab/kb/venv/bin/mail-imap-sync --apply >> "$LOG_FILE" 2>&1 diff --git a/jobs/mail-imap-sync/systemd/kb-mail-sync.service b/jobs/mail-imap-sync/systemd/kb-mail-sync.service new file mode 100644 index 0000000..56deeb7 --- /dev/null +++ b/jobs/mail-imap-sync/systemd/kb-mail-sync.service @@ -0,0 +1,22 @@ +# kb-mail-sync.service — faza mailowa Krok 7 (recon kb/audits/mail-sync-2026-08-06.md §3.3). +# Host-level oneshot on PIHA, 1:1 with the kb-ingest pattern: the job needs the canonical +# .eml archive (local path), kb-postgres (localhost:5433) and outbound TLS to two IMAP +# servers. Containerizing it would buy nothing. +# +# NOT ENABLED BY THE REPO. Installation and activation are the operator's steps — see +# kb/runbooks/mail-sync-run.md. Nothing here starts on its own after a `git pull`. +# +# EnvironmentFile is read by systemd AS ROOT before dropping to User=oskar, so the IMAP app +# passwords in /opt/homelab/kb/.env (root-owned 0600) reach the process without ever being +# readable by `oskar` at rest. Keep that property: do not "fix" the file's ownership. +[Unit] +Description=KB incremental IMAP mail sync (gmail + fastmail) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=oskar +EnvironmentFile=/opt/homelab/kb/.env +WorkingDirectory=/opt/homelab/kb +ExecStart=/opt/homelab/kb/kb-mail-sync-run.sh diff --git a/jobs/mail-imap-sync/systemd/kb-mail-sync.timer b/jobs/mail-imap-sync/systemd/kb-mail-sync.timer new file mode 100644 index 0000000..3855c15 --- /dev/null +++ b/jobs/mail-imap-sync/systemd/kb-mail-sync.timer @@ -0,0 +1,22 @@ +# kb-mail-sync.timer — faza mailowa Krok 7 (recon kb/audits/mail-sync-2026-08-06.md §3.3). +# +# Hourly. At the measured ~37 messages a day a tick pulls one or two mails — effectively +# free — while keeping the corpus fresh to within an hour, which is what a future dispatcher +# reading this KB will need. Fetching does not touch the GPU, so this cadence is independent +# of SOLARIA's daily power cycle; indexing catches up separately via kb-ingest.timer. +# +# Persistent=true so a PIHA reboot catches the missed tick up instead of silently skipping it. +# RandomizedDelaySec keeps two providers from being hit at exactly :00 every hour. +# +# NOT ENABLED BY THE REPO — `systemctl enable --now` is an operator step (kb/runbooks/ +# mail-sync-run.md), deliberately after the first supervised --apply run. +[Unit] +Description=Run kb-mail-sync.service hourly (faza mailowa Krok 7) + +[Timer] +OnCalendar=hourly +Persistent=true +RandomizedDelaySec=120 + +[Install] +WantedBy=timers.target diff --git a/kb/phases/kb-m5-faza-mailowa.md b/kb/phases/kb-m5-faza-mailowa.md index 7e7a6c6..2978a06 100644 --- a/kb/phases/kb-m5-faza-mailowa.md +++ b/kb/phases/kb-m5-faza-mailowa.md @@ -13,10 +13,12 @@ links: [] > retrieval, Etap A apply na żywej bazie, bramka jakościowa **PASS** — patrz §8). > **Etap B (Krok 6) ZAMKNIĘTY 2026-08-06**: pełny korpus gmail jest zchunkowany > i zembedowany (389 012 chunków, zero nie-excluded bez wektora) — patrz §9 -> „Wynik Etapu B". **Krok 7 (recon przyrostówki) WYKONANY 2026-08-06**: -> `kb/audits/mail-sync-2026-08-06.md` — czeka na decyzje operatora (a)-(g), -> w tym rozstrzygnięcie IMAP vs JMAP dla Fastmaila (§10 niżej mówi JMAP, -> recon rekomenduje wspólny IMAP). +> „Wynik Etapu B". **Krok 7 (przyrostówka IMAP) — IN PROGRESS 2026-08-06**: +> recon `kb/audits/mail-sync-2026-08-06.md` wykonany, decyzje operatora (a)-(g) +> **zatwierdzone w całości 2026-08-06**, implementacja w repo (§10 „Zakres +> wdrożony"). **Kod nie łączył się z żywym kontem** — pierwszy sync, pomiar +> Fastmaila i aktywacja timera to kroki operatora wg +> `kb/runbooks/mail-sync-run.md`. > > Kontynuacja `05-faza4-plan.md` (faza 4: `packages/kb-retrieval` wydzielone, > serwis `kb-query` z UI działa na PIHA — „KB po raz pierwszy odpowiada przez @@ -698,7 +700,9 @@ oznacza co innego: backend embed padł, trzeba wznowić plaster po naprawie Olla ## 10. Krok 7 — IMAP/JMAP przyrostówka (zarys; szczegóły = osobny recon) -> **Recon wykonany 2026-08-06: `kb/audits/mail-sync-2026-08-06.md`.** Zarys +> **Stan: IN PROGRESS (2026-08-06).** Recon: +> `kb/audits/mail-sync-2026-08-06.md`; decyzje (a)-(g) zatwierdzone przez +> operatora 2026-08-06 w całości, implementacja opisana niżej. Zarys > poniżej pochodzi z 2026-07-22 i zachowuję go jako zapis intencji. Recon > rozstrzyga inaczej dwa jego punkty: (1) **Fastmail przez IMAP, nie JMAP** > (unifikacja adaptera — jeden `jobs/mail-imap-sync` zamiast @@ -707,6 +711,35 @@ oznacza co innego: backend embed padł, trzeba wznowić plaster po naprawie Olla > IDLE, reuse `save_eml`/`insert_envelope`, sekrety w `/opt/homelab/config/`) > się potwierdziła. +### Zakres wdrożony (2026-08-06) + +| # | Element | Gdzie | +|---|---|---| +| 1 | Adapter IMAP, jeden na oba konta — EXAMINE + `BODY.PEEK[]` (nigdy nie ustawia `\Seen`), wybór folderu po atrybucie SPECIAL-USE | `packages/kb-mail/src/kb_mail/imap.py` | +| 2 | Model stanu synca: `plan_folder_sync` (pierwszy tick / przyrost / unieważnienie UIDVALIDITY) + `contiguous_last_uid` (kursor tylko po nieprzerwanym ciągu sukcesów) | `packages/kb-mail/src/kb_mail/sync_state.py` | +| 3 | Migracja `005_mail_sync_state.sql`, klucz `(account, folder)` — Decyzja (f) | `services/kb-postgres/init/` | +| 4 | Job przyrostówki: fetch → `save_eml` → `insert_envelope(entities=[headers, attachment…])` → kursor. **Nie chunkuje i nie embeduje** | `jobs/mail-imap-sync/` | +| 5 | Ekstrakcja wspólnego parse'u (`parse_headers`, `message_id`, `parse_date`, `parse_attachments`) do `kb-mail` — klucz dedup z jednej implementacji | `packages/kb-mail/src/kb_mail/{headers,message}.py` | +| 6 | `--sources` + `--only-unchunked` w `mail-body-ingest`; pre-fetch kluczy chunków zawężony do zbioru roboczego | `jobs/mail-body-ingest/` | +| 7 | `DEFAULT_SUMMARYLESS_SOURCES += "fastmail"` — Decyzja (g), w tym samym commicie co źródło | `packages/kb-retrieval/` | +| 8 | Etap mailowy w `kb-ingest` (kolejka „koperty bez chunków") + takt timera 03:30 → co 2 h — Decyzja (d) | `jobs/documents-ingest/` | +| 9 | Jednostki systemd (**nieaktywowane**) + deklaracja jednostek host-level | `jobs/mail-imap-sync/systemd/`, `hosts/piha/jobs.yaml` | +| 10 | Metryki `.prom` per konto + reguła `KbMailSyncStale` | `services/fleet-prometheus/rules/kb-mail-sync.yml` | +| 11 | `env.example` z placeholderami; poświadczenia wyłącznie ze środowiska — Decyzja (c) | `jobs/mail-imap-sync/env.example` | +| 12 | Runbook pierwszego uruchomienia + checklista punktów `[do weryfikacji na żywo]` z reconu | `kb/runbooks/mail-sync-run.md` | + +Dokumentacja serwisu: `kb/services/job-mail-imap-sync.md`. + +**Poza zakresem tej implementacji, świadomie:** pierwszy żywy sync, pomiar +`STATUS (MESSAGES)` na Fastmailu i wynikająca z niego **decyzja o historii +Fastmaila** (Decyzja (e) — recon celowo jej nie podejmuje, bo zależy od liczby, +której nikt jeszcze nie zna), oraz aktywacja timera. Wszystko to robi operator +wg runbooka. Alert „cisza w skrzynce" **odrzucony** (decyzja operatora, zgodna +z reconem §3.4): zero nowych maili to legalny stan skrzynki, a alert zapalający +się na zdrowym systemie zostaje wyciszony — i przestaje działać wtedy, gdy jest +potrzebny. Jedyny alert to `KbMailSyncStale` („czy poller w ogóle działa"), +który fałszywych trafień nie ma. + Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`, `jobs/gmail-imap-poller`). Zarys decyzji do tamtego reconu: @@ -733,7 +766,10 @@ Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`, | `mail_ui_url` (klikalny link do maila w UI) | kb-00 etap 6, pole zarezerwowane w kb-query | z modułem mail-UI | | Graf wątków / entity_link z `entities[type=threading]` | ta faza tylko zapisuje surowiec (Decyzja 10) | przyszła faza „połączenia" | | Streszczenia selektywne maili (hybryda Haiku) | Decyzja 6 — odłożona | po ocenie trybu hybrid w praktyce | -| IMAP/JMAP przyrostówka — implementacja | Krok 7 (zarys) | osobny recon + pakiet | +| ~~IMAP/JMAP przyrostówka — implementacja~~ | Krok 7 | **wykonane 2026-08-06** (§10 „Zakres wdrożony"); pierwszy żywy sync po stronie operatora | +| Historia Fastmaila (pełny zaciąg vs tylko przyrost) | Decyzja (e) reconu | po pomiarze `mail-imap-sync --measure` — runbook §5 | +| Alert per konto na wiek najnowszego maila (`kb_mail_sync_last_message_ts`) | recon §3.4 | po miesiącu obserwacji; próg z pomiaru, nie z góry | +| Etykiety Gmaila w `entities` (`X-GM-LABELS`) | recon §2.3 | odłożone — `X-GM-EXT-1` przywiązuje kod do Google, wprost wbrew „protokół, nie provider" | ## 12. Plan implementacji (kolejność = zależności) @@ -746,7 +782,8 @@ Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`, | 4 | rsync + Etap A (12 mies.) + kalibracja | 2 | 1 sesja | **WYKONANE** | §7 „Wynik Etapu A" (run na żywo 2026-07-23); potwierdzone na żywej bazie 2026-08-04: `document_chunk` gmail = 33 871 (6 398 z embeddingiem + 27 473 `newsletter`) — zgodne co do sztuki z tabelą §7 | | 5 | Bramka jakościowa (eval mailowy + regresja) | 3, 4 + zapytania od operatora | 1 sesja | **WYKONANE** (PASS) | §8 „Wynik bramki"; `56f64e9` (eval + queries.yaml dla hybrid), `bce635c` (`mail_hit@3`, próg N2, werdykt PASS), `71eb264` (`--transport http`) | | 6 | Etap B (pełne archiwum) + regresja + obserwacja PIHA | 5 = PASS | 1 sesja | **WYKONANE** (2026-08-06) | §9 „Wynik Etapu B"; żywa baza: 389 012 chunków, 0 nie-excluded bez embeddingu. Weryfikacja plastrami 0-4 (wszystkie EXIT 0) + fix NUL `4ec0b78`; `docs/sessions/2026-08-06-kb-etapb-backfill.md` | -| 7 | Recon przyrostówki IMAP/JMAP | — (po 6) | 1 sesja (poza DoD fazy) | **WYKONANE** (2026-08-06) | `kb/audits/mail-sync-2026-08-06.md`. Ustalenia: korpus urywa się 2026-06-19 (dziura 48 dni ≈ 1 800 maili), zero kodu IMAP w repo, brak modelu stanu synca; cały nowy kod to jeden `jobs/mail-imap-sync` + 4 drobne zmiany w istniejącym torze. Do decyzji operatora: (a)-(g) w §4 tamtego dokumentu | +| 7 | Recon przyrostówki IMAP/JMAP | — (po 6) | 1 sesja (poza DoD fazy) | **WYKONANE** (2026-08-06) | `kb/audits/mail-sync-2026-08-06.md`. Ustalenia: korpus urywa się 2026-06-19 (dziura 48 dni ≈ 1 800 maili), zero kodu IMAP w repo, brak modelu stanu synca; cały nowy kod to jeden `jobs/mail-imap-sync` + 4 drobne zmiany w istniejącym torze. Decyzje (a)-(g) zatwierdzone przez operatora 2026-08-06 | +| 8 | Implementacja przyrostówki IMAP | 7 + decyzje (a)-(g) | 2 sesje (poza DoD fazy) | **IN PROGRESS** (2026-08-06) | Zakres w §10 „Zakres wdrożony". Testy zielone: 80 (`mail-imap-sync`) + 111 (`kb-mail`) + 191 (`documents-ingest`) + 127 (`mail-body-ingest` / `kb-retrieval`). **Nie uruchomione na żywym koncie** — pierwszy sync, pomiar Fastmaila i aktywacja timera po stronie operatora (`kb/runbooks/mail-sync-run.md`) | **Kryterium ukończenia fazy mailowej:** (a) pełny korpus gmail zchunkowany (bilans domknięty, `parse_errors` na poziomie pojedynczych sztuk jak diff --git a/kb/phases/kb-m5-faza3.md b/kb/phases/kb-m5-faza3.md index 703afad..cb34187 100644 --- a/kb/phases/kb-m5-faza3.md +++ b/kb/phases/kb-m5-faza3.md @@ -560,10 +560,19 @@ Adapter i embed są już idempotentne — nowość to wyłącznie orkiestracja i `/etc/systemd/system/`, `systemctl enable --now kb-ingest.timer`). Pierwszy systemd-timer w repo — świadomie host-level, nie kontener (joby potrzebują jednocześnie LAN, DB i plików hosta; konteneryzacja nic tu nie daje). -- **Harmonogram**: `OnCalendar=*-*-* 03:30`, `Persistent=true` (nadgania po reboocie). +- **Harmonogram**: ~~`OnCalendar=*-*-* 03:30`~~ → **`OnCalendar=0/2:00:00` (co 2 h) od + 2026-08-06**, `Persistent=true` (nadgania po reboocie). Zmiana wynika z Decyzji (d) reconu + przyrostówki (`kb/audits/mail-sync-2026-08-06.md` §3.3): o 03:30 SOLARIA prawie na pewno + śpi (potwierdzone odczytem `kb_ingest_embed_skipped 1`), a od tej daty tick dostaje też + etap mailowy — ~60 nowych chunków na dobę pomijanych każdej nocy zapaliłyby + `KbEmbedBacklogGrowing` na stałe. Co 2 h zamiast stałej godziny dopasowanej do nawyków + operatora: probe Ollamy sam wybiera okno, więc któryś tick w nie trafi niezależnie od tego, + o której SOLARIA wstaje w danym tygodniu. - **Sekwencja skryptu**: adapter `--apply` → chunk_embed `--apply` - (`OLLAMA_URL=http://solaria:11434`) → (po decyzji z pilota, rozszerzenie później: - summarize nowych dokumentów). Log do `/opt/homelab/logs/kb-ingest/run-YYYYMMDD.log`. + (`OLLAMA_URL=http://solaria:11434`) → **mail_body_ingest `--only-unchunked`** (dodane + 2026-08-06 — konsument kolejki, którą wypełnia `jobs/mail-imap-sync`; import miękki, więc + venv bez tego pakietu pomija etap zamiast wywracać wrapper) → summarize → embed-summaries. + Log do `/opt/homelab/logs/kb-ingest/run-YYYYMMDD.log`. - **Tolerancja na SOLARIĘ offline** (`availability_target: medium`): wrapper odróżnia „Ollama nieosiągalna" (probe `GET /api/tags` przed embedem; brak → pomiń embed, odnotuj, **to nie jest fail** — nadrobi następny run, bo embed jest idempotentny) od diff --git a/kb/runbooks/mail-sync-run.md b/kb/runbooks/mail-sync-run.md new file mode 100644 index 0000000..64228be --- /dev/null +++ b/kb/runbooks/mail-sync-run.md @@ -0,0 +1,352 @@ +--- +okf: "0.1" +type: runbook +visibility: private +status: active +updated: 2026-08-06 +links: + - ../services/job-mail-imap-sync.md + - ../audits/mail-sync-2026-08-06.md +--- + +# mail-imap-sync — pierwsze uruchomienie i eksploatacja + +Wszystko poniżej wykonuje **operator na PIHA**. Kod nie łączył się nigdy z żadnym żywym +kontem — pierwszy sync jest nadzorowany i idzie krokami 1-9. Timer aktywujesz dopiero +w kroku 9, po tym jak ręczny run przejdzie czysto. + +--- + +## 0. Punkty [do weryfikacji na żywo] + +Recon (`kb/audits/mail-sync-2026-08-06.md`) oznaczył cztery twierdzenia o zewnętrznych +dostawcach jako **niesprawdzone** — stan wiedzy modelu sięga maja 2026 i **nie jest +wiarygodnym źródłem dla polityki dostawcy z sierpnia 2026**. Sprawdź je przy okazji kroków +1-3 i dopisz wynik tutaj. Nie zgaduj. + +| # | Do sprawdzenia | Gdzie | Wynik | +|---|---|---|---| +| 1 | Czy Google nadal wydaje hasła aplikacji (i czy wymagają 2FA na koncie)? Gdyby metoda odpadła, przymusem staje się OAuth2 — reszta architektury się nie zmienia, wymianie podlega **wyłącznie** sposób uwierzytelnienia w `kb_mail.imap.ImapClient.connect`. | Krok 1 | *(wpisz)* | +| 2 | Czy Fastmail pozwala wydać hasło aplikacji **o zakresie tylko-IMAP** (bez dostępu do panelu)? Jeśli tak, jest to uprawnienie ściślejsze niż token JMAP — i to domyka argument za unifikacją adaptera. | Krok 2 | *(wpisz)* | +| 3 | Czy konto Fastmail wystawia jakiś folder wirtualny obejmujący całość (odpowiednik gmailowego `\All`)? Jeśli tak, zakres `INBOX+Archive+Sent` można uprościć. | Krok 4 (`--measure` wypisze wszystkie foldery po `LIST`) | *(wpisz)* | +| 4 | Dobowy limit transferu IMAP Gmaila (rzędu kilku GB). Przy ~1 800 zaległych maili (~225 MB) i ~4,6 MB/dobę bieżąco nie ma znaczenia — ma, gdyby ktoś kiedyś ciągnął IMAP-em pełną historię. | Krok 6, jeśli pierwszy sweep się urwie | *(wpisz)* | + +--- + +## 1. Hasło aplikacji — Gmail + +Wymaga włączonego 2FA na koncie (bez tego Google nie pokaże opcji haseł aplikacji). +Wygeneruj hasło aplikacji dla „Mail", zapisz je **od razu do menedżera haseł** — Google +pokazuje je jeden raz. + +> **Nie wypisuj hasła na ekran i nie wklejaj go do żadnej sesji.** W tym repo są dwa +> udokumentowane przypadki rotacji sekretu po wycieku do transkryptu sesji +> (`docs/sessions/2026-07-15.md` §Krok 3, `docs/sessions/2026-07-21.md` incydent 3). To jedyna +> droga, którą sekrety w tym homelabie dotąd wyciekały. + +Google wyświetla hasło w czterech grupach po cztery znaki — **wpisuj bez spacji**. + +## 2. Hasło aplikacji — Fastmail + +Wydaj hasło aplikacji o zakresie **IMAP** (nie „full access"), jeśli konto na to pozwala — +to punkt (2) z tabeli wyżej. Zapisz tak samo jak wyżej. + +## 3. Wypełnienie `.env` + +Sekrety dopisujesz do **istniejącego** `/opt/homelab/kb/.env` (już `root:root 0600`, już +trzyma `KB_DSN`, `PAPERLESS_API_TOKEN`, `ANTHROPIC_API_KEY`). Edytorem, nie `echo >>` — +`echo` zostawia hasło w historii powłoki. + +```bash +sudo -e /opt/homelab/kb/.env +``` + +Wzorzec kluczy i komentarze: `jobs/mail-imap-sync/env.example`. Minimum: + +``` +MAIL_ACCOUNTS=gmail,fastmail +MAIL_GMAIL_USER=... +MAIL_GMAIL_APP_PASSWORD=... +MAIL_GMAIL_INITIAL_MODE=since +MAIL_GMAIL_INITIAL_SINCE=2026-06-15 +MAIL_FASTMAIL_USER=... +MAIL_FASTMAIL_APP_PASSWORD=... +MAIL_FASTMAIL_INITIAL_MODE=new-only +``` + +Sprawdź uprawnienia po edycji — muszą zostać `600 root:root`: + +```bash +sudo ls -l /opt/homelab/kb/.env +``` + +Dlaczego akurat tak: systemd czyta `EnvironmentFile` **jako root, przed zrzuceniem uprawnień +do `User=oskar`**. Sekret trafia do procesu, ale nie jest czytelny dla `oskar` w spoczynku. +To lepsza własność niż plik `600 oskar:oskar` — nie „napraw" jej. + +### Skąd data dla `MAIL_GMAIL_INITIAL_SINCE` + +Kilka dni **przed** najnowszą kopertą w bazie — nakładka jest darmowa (dedup po Message-ID), +a luka nie jest: + +```bash +docker exec kb-postgres psql -U kb -d kb -c \ + "SELECT source, count(*), max(ts) FROM envelope GROUP BY source ORDER BY source;" +``` + +Stan na 2026-08-06: gmail 225 030 kopert, `max(ts) = 2026-06-19 19:13:39+02`. Stąd +`2026-06-15` w przykładzie wyżej. `SINCE` filtruje po INTERNALDATE serwera (kiedy wiadomość +przyszła), nie po nagłówku `Date:` — czyli po właściwej stronie tego rozróżnienia. + +## 4. Instalacja i migracja + +```bash +cd ~/homelab-codex-ws && git pull +source /opt/homelab/kb/venv/bin/activate +pip install -e packages/kb-mail/ -e jobs/mail-imap-sync/ +# mail-body-ingest jest potrzebny etapowi mailowemu w kb-ingest (patrz krok 8): +pip install -e packages/kb-retrieval/ -e jobs/mail-body-ingest/ +``` + +Migracja `005` — `initdb` odpala skrypty z `init/` **tylko na świeżym wolumenie**, więc na +istniejącej bazie aplikujesz ją ręcznie: + +```bash +docker exec -i kb-postgres psql -U kb -d kb \ + < ~/homelab-codex-ws/services/kb-postgres/init/005_mail_sync_state.sql + +docker exec kb-postgres psql -U kb -d kb -c "\d mail_sync_state" +``` + +Oczekiwane: kolumny `account, folder, uidvalidity, last_uid, last_sync_ts` i +`PRIMARY KEY (account, folder)`. DDL jest `IF NOT EXISTS` — powtórzenie jest bezpieczne. + +## 5. Pomiar — ile waży Fastmail + +**To jest moment, w którym zapada decyzja o historii Fastmaila.** Recon celowo jej nie +podjął: zależy od liczby, której nikt jeszcze nie znał (Decyzja (e), „najpierw pomiar, potem +decyzja"). Odłożona **do tego kroku**. + +```bash +set -a; . /opt/homelab/kb/.env; set +a +mail-imap-sync --measure +``` + +Tryb jest read-only: `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, zero zapisów, zero +dotknięcia flag wiadomości. Weryfikuje przy okazji, że oba hasła aplikacji działają. + +Odczytaj sumę `messages` dla `INBOX + Archive + Sent` i wybierz: + +| Rozmiar | `MAIL_FASTMAIL_INITIAL_MODE` | Uwagi | +|---|---|---| +| kilka-kilkanaście tysięcy | `full` | Rzędu godzin transferu; mechanizm ten sam co przyrost, więc nie kosztuje osobnego kodu. To prawdopodobnie ta „sensowna poczta", o którą chodziło bardziej niż o gmailowe newslettery. | +| dziesiątki tysięcy+ | `full` **z `--limit`** | Pierwszy sweep w kawałkach, np. `--limit 2000` na tick; kursor idzie od najstarszych, reszta dochodzi kolejnymi tickami. | +| nieinteresująca | `new-only` | Historia zostaje poza KB. | + +Wpisz wybór do `.env` **zanim** puścisz pierwszy `--apply` — po pierwszym runie istnieje +wiersz w `mail_sync_state` i `INITIAL_MODE` przestaje mieć jakikolwiek wpływ. + +Przy okazji: `--measure` po drodze robi `LIST`, więc log pokaże wszystkie foldery konta — +stąd odpowiedź na punkt (3) z tabeli w §0. + +## 6. Dry run, potem pierwszy sync + +Dry run nie pobiera ani jednej wiadomości — robi EXAMINE + SEARCH i mówi, ile by ściągnął: + +```bash +mail-imap-sync +``` + +Czytaj `candidates` per folder. Dla gmaila przy `since=2026-06-15` spodziewaj się rzędu +**~1 800** (48 dni × ~37/dobę). Rząd wielkości zupełnie inny niż oczekiwany = zatrzymaj się +i sprawdź konfigurację, nie puszczaj `--apply`. + +Pierwszy prawdziwy run — z hamulcem, do pliku, na niskim priorytecie: + +```bash +mkdir -p /opt/homelab/logs/kb-mail-sync +nice -n 10 ionice -c2 -n7 mail-imap-sync --apply --limit 200 \ + > /opt/homelab/logs/kb-mail-sync/first-run.log 2>&1 +echo "exit=$?" +``` + +Sprawdź w logu linię `summary`: + +- `envelopes_inserted` > 0, `errors` = 0, +- `archived` + `archive_exists` = `processed`, +- `balance_problems` puste, +- `headers_fallback` — pojedyncze sztuki są normalne (8-bitowe nagłówki), dziesiątki nie. + +Potem puszczaj bez `--limit`, aż `candidates` w dry runie spadnie do zera: + +```bash +mail-imap-sync --apply >> /opt/homelab/logs/kb-mail-sync/first-run.log 2>&1 +``` + +Kontrola w bazie: + +```bash +docker exec kb-postgres psql -U kb -d kb -c \ + "SELECT account, folder, uidvalidity, last_uid, last_sync_ts FROM mail_sync_state ORDER BY 1,2;" +docker exec kb-postgres psql -U kb -d kb -c \ + "SELECT source, count(*), max(ts) FROM envelope GROUP BY source ORDER BY 1;" +``` + +`max(ts)` dla gmaila powinien być z ostatnich dni, a `source='fastmail'` pojawić się po raz +pierwszy w historii tej bazy. + +## 7. Indeksowanie i weryfikacja end-to-end + +Sam sync **nie chunkuje i nie embeduje** — celowo (recon §3.2). Nowe koperty czekają +w kolejce, którą jest brak chunków. Zdrenuj ją ręcznie raz, przy włączonej SOLARII: + +```bash +mail-body-ingest --only-unchunked --apply \ + >> /opt/homelab/logs/kb-mail-sync/first-ingest.log 2>&1 +``` + +Pełny test od końca do końca — wyślij sobie maila z rozpoznawalną frazą, potem: + +```bash +# 1. sync go pobiera +mail-imap-sync --apply + +# 2. koperta jest w bazie +docker exec kb-postgres psql -U kb -d kb -c \ + "SELECT id, source, ts FROM envelope ORDER BY ts DESC LIMIT 5;" + +# 3. ingest robi chunki i embeddingi (SOLARIA musi być włączona) +mail-body-ingest --only-unchunked --apply + +# 4. widać go w wyszukiwarce, trybem domyślnym (hybrid) +curl -s "http://localhost:8230/search?q=" | jq '.results[] | {envelope_id, dist}' +``` + +Krok 4 jest tym, co weryfikuje Decyzję (g): `fastmail` musi być w +`DEFAULT_SUMMARYLESS_SOURCES`, inaczej koperty zaindeksują się poprawnie i **nie pojawią się +w żadnym wyniku** — bez błędu gdziekolwiek. Jeśli mail z Fastmaila nie wychodzi w `/search`, +a z gmaila wychodzi, to jest pierwsze miejsce do sprawdzenia. + +## 8. Instalacja jednostek systemd (jeszcze bez włączenia) + +```bash +sudo install -m 0755 ~/homelab-codex-ws/jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh \ + /opt/homelab/kb/kb-mail-sync-run.sh +sudo cp ~/homelab-codex-ws/jobs/mail-imap-sync/systemd/kb-mail-sync.{service,timer} \ + /etc/systemd/system/ +sudo systemctl daemon-reload +``` + +**Równocześnie: zmiana taktu `kb-ingest`** (Decyzja (d), recon §3.3). Timer przechodzi +z 03:30 raz na dobę na **co 2 h**, a `kb-ingest` dostaje etap mailowy: + +```bash +sudo cp ~/homelab-codex-ws/jobs/documents-ingest/systemd/kb-ingest.timer /etc/systemd/system/ +sudo systemctl daemon-reload && sudo systemctl restart kb-ingest.timer +systemctl list-timers kb-ingest.timer +``` + +Dlaczego to musi iść razem z uruchomieniem przyrostówki: o 03:30 SOLARIA prawie na pewno śpi +(potwierdzone 2026-08-06 odczytem `kb_ingest_embed_skipped 1`). Dopięcie ~60 nowych chunków +na dobę do ticku, który każdej nocy pomija embedowanie, sprawiłoby, że backlog rośnie +monotonicznie i `KbEmbedBacklogGrowing` (próg `> 0` przez 72 h) **zapala się na stałe** — nie +sygnalizując awarii, tylko rozjazd harmonogramu z dobowym cyklem SOLARII. Alert, który świeci +zawsze, przestaje być alertem. + +Reguła alertowa dla samego synca (`services/fleet-prometheus/rules/kb-mail-sync.yml`) wchodzi +przy najbliższym deployu fleet-prometheus. Dopóki timer nie chodzi, metryka nie istnieje, +`time() - ` nie daje wyniku i reguła jest bezczynna — kolejność nie ma znaczenia. + +## 9. Aktywacja timera + +Dopiero teraz, i tylko jeśli kroki 6-7 przeszły czysto: + +```bash +sudo systemctl enable --now kb-mail-sync.timer +systemctl list-timers kb-mail-sync.timer +``` + +Po pierwszym automatycznym ticku: + +```bash +journalctl -u kb-mail-sync.service -n 50 +cat /opt/homelab/state/node-exporter/kb-mail-sync.prom +``` + +W `.prom` szukaj `kb_mail_sync_last_success_timestamp` (niezerowy) i +`kb_mail_sync_envelopes_inserted{account="..."}`. + +--- + +## Eksploatacja + +### Codzienne sprawdzenie + +```bash +tail -n 20 /opt/homelab/logs/kb-mail-sync/run-$(date +%Y%m%d).log +docker exec kb-postgres psql -U kb -d kb -c \ + "SELECT account, folder, last_uid, last_sync_ts FROM mail_sync_state ORDER BY 1,2;" +``` + +`last_sync_ts` ma się przesuwać co godzinę **nawet gdy nic nie przyszło** — to jest odpowiedź +na pytanie „czy poller w ogóle działa", niezależna od tego, czy ktoś do nas napisał. + +### Folder stanął — jedna wiadomość blokuje kursor + +Objaw: `errors` niezerowe co tick, `last_uid` w `mail_sync_state` stoi, w logu ten sam UID +przy `skip.fetch_error` / `skip.store_error`. + +To jest zaprojektowane zachowanie: kursor nie przeskakuje wiadomości, której nie udało się +zapisać (przeskakiwanie = ciche gubienie poczty). Wiadomości *po* niej i tak lądują +w archiwum i w bazie — blokada dotyczy wyłącznie kursora. + +Jeśli wiadomość jest trwale nie do pobrania, przestaw kursor ręcznie o jeden: + +```bash +docker exec kb-postgres psql -U kb -d kb -c \ + "UPDATE mail_sync_state SET last_uid = WHERE account='gmail' AND folder='';" +``` + +Zanotuj w `kb/incidents/`, którego maila świadomie pominięto. + +### Serwer unieważnił UIDVALIDITY + +W logu `imap.uidvalidity_reset`, w metrykach `kb_mail_sync_uidvalidity_resets{account} = 1`. +Job sam robi pełne przemiecenie folderu i opiera się na dedupie po Message-ID — nie ma tu nic +do zrobienia ręcznie. Tick będzie dłuższy i `envelopes_skipped_dup` skoczy do rozmiaru +folderu; to jest poprawny przebieg, nie awaria. + +### Rotacja hasła aplikacji + +Odwołaj stare u dostawcy, wydaj nowe, podmień w `/opt/homelab/kb/.env`, `systemctl start +kb-mail-sync.service`. Restart timera nie jest potrzebny — `EnvironmentFile` czytany jest przy +każdym starcie serwisu. Nic w stanie synca nie zależy od poświadczeń. + +### Dołożenie folderu + +Dopisz go do `MAIL__FOLDERS` w `.env`. Nowy folder nie ma wiersza w `mail_sync_state`, +więc obowiązuje go `INITIAL_MODE` tego konta — jeśli chcesz jego historię, przestaw na `full` +**na jeden run**, potem wróć do poprzedniej wartości (istniejące foldery mają już kursory +i zmiana ich nie dotyczy). Zawartość pokrywająca się z innym folderem niczego nie zduplikuje. + +## Testy + +```bash +pip install -e "jobs/mail-imap-sync[dev]" +cd jobs/mail-imap-sync && pytest +``` + +80 testów, bez sieci i bez bazy: `ImapClient` jest napędzany podstawionym połączeniem +`imaplib` (więc testowane jest prawdziwe parsowanie protokołu, nie mock adaptera), a +`envelope`/`mail_sync_state` są w pamięci. Pokrycie: nowe wiadomości, uniewaznienie +UIDVALIDITY, dedup na trzech warstwach, wznowienie po przerwaniu, kolizja Message-ID między +kontami, izolacja kont, wygasłe UID-y, dry-run bez sieci, przenoszenie +`last_success_timestamp` przez nieudany run. + +W worktree bez gotowego venva: + +```bash +python3 -m venv /tmp/kbvenv && /tmp/kbvenv/bin/pip install -q pytest pytest-asyncio \ + -e packages/kb-mail/ -e jobs/mail-imap-sync/ +/tmp/kbvenv/bin/python -m pytest jobs/mail-imap-sync/tests packages/kb-mail/tests -q +``` diff --git a/kb/services/job-mail-body-ingest.md b/kb/services/job-mail-body-ingest.md index 8287dae..7e6d009 100644 --- a/kb/services/job-mail-body-ingest.md +++ b/kb/services/job-mail-body-ingest.md @@ -3,9 +3,10 @@ okf: "0.1" type: service visibility: private status: active -updated: 2026-08-05 +updated: 2026-08-06 links: - ../runbooks/mail-body-ingest-run.md + - job-mail-imap-sync.md --- # mail-body-ingest @@ -26,9 +27,16 @@ only thing genuinely shared is the chunker itself, which is why it was extracted ## Where it runs -**On SOLARIA** (needs Ollama on `localhost` for `/api/embed`), against `kb-postgres@PIHA` over -Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than -read live over the network — 225k small files over Tailscale would be slow and fragile. +**On SOLARIA** for backfill runs (needs Ollama on `localhost` for `/api/embed`), against +`kb-postgres@PIHA` over Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7, +Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow +and fragile. + +**Also on PIHA since 2026-08-06**, as the mail stage of `kb-ingest` (see "Two modes" below). +There it reads the canonical archive locally and calls Ollama@SOLARIA over Tailscale — which +is fine at this scale: the daily increment is ~60 chunks at 8-18 ms each, measured at 8.2 ms +round-trip from PIHA. The argument against embedding from PIHA (recon §3.1) was about 271k +backfill chunks and does not carry over. Install (from repo root, on SOLARIA): @@ -71,6 +79,30 @@ pip install -e jobs/mail-body-ingest/ `gmail-header-backfill` — done for every parsed envelope regardless of newsletter/ body_empty status, since it's the same read either way. +## Two modes: backfill slice vs cyclic queue + +The job was written for supervised full-corpus slices (`--limit`/`--offset`/`--since`). Since +2026-08-06 it also runs as a stage of `kb-ingest` every 2 hours, draining what the incremental +IMAP sync fetched: + +```bash +mail-body-ingest --only-unchunked --apply # cyclic mode +``` + +`--only-unchunked` restricts the set to envelopes with **no `document_chunk` rows at all** — +the queue that the data itself defines (recon `kb/audits/mail-sync-2026-08-06.md` §3.2). It +needs no separate state and is self-healing: an interrupted run, a mail fetched while Ollama +was down, a message inserted with a backdated header — each stays pending until it has chunks. + +`--since` was explicitly rejected for that role. It filters on `envelope.ts`, the sender's +`Date:` header, so a mail delivered today carrying a month-old date would fall outside the +window and never be chunked at all. `--since` stays what it was: a tool for staging manual runs. + +`--sources` defaults to `gmail,fastmail`. The pre-fetched chunk-key set is scoped to the +working set (`WHERE model = $1 AND envelope_id = ANY($2)`); unbounded it read all 389 012 keys +(~26 MB, ~1.0 s) on every run, which is nothing once for a 50k slice and wasteful every two +hours on a node with 2.4 GB available (recon §2.5 iv). + ## Stats must balance ``` diff --git a/kb/services/job-mail-imap-sync.md b/kb/services/job-mail-imap-sync.md new file mode 100644 index 0000000..7d037a2 --- /dev/null +++ b/kb/services/job-mail-imap-sync.md @@ -0,0 +1,256 @@ +--- +okf: "0.1" +type: service +visibility: private +status: active +updated: 2026-08-06 +links: + - ../runbooks/mail-sync-run.md + - ../audits/mail-sync-2026-08-06.md + - ../phases/kb-m5-faza-mailowa.md +--- + +# mail-imap-sync + +Module 5, faza mailowa, Krok 7 (`kb/phases/kb-m5-faza-mailowa.md` §10), built to the recon +`kb/audits/mail-sync-2026-08-06.md` after the operator approved decisions (a)-(g) on +2026-08-06. It is the first code in this repo that talks to a mail server. + +Before it, the mail corpus was a photograph: 225 030 envelopes ending at **2026-06-19**, +imported once from a Gmail Takeout, with a gap that grew by roughly 37 messages a day. This +job closes the gap and keeps it closed. + +## Where it runs + +**On PIHA**, hourly, as a host-level systemd oneshot (`jobs/mail-imap-sync/systemd/`). The +choice is not close (recon §3.1): PIHA is up 24/7, holds the canonical `.eml` archive, and +runs kb-postgres locally. SOLARIA is powered off ~16 h a day by design, so a poller living +there would read the mailbox only when the desktop happened to be on. + +Install (from repo root, on PIHA): + +```bash +pip install -e packages/kb-mail/ +pip install -e jobs/mail-imap-sync/ +``` + +## What one tick does + +Per account, per folder: + +1. **EXAMINE** the folder — read-only. Never `SELECT`, and bodies come via `BODY.PEEK[]`, so + the job never sets `\Seen` on the operator's own mail. +2. **Plan** the tick from `mail_sync_state` and the server's UIDVALIDITY/UIDNEXT — first + contact, ordinary increment, or a UIDVALIDITY reset (see below). +3. **UID SEARCH** for the UIDs this tick owns. +4. Per UID, ascending: **FETCH** → `save_eml()` (append-only; `FileExistsError` just means + "already have it") → `insert_envelope()` with `entities = [headers, attachment…]`. +5. **UPDATE `mail_sync_state`** — but only over the contiguous prefix of messages that are + fully durable in both the archive and the DB. + +## Folder scope (Decyzja (e)) + +| Account | Scope | Why | +|---|---|---| +| gmail | SPECIAL-USE `\All` | one fetch per message despite many labels; the same population as the Takeout the existing 225 030 envelopes came from, so the corpus stays continuous | +| fastmail | `INBOX` + `Archive` + `Sent` | a classic IMAP layout has no `\All` equivalent | + +Spam, Trash and Drafts are excluded from both. Spam and Trash are defined noise and were not +in the Takeout either; drafts are not correspondence, have no stable Message-ID, and mutate. + +The gmail folder is resolved **by SPECIAL-USE attribute, never by name**. Gmail localizes the +mailbox — with a Polish UI it is `[Gmail]/Wszystkie` — so a hardcoded `[Gmail]/All Mail` would +sync nothing while reporting a clean run every hour. + +Scope is per-account configuration (`MAIL__FOLDERS` / `_SPECIAL_USE`), so adding a +folder later is a config change, not a code change; Message-ID dedup absorbs any overlap. + +## The sync cursor + +`mail_sync_state` (migration `005`), keyed `(account, folder)`, holding `uidvalidity`, +`last_uid` and `last_sync_ts`. A table rather than a file in `/opt/homelab/state/` +(Decyzja (f)) for one decisive reason: **the cursor and the envelopes it describes must +restore together or not at all.** A state file surviving a DB restore would make the poller +silently skip everything between the restored rows and the file's `last_uid` — a failure with +no symptom, noticed months later as missing mail. + +Deriving the cursor from `max(envelope.ts)` instead was rejected outright: `ts` is the +sender's `Date:` header, not delivery time. 2 559 envelopes in the live corpus already sit at +epoch 1970. + +### UIDVALIDITY + +When the server changes UIDVALIDITY, every stored UID becomes meaningless. The job sweeps the +folder (`UID SEARCH ALL`), leans on Message-ID dedup to make that cheap, and stores the new +value. This is rare — and it is the failure that loses mail with no error anywhere if it is +not handled. + +### Why the cursor only crosses a contiguous prefix + +A message that fails is **not** stepped over. The cursor stops at the message before it, and +the next tick refetches from there — free, because the archive write is append-only and the +insert is `ON CONFLICT DO NOTHING`. Messages *after* the failure are still stored in the same +tick; they simply do not move the cursor. + +The consequence is deliberate: a message that fails permanently **stalls its folder**. It +does so visibly — a non-zero error counter every tick, a cursor that stops moving, the +offending UID in the log — which is strictly better than the alternative of dropping mail +while reporting success. The runbook documents the manual `UPDATE mail_sync_state` escape +hatch for that case. + +### The `n:*` trap + +`UID SEARCH UID n:*` is a **range**, and when `n` exceeds the highest existing UID the server +resolves it as `highest:n` and returns the last message anyway. Without the client-side +filter, "new mails this tick" would never read zero on an idle mailbox, and every +observability claim built on that counter would be a small permanent lie. + +## First contact with a folder + +`MAIL__INITIAL_MODE` decides what the first-ever tick on a folder fetches. It has no +effect once a `mail_sync_state` row exists. + +| Mode | Behaviour | When | +|---|---|---| +| `new-only` | record `UIDNEXT-1`, fetch nothing | start the corpus from now | +| `since` | `UID SEARCH SINCE ` (server INTERNALDATE, not the `Date:` header) | close a known gap | +| `full` | `UID SEARCH ALL` | pull a mailbox's whole history | + +**gmail wants `since`.** A `new-only` first tick would leave the 48-day gap the recon measured +unfilled forever — and that gap is the entire reason this job exists. Point +`MAIL_GMAIL_INITIAL_SINCE` a few days before the newest envelope in the DB; dedup makes the +overlap free. + +**fastmail is undecided by design.** The account is greenfield (zero rows, zero archive +files), and whether to pull its history depends on how big it is — a number nobody has yet. +Run `mail-imap-sync --measure` first (recon Decyzja (e) explicitly refuses to guess it). + +## Message-ID collisions across accounts + +`envelope.id` is a bare Message-ID and therefore **globally unique in the table**, with no +source prefix (unlike paperless, which uses `paperless:N`). A mail present in both mailboxes — +a mailing list, a forward, a CC to both addresses — has the same Message-ID in both, so +whoever inserts first sets `source` and the second is silently skipped by +`ON CONFLICT DO NOTHING`. `save_eml` still writes both `.eml` copies, because the archive path +contains the source; `raw_ref` points at one of them. + +This is not corruption and it is the desired index behaviour: the content is indexed once. +It has two measurable effects worth knowing about **before** someone starts wondering about +the numbers — "new fastmail mails" reads systematically low by the shared part, and +`source` filtering in retrieval attributes such a mail to whichever account won the race. + +The recon's resolution (§2.4) is to keep the behaviour and make the phenomenon visible: +`envelopes_conflict_other_source`, one extra query per suppressed insert. Re-keying to +`(source, message_id)` was rejected — it breaks the frozen envelope contract, requires +rewriting 225 030 ids and every `raw_ref`, and buys duplicates in the index. + +## What this job does NOT do + +**It does not chunk and does not embed.** That split is the architecture, not an omission +(recon §3.2): fetching is network-bound and belongs on the 24/7 node; chunking and embedding +need Ollama on SOLARIA. Coupling them would mean mail only arrives when the desktop is on. + +The handoff needs no queue of its own — **an envelope with no `document_chunk` rows *is* the +queue**, drained by `mail-body-ingest --only-unchunked` from the `kb-ingest` timer. That is +self-healing in a way a timestamp cursor is not: an interrupted run, a mail fetched while the +GPU slept, a message inserted with a backdated header — each stays pending until it has +chunks, with nothing to reconcile. + +`--since` was explicitly rejected as the glue: it filters on `envelope.ts`, so a mail +delivered today with a month-old header date would fall outside the window and never be +chunked at all. + +## Headers at INSERT time + +New envelopes carry `entities[type=headers]` immediately, via +`kb_mail.headers.parse_headers_resilient` (typed parse, compat32 fallback). Historical +envelopes only have headers because a separate backfill job walked all 225 030 of them later. +Without this, `mail_body_ingest.build_prefix` would produce +`Temat: (brak tematu) | Od: ? | Data: …` for every new mail — with no error, just permanently +worse retrieval (recon §2.5 i). + +## Secrets + +Environment only (Decyzja (c)). There is no `--password` or `--user` flag: `--dsn ` +lands in `ps` output and shell history. Values go into the existing `/opt/homelab/kb/.env` +(root-owned `0600`), which systemd reads **as root before dropping to `User=oskar`** — so the +app passwords reach the process without being readable by `oskar` at rest. The repo ships only +`jobs/mail-imap-sync/env.example` with placeholders. + +## Stats must balance + +``` +uids_seen = processed + vanished + errors +processed = archived + archive_exists +processed = envelopes_inserted + envelopes_skipped_dup + envelopes_conflict_other_source +``` + +`headers_fallback`, `uidvalidity_resets`, `folder_errors` and `account_errors` are labels and +diagnostics, deliberately outside the equations — a folder that failed on EXAMINE never +produced a UID to account for. An unbalanced run is a failure regardless of what else it +reports. + +## Exit codes + +| Code | Meaning | +|---|---| +| 0 | Balanced, zero errors at any level | +| 1 | Any message/folder/account error, or an unbalanced sum | +| 2 | Configuration error — missing/malformed env, no DSN, no archive directory | + +Failures are isolated per account: a Gmail app password that stopped working must not also +stop Fastmail from syncing (the same stage isolation `cyclic_ingest` uses). + +## Modes + +| Invocation | Effect | +|---|---| +| *(default)* | Dry run: EXAMINE + SEARCH, report what a real tick would pull. No FETCH, no writes, no `.prom`. | +| `--apply` | The real thing, and the only mode that publishes metrics. | +| `--measure` | `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, then exit. Read-only, no DB. | +| `--limit N` | At most N messages per folder per tick — a brake for a first big sweep; the rest follows next tick. | + +## Metrics + +`/opt/homelab/state/node-exporter/kb-mail-sync.prom`, written atomically (tmp + rename) and +scraped by node_exporter's textfile collector on PIHA. + +| Metric | Notes | +|---|---| +| `kb_mail_sync_last_run_timestamp` | every run | +| `kb_mail_sync_last_success_timestamp` | carried forward across a failed run, so a blip does not reset the staleness clock | +| `kb_mail_sync_last_exit_code` | | +| `kb_mail_sync_envelopes_inserted{account}` | | +| `kb_mail_sync_envelopes_skipped_dup{account}` | | +| `kb_mail_sync_conflict_other_source{account}` | the cross-account overlap above | +| `kb_mail_sync_uidvalidity_resets{account}` | | +| `kb_mail_sync_errors{account}` | message + folder + account levels | +| `kb_mail_sync_last_message_ts{account}` | `max(envelope.ts)`; omitted for an account with no rows | + +One alert, `KbMailSyncStale` (`services/fleet-prometheus/rules/kb-mail-sync.yml`): 6 h without +a successful tick. **"No new mail for X days" was considered and rejected** (recon §3.4) — +zero new mail is a legal state of a mailbox, an alert that fires on a healthy system gets +muted, and a muted alert is not there on the day it matters. A per-account warning on +`kb_mail_sync_last_message_ts` may be worth adding after a month of observation, with a +threshold taken from measured behaviour rather than a guess. + +## Definition of Done + +Per `CLAUDE.md`: `pytest` passes (80 for this job, 111 for `kb-mail` including the adapter and +the cursor planner) plus a CLI smoke run. The tests drive the real `ImapClient` over a fake +`imaplib` connection, so what is covered is the adapter's actual protocol parsing: new +messages, UIDVALIDITY invalidation, dedup, resumption after an interrupted run, cross-account +collisions, account isolation, expunged UIDs, a dry run that touches nothing, and the metric +carry-forward. + +**Nothing here has ever connected to a live mailbox.** The first real sync is an operator +step, supervised, per `kb/runbooks/mail-sync-run.md`. + +## Follow-ups + +- `mail_imap_sync.prom` and `documents_ingest.cyclic_ingest`'s renderer are two copies of the + same idea (this one supports labels, that one does not). Worth folding into one shared + helper the next time either changes. +- `kb-mail-sync.timer` joins the "shadow-deploy family" — units installed outside GitOps drift + detection — listed in `hosts/piha/jobs.yaml`. That is open question 5 of + `kb/subsystems/recon-multiagent.md`; the list is now one item longer, knowingly. diff --git a/kb/services/kb-postgres.md b/kb/services/kb-postgres.md index 13846ae..994ae65 100644 --- a/kb/services/kb-postgres.md +++ b/kb/services/kb-postgres.md @@ -102,6 +102,10 @@ The `envelope` table is the frozen cross-source envelope (see `kb/subsystems/kb- Future migrations go in `init/` as `002_*.sql`, `003_*.sql`, …. Postgres runs `initdb` scripts only on a fresh volume — for existing instances apply migrations with `psql` directly. +Applied so far: `001` envelope, `002` document_chunk, `003` chunk model key + `excluded_reason`, `004` document_summary, `005` `mail_sync_state`. + +`005_mail_sync_state.sql` (2026-08-06) adds the per-folder IMAP sync cursor for `jobs/mail-imap-sync`, keyed `(account, folder)`. A table rather than a file under `/opt/homelab/state/` for one decisive reason: the cursor and the envelopes it describes must restore together or not at all. A state file surviving a DB restore would make the poller silently skip everything between the restored rows and the file's `last_uid` — a failure with no symptom. Apply it by hand on the live instance (`kb/runbooks/mail-sync-run.md` §4); the DDL is `IF NOT EXISTS`, so repeating it is safe. + ## Connection string ``` diff --git a/kb/services/pkg-kb-mail.md b/kb/services/pkg-kb-mail.md index 2a7e454..496c26b 100644 --- a/kb/services/pkg-kb-mail.md +++ b/kb/services/pkg-kb-mail.md @@ -3,15 +3,58 @@ okf: "0.1" type: service visibility: private status: active -updated: 2026-08-03 -stub: true -links: [] +updated: 2026-08-06 +links: + - job-mail-imap-sync.md + - job-mail-body-ingest.md + - job-gmail-bulk-import.md --- # kb-mail -Biblioteka wspoldzielona: model koperty KB, helpery asyncpg do bazy, append-only archiwum .eml. +Shared library for the mail pillar. Everything that more than one mail job needs lives here, +by rule rather than by taste: the corpus rests on 225 030 envelopes whose ids, archive paths +and header entities were produced by one implementation, and a second copy of any of those +derivations would silently stop agreeing with the first. -> **Stub.** Katalog z kodem nie ma README — ten dokument powstal automatycznie w migracji do KB OKF (etap 2). Opis jednozdaniowy wyciagniety z: CLAUDE.md, sekcja `Shared Python Libraries (packages/)`. Pelna dokumentacja do napisania. +**Kod:** `packages/kb-mail/` (`src/kb_mail/`, `pyproject.toml`, `tests/`) -**Kod:** `packages/kb-mail/ (src/, pyproject.toml, tests/)` +Dependencies: `asyncpg`, `structlog`. Nothing else — the IMAP transport is stdlib `imaplib`. + +## Modules + +| Module | What it holds | Used by | +|---|---|---| +| `envelope` | The frozen cross-source `Envelope` dataclass (mirrors `envelope` in kb-postgres). Additive contract. | everything | +| `db` | asyncpg helpers: `insert_envelope` (returns the command tag), `get_envelope`, `rows_affected`, `envelope_source` | all jobs | +| `archive` | `save_eml` — append-only `.eml` writer, `{root}/{source}/{YYYY}/{MM}/{id}.eml`. `FileExistsError` is the "already have it" signal, never masked. | bulk-import, imap-sync | +| `text` | `sanitize_surrogates`, `strip_nul` — both mandatory before anything reaches `text`/`jsonb` | all parsing paths | +| `chunking` | `chunk_text` (2400/600 chars), extracted in Krok 0 | mail-body-ingest, documents-ingest | +| `headers` | `parse_headers`, `parse_headers_fallback`, `parse_headers_resilient` — the `entities[type=headers]` shape, typed parse with a compat32 fallback | header-backfill, imap-sync | +| `message` | `message_id`, `parse_date`, `parse_attachments`, `eml_ref`, `EPOCH` — envelope id, timestamp, attachment manifest, archive path | bulk-import, imap-sync | +| `imap` | `ImapAccount`, `ImapClient`, `FolderStatus` — read-only IMAP over stdlib `imaplib` | imap-sync | +| `sync_state` | `mail_sync_state` accessors + `plan_folder_sync` / `contiguous_last_uid` | imap-sync | + +`headers` and `message` were extracted from `gmail-header-backfill` and `gmail-bulk-import` +respectively when `mail-imap-sync` needed the same derivations (2026-08-06). Both jobs +re-export them under their original names, so their CLIs and test suites are unchanged. This +is the same move as `chunking` in Krok 0 — extract, do not copy. + +## Why the IMAP transport lives here and not in the job + +The adapter is protocol code, and the sync-cursor semantics (UIDVALIDITY invalidation, how far +a cursor may advance) are protocol semantics — both are worth testing in isolation, and both +would otherwise be buried in a job's loop where a wrong branch is invisible. `kb_mail.imap` and +`kb_mail.sync_state` therefore carry the rules; `jobs/mail-imap-sync` carries the orchestration +and the policy. See `kb/services/job-mail-imap-sync.md` for the behaviour they add up to. + +## Tests + +```bash +pip install -e "packages/kb-mail[dev]" +cd packages/kb-mail && pytest +``` + +111 unit tests, no DB or network. `tests/test_db.py` is additionally marked `integration` and +needs a live kb-postgres (`KB_TEST_DSN`, default `localhost:5433`) — it is the only file that +does, and it errors rather than skips when the DB is absent. diff --git a/kb/subsystems/kb-mail-pillar.md b/kb/subsystems/kb-mail-pillar.md index 9db7766..d686845 100644 --- a/kb/subsystems/kb-mail-pillar.md +++ b/kb/subsystems/kb-mail-pillar.md @@ -3,7 +3,7 @@ okf: "0.1" type: subsystem visibility: private status: active -updated: 2026-06-24 +updated: 2026-08-06 links: [] --- @@ -27,9 +27,22 @@ Realna potrzeba to najpierw **archiwum**, nie „RAG nad mailami". Dwie warstwy, Gmail **nie jest porzucany** — zostaje jako konto śmieciowe / loginy / 2FA. Stąd maile mają dwa żywe wejścia: -- **Fastmail** — `source: fastmail`, adapter **JMAP** (read-only token). Primary: tu ląduje sensowna poczta na przyszłość. +- **Fastmail** — `source: fastmail`, adapter **IMAP** (hasło aplikacji). Primary: tu ląduje sensowna poczta na przyszłość. - **Gmail** — `source: gmail`, adapter **IMAP** (protokół, nie Gmail API → przenośność). Ciągły sync żywej skrzynki. +> **Korekta 2026-08-06 — Fastmail przez IMAP, nie JMAP.** Do tej daty ten dokument (i §7, §9 +> oraz §10 planu fazy mailowej) przewidywał dla Fastmaila **JMAP** i osobny `jobs/fastmail-poller`. +> Zapis historyczny: *„Fastmail — adapter JMAP (read-only token)"*, 2026-06-24. +> +> Decyzja z 2026-08-06 (recon `kb/audits/mail-sync-2026-08-06.md` Decyzja (b), zatwierdzona +> przez operatora) domyka otwartą od czerwca decyzję „unifikacja adaptera" z §9 na rzecz +> **jednego wspólnego IMAP-a dla obu kont**, w jednym jobie `jobs/mail-imap-sync`. Powody: +> JMAP synchronizuje po `state` — elegancko i niepotrzebnie przy jednym ticku na godzinę +> i ~37 mailach na dobę, skoro UIDVALIDITY/UIDNEXT rozwiązuje ten sam problem i tak trzeba go +> zaimplementować dla Gmaila; jeden adapter to jeden zestaw testów, jedna klasa błędów i jedna +> ścieżka hardeningu 8-bitowych nagłówków. JMAP nie jest zamknięty na zawsze — koperta +> i archiwum są protokołowo obojętne, więc wymiana transportu nie dotyka danych. + Plus jednorazowy **bulk historyczny Gmaila** (eksport „All Mail" / Takeout → surowy dump do archiwum). Operacja odwracalna i niezależna od reszty pipeline'u — robimy pierwsza. Urgency spadła (konto żyje), ale historia warta zassania od razu. --- @@ -81,7 +94,9 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki). ## 7. Deploy - Wszystko w `homelab-codex`, przez Git na SATURN, konwencja override `hosts//runtime//`. -- Usługi: `jmap-poller` (Fastmail), `imap-poller` (Gmail), `indexer`, embed (ollama na SOLARIA), `postgres+pgvector`, `mail-agent`; bulk importer jako one-shot job. +- Usługi: `mail-imap-sync` (Fastmail + Gmail, jeden job — korekta 2026-08-06; wcześniej + planowane jako osobne `jmap-poller` + `imap-poller`), `indexer`, embed (ollama na SOLARIA), + `postgres+pgvector`, `mail-agent`; bulk importer jako one-shot job. - Deploy skryptem czytającym `inventory/topology.yaml`. --- @@ -90,8 +105,10 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki). 1. ✅ Zamroź kopertę + postaw Postgres+pgvector. *(2026-06-17)* 2. ✅ **Bulk Gmail historyczny → archiwum** — `jobs/gmail-bulk-import/` — **KOD GOTOWY** *(2026-06-24)*; nie uruchomiony (Takeout ~27 GB na SOLARIA, do transferu na PIHA). -3. Fastmail JMAP live ingest → archiwum. -4. Gmail IMAP live sync → archiwum. +3. ~~Fastmail JMAP live ingest~~ → **Fastmail IMAP live sync** → archiwum. *(korekta + 2026-08-06; kod gotowy, pierwszy żywy run po stronie operatora — + `kb/runbooks/mail-sync-run.md`)* +4. Gmail IMAP live sync → archiwum. *(j.w. — ten sam job `jobs/mail-imap-sync`)* 5. Filtr archiwum→indeks. 6. Indexer (parse → chunk → embed bge-m3) → pgvector. 7. Cienki agent maili + tool dla warstwy 4. @@ -100,8 +117,12 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki). ## 9. Decyzje otwarte (do przyklepania przed/w trakcie startu) -- **Sizing Gmaila** — ile realnie waży „All Mail"? (przesądza node/dysk archiwum). -- **Unifikacja adaptera** — jeden wspólny IMAP dla Fastmail + Gmail (mniej kodu) vs JMAP dla Fastmail + IMAP dla Gmail (JMAP bogatszy)? +- ✅ **Sizing Gmaila** — ZAMKNIĘTE: 225 030 kopert, archiwum ~27 GB na PIHA (Etap B, 2026-08-06). +- ✅ **Unifikacja adaptera** — ZAMKNIĘTE 2026-08-06 na rzecz **jednego IMAP-a** dla obu kont + (Decyzja (b) reconu, uzasadnienie w §2 wyżej). +- **Sizing Fastmaila** — OTWARTE, i celowo: przesądza o tym, czy ciągniemy historię konta czy + tylko przyrost. Rozstrzyga pomiar `mail-imap-sync --measure`, nie zgadywanie — + `kb/runbooks/mail-sync-run.md` §5. - **Reguły filtra** — startowa lista blacklist domen/nagłówków. - Vector store: pgvector **przyklepane** (spine). - Embed model: bge-m3 **przyklepane**. diff --git a/services/fleet-prometheus/rules/kb-mail-sync.yml b/services/fleet-prometheus/rules/kb-mail-sync.yml new file mode 100644 index 0000000..4deb13d --- /dev/null +++ b/services/fleet-prometheus/rules/kb-mail-sync.yml @@ -0,0 +1,40 @@ +# fleet-prometheus kb-mail-sync rules — faza mailowa Krok 7 +# (recon kb/audits/mail-sync-2026-08-06.md §3.4). +# +# Same delivery convention as kb-ingest.yml and liveness.yml: no Alertmanager. These rules +# only make alerts FIRING (visible at GET /api/v1/alerts on this Prometheus instance); +# brain-watchdog@PIHA polls that API and forwards to Telegram. Do NOT add Alertmanager here. +# +# Metrics come from jobs/mail-imap-sync's textfile output +# (/opt/homelab/state/node-exporter/kb-mail-sync.prom), scraped by node_exporter on PIHA — +# every series from that target already carries node="piha" (fleet-node job's static label). +# +# Until the operator installs and enables kb-mail-sync.timer the metric does not exist, and +# `time() - ` produces no result — so this file is inert on a fleet where the +# job has not been set up. It does not need to be deployed in lockstep with the job. + +groups: + - name: kb-mail-sync + rules: + - alert: KbMailSyncStale + # 6 missed hourly ticks. This is the ONE alert this job gets, and deliberately so: + # it answers "is the poller working at all", which has no false positives — unlike + # any threshold on how much mail arrived. + # + # "No new mail for X days" was considered and REJECTED (recon §3.4). Zero new mail is + # a legal state of a mailbox — a holiday, a weekend, traffic moved to the other + # account. An alert that fires on a healthy system gets muted, and a muted alert is + # not there on the day it is needed. + expr: time() - kb_mail_sync_last_success_timestamp{node="piha"} > 21600 + for: 5m + labels: + severity: critical + annotations: + summary: "kb-mail-sync stale on {{ $labels.node }}" + description: "kb-mail-sync.timer has not completed a successful run in over 6h on {{ $labels.node }} — the mail corpus has stopped growing. Check `journalctl -u kb-mail-sync.service` and the latest /opt/homelab/logs/kb-mail-sync/run-*.log; an expired or revoked IMAP app password is the most likely cause." + + # Deliberately NOT defined here, pending a month of observation (recon §3.4): + # a per-account warning on kb_mail_sync_last_message_ts (age of the newest mail). + # It needs a per-account threshold — gmail is the junk account and days of silence + # there would be odd, while for fastmail it would not be — and that threshold should + # come from measured behaviour, not from a guess made before the first run.