oskar/homelab-codex-ws
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

feat(mail-sync): scheduler PIHA, takt kb-ingest, runbook i dokumentacja

Browse source 
Domkniecie Kroku 7. Realizuje Decyzje (d) reconu (host schedulera + korekta
kadencji indeksowania) i doklada dokumentacje wg konwencji OKF.

Scheduler (NIEAKTYWOWANY — wlacza operator):
- jobs/mail-imap-sync/systemd/{service,timer,run.sh} — wzorzec 1:1 z kb-ingest,
  OnCalendar=hourly, Persistent=true, log do pliku (nigdy sam journal).
- hosts/piha/jobs.yaml — deklaracja jednostek host-level na PIHA. Nowy plik, bo
  services.yaml jest dla kontenerow (supervisor dopasowuje jego wpisy do world-state
  i wpis niekontenerowy dryfowalby wiecznie jako missing_service). Nic tego pliku
  nie czyta — istnieje po to, zeby "shadow-deploy family" z otwartego pytania 5
  reconu multiagentowego byla spisana, a nie tylko na nodzie.

Takt indeksowania (Decyzja (d), recon §3.3):
- kb-ingest.timer: 03:30 raz na dobe -> co 2 h. O 03:30 SOLARIA prawie na pewno spi
  (potwierdzone odczytem kb_ingest_embed_skipped 1 z 2026-08-06), a tick dostaje
  teraz etap mailowy: ~60 nowych chunkow na dobe pomijanych kazdej nocy sprawiloby,
  ze backlog rosnie monotonicznie i KbEmbedBacklogGrowing zapala sie NA STALE.
  Co 2 h zamiast stalej godziny — probe Ollamy sam wybiera okno, wiec ktorys tick
  w nie trafi niezaleznie od nawykow operatora.
- cyclic_ingest: etap mailowy (mail_body_ingest --only-unchunked), import miekki,
  wiec venv bez tego pakietu pomija etap zamiast wywracac wrapper. Predykat bledu
  JEST luzniejszy niz wlasne main() tamtego joba i to jedyne takie miejsce w tym
  wrapperze: pojedynczy trwale nieparsowalny mail nie moze zamrozic
  last_success_timestamp i zapalic KbIngestStale na zawsze. Bledy per-mail sa
  publikowane jako kb_ingest_mail_parse_errors, nie chowane.

Obserwowalnosc: KbMailSyncStale (6 h bez udanego ticku). Alert na cisze w skrzynce
ODRZUCONY (decyzja operatora, zgodna z reconem §3.4) — zero nowych maili to legalny
stan skrzynki, a alert zapalajacy sie na zdrowym systemie zostaje wyciszony
i przestaje dzialac wtedy, gdy jest potrzebny.

Dokumentacja:
- kb/services/job-mail-imap-sync.md (OKF), kb/runbooks/mail-sync-run.md — 9 krokow
  pierwszego uruchomienia, w tym checklista 4 punktow [do weryfikacji na zywo]
  z reconu (polityki dostawcow — do sprawdzenia, nie do zgadniecia) oraz pomiar
  STATUS (MESSAGES) na Fastmailu, na ktorym zapada ODLOZONA decyzja o historii.
- kb-mail-pillar.md: KOREKTA JMAP -> IMAP dla Fastmaila jako decyzja 2026-08-06;
  stary zapis zostaje jako historia z data. Zamkniete "unifikacja adaptera"
  i "sizing Gmaila"; otwarte zostaje "sizing Fastmaila" — celowo, bo rozstrzyga
  je pomiar, nie dyskusja.
- kb-m5-faza-mailowa.md: Krok 7 IN PROGRESS + tabela zakresu wdrozonego,
  kb-m5-faza3.md: korekta harmonogramu i sekwencji wrappera,
  pkg-kb-mail.md: rozpisany ze stubu, kb-postgres.md: lista migracji + 005.

Testy: 642 passed (calosc kb-mail, kb-retrieval i jobs). systemd-analyze verify
na timerze przechodzi, OnCalendar=0/2:00:00 normalizuje sie do co 2 h.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
oskar 2026-08-06 14:49:58 +02:00
parent f056b08574
commit ae16deb8d3
16 changed files with 1234 additions and 41 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

48
hosts/piha/jobs.yaml Normal file
View file

@ -0,0 +1,48 @@
# Host-level systemd units on PIHA — declaration only.
#
# These are NOT docker-compose services, so they do not belong in services.yaml (whose
# entries the supervisor matches against world-state service keys; adding a non-container
# entry there would drift forever as missing_service). Nothing reads this file: it exists so
# the units installed outside the compose pipeline are written down in the repo rather than
# living only on the node and in a runbook.
#
# This is the "shadow-deploy family" the multiagent recon flags in its open question 5
# (kb/subsystems/recon-multiagent.md) — units installed outside GitOps drift detection.
# kb-mail-sync joins that list knowingly, not by oversight. When question 5 is settled, this
# file is the inventory to settle it against.
#
# Installation and activation are always operator steps. A `git pull` never starts a timer.
host: piha
systemd_units:
kb-ingest:
unit: kb-ingest.timer
service: kb-ingest.service
source: jobs/documents-ingest/systemd/
schedule: "0/2:00:00" # every 2 h — see the note in the .timer file
runs_as: oskar
environment_file: /opt/homelab/kb/.env
log_path: /opt/homelab/logs/kb-ingest/
metrics: /opt/homelab/state/node-exporter/kb-ingest.prom
state: active # installed and enabled since 2026-07-30
description: >
Cyclic KB ingest: paperless adapter, chunk+embed, summarize, embed summaries, and
(since 2026-08-06) the mail body stage that drains the unchunked-envelope queue
mail-imap-sync fills. Both embed stages are gated on an Ollama@SOLARIA probe.
kb-mail-sync:
unit: kb-mail-sync.timer
service: kb-mail-sync.service
source: jobs/mail-imap-sync/systemd/
schedule: hourly
runs_as: oskar
environment_file: /opt/homelab/kb/.env
log_path: /opt/homelab/logs/kb-mail-sync/
metrics: /opt/homelab/state/node-exporter/kb-mail-sync.prom
state: declared # NOT installed, NOT enabled — operator activates
runbook: kb/runbooks/mail-sync-run.md
description: >
Incremental IMAP fetch for gmail (\All) and fastmail (INBOX, Archive, Sent) into the
.eml archive and the envelope table. Network-bound, no GPU — which is why it lives on
PIHA (24/7) and is decoupled from the embedding stages on SOLARIA (~16 h/day off).

135
jobs/documents-ingest/src/documents_ingest/cyclic_ingest.py
View file

@ -3,9 +3,19 @@
paperless_adapter.run() -- new source='paperless' envelopes
-> chunk_embed.run() -- new document_chunk rows (bge-m3 via Ollama@SOLARIA)
-> mail_body_ingest.run(only_unchunked=True) -- chunks+embeds mail the incremental
IMAP sync fetched (faza mailowa Krok 7)
-> summarize.run_summarize(backend='anthropic') -- new document_summary rows
-> summarize.run_embed_summaries() -- embeds those summaries (bge-m3)
The mail stage is the consumer end of `jobs/mail-imap-sync`, which runs on its own hourly
timer and deliberately does no GPU work (recon kb/audits/mail-sync-2026-08-06.md §3.2). The
handoff between them needs no queue of its own: an envelope with no `document_chunk` rows IS
the queue, so an interrupted run, a mail fetched while SOLARIA slept, or one inserted with a
backdated header all stay pending until they have chunks, with no state to reconcile. The
stage is soft-imported -- a venv without `mail-body-ingest` installed skips it and says so,
rather than failing to start the whole wrapper.
The two embed stages are gated on a pre-flight Ollama reachability probe
(`GET /api/tags`) because SOLARIA has `availability_target: medium` (planned power-off,
plan §1.3): "Ollama unreachable" is skipped, not failed -- new chunks/summaries are picked
@ -50,6 +60,11 @@ import structlog
from documents_ingest import chunk_embed, paperless_adapter, summarize
try: # optional: only installed in venvs that also run the mail pipeline
from mail_body_ingest import ingest as mail_body_ingest
except ImportError: # pragma: no cover - exercised by the "not installed" test via monkeypatch
mail_body_ingest = None
_log = structlog.get_logger(__name__)
# Wrapper always runs on PIHA -- Ollama lives on SOLARIA, reached over Tailscale (plan §7.1).
@ -60,6 +75,10 @@ DEFAULT_SUMMARY_MODEL = "claude-haiku-4-5"
DEFAULT_EMBED_MODEL = "bge-m3"
DEFAULT_OLLAMA_PROBE_TIMEOUT = 5.0
DEFAULT_PROM_PATH = Path("/opt/homelab/state/node-exporter/kb-ingest.prom")
# Mail sources whose envelopes this wrapper chunks+embeds. The archive root is PIHA's
# canonical one -- this wrapper only ever runs there.
DEFAULT_MAIL_SOURCES = ("gmail", "fastmail")
DEFAULT_MAIL_ARCHIVE_ROOT = Path("/home/oskar/kb/mail/archive")
_BACKLOG_QUERY = (
"SELECT count(*) AS n FROM document_chunk WHERE excluded_reason IS NULL AND embedding IS NULL"
@ -95,6 +114,19 @@ PROM_METRIC_HELP: dict[str, tuple[str, str]] = {
"gauge",
"Active document_chunk rows (excluded_reason IS NULL) still missing an embedding.",
),
"kb_ingest_mail_chunks_inserted": (
"gauge", "New document_chunk rows inserted by the mail body stage in the last run.",
),
"kb_ingest_mail_parse_errors": (
"gauge",
"Mails the body stage could not read or parse in the last run. Tolerated (they do "
"not fail the wrapper) but tracked -- a rising number is a real signal.",
),
"kb_ingest_mail_skipped": (
"gauge",
"1 if the mail body stage was skipped this run (Ollama unreachable, the "
"mail-body-ingest package not installed, or no .eml archive on this host).",
),
}
_PROM_LINE_RE = re.compile(r"^(kb_ingest_\w+)\s+([0-9.eE+-]+)\s*$")
@ -139,6 +171,40 @@ def _embed_summaries_failed(stats: dict) -> bool:
return stats["errors"] > 0 or not balanced
def _mail_body_failed(stats: dict) -> bool:
"""Deliberately NOT a 1:1 mirror of mail_body_ingest.main() -- the one predicate in this
wrapper that is looser than its job's own.
That CLI exits 1 on any `parse_errors`/`missing_file`/`read_errors`, which is right for a
supervised slice run: across 225k mails a handful is expected and worth a look. On a
recurring unattended tick it would be corrosive. A single permanently-unparseable mail
would mark every run failed, `kb_ingest_last_success_timestamp` would stop advancing, and
KbIngestStale would fire after 48h and stay lit -- reporting nothing but one bad mail
nobody can fix. That is the same "an alert that always fires is not an alert" failure the
schedule change in kb-ingest.timer exists to avoid.
So per-mail failures are tolerated and published as `kb_ingest_mail_parse_errors` instead.
What still fails the wrapper is what means the stage itself is broken: unbalanced stats,
or `chunks_conflict_skipped` (an insert silently no-op'd -- a real defect, not bad input).
"""
balanced = (
stats["mails_scanned"] == (
stats["missing_file"] + stats["read_errors"] + stats["parse_errors"]
+ stats["body_empty"] + stats["mails_chunked"]
)
and stats["chunks_total"] == (
stats["chunks_inserted"] + stats["chunks_newsletter_flagged"]
+ stats["chunks_already_embedded"] + stats["chunks_conflict_skipped"]
+ stats["chunks_errors"]
)
)
return stats["chunks_conflict_skipped"] > 0 or not balanced
def _mail_parse_errors(stats: dict) -> int:
return stats["missing_file"] + stats["read_errors"] + stats["parse_errors"]
async def probe_ollama(ollama_url: str, timeout: float = DEFAULT_OLLAMA_PROBE_TIMEOUT) -> bool:
"""`GET /api/tags` reachability check (plan §7.1). Any exception or non-2xx means
"treat as offline, skip the embed stages this tick" -- never raises."""
@ -178,11 +244,14 @@ async def run_cyclic(
tags_vocab_path: Path = summarize.DEFAULT_TAGS_VOCAB_PATH,
apply: bool = False,
ollama_probe_timeout: float = DEFAULT_OLLAMA_PROBE_TIMEOUT,
mail_sources: tuple[str, ...] = DEFAULT_MAIL_SOURCES,
mail_archive_root: Path = DEFAULT_MAIL_ARCHIVE_ROOT,
) -> dict:
"""Runs all four stages, isolating failures per stage (an earlier failure never skips
"""Runs all five stages, isolating failures per stage (an earlier failure never skips
a later stage). Returns a dict with one entry per stage (`adapter`, `chunk_embed`,
`summarize`, `embed_summaries`, each `_stage_result()`-shaped), plus `ollama_up`,
`embed_backlog`, and the aggregate `failed` bool the caller should exit non-zero on."""
`mail_body`, `summarize`, `embed_summaries`, each `_stage_result()`-shaped), plus
`ollama_up`, `embed_backlog`, and the aggregate `failed` bool the caller should exit
non-zero on."""
ollama_up = await probe_ollama(ollama_url, timeout=ollama_probe_timeout)
adapter_stage = _stage_result()
@ -208,6 +277,39 @@ async def run_cyclic(
else:
_log.warning("skip.chunk_embed", reason="ollama_unreachable")
# Mail body stage -- the consumer end of the incremental sync (recon §3.2). It drains the
# queue that IS the data: envelopes with no document_chunk rows. Nothing to reconcile and
# nothing to reset, so a tick that skips it (GPU asleep) costs only latency.
mail_stage = _stage_result(skipped=True)
mail_skip_reason = None
if not mail_sources:
mail_skip_reason = "no_mail_sources_configured"
elif not ollama_up:
mail_skip_reason = "ollama_unreachable"
elif mail_body_ingest is None:
mail_skip_reason = "mail_body_ingest_not_installed"
elif not Path(mail_archive_root).is_dir():
mail_skip_reason = "archive_root_missing"
if mail_skip_reason is None:
try:
stats = await mail_body_ingest.run(
dsn=dsn,
archive_root=Path(mail_archive_root),
ollama_url=ollama_url,
model=embed_model,
sources=tuple(mail_sources),
only_unchunked=True,
apply=apply,
)
mail_stage = _stage_result(stats=stats, failed=_mail_body_failed(stats))
except Exception as exc:
_log.error("mail_body_exception", error=str(exc))
mail_stage = _stage_result(failed=True, error=str(exc))
else:
_log.warning("skip.mail_body", reason=mail_skip_reason)
mail_stage = _stage_result(skipped=True, error=None)
summarize_stage = _stage_result()
try:
stats = await summarize.run_summarize(
@ -241,13 +343,16 @@ async def run_cyclic(
failed = any(
stage["failed"]
for stage in (adapter_stage, chunk_stage, summarize_stage, embed_summaries_stage)
for stage in (adapter_stage, chunk_stage, mail_stage, summarize_stage,
embed_summaries_stage)
)
return {
"ollama_up": ollama_up,
"adapter": adapter_stage,
"chunk_embed": chunk_stage,
"mail_body": mail_stage,
"mail_skip_reason": mail_skip_reason,
"summarize": summarize_stage,
"embed_summaries": embed_summaries_stage,
"embed_backlog": embed_backlog,
@ -285,6 +390,11 @@ def build_metrics(result: dict, now_ts: float, prom_path: Path) -> dict:
"kb_ingest_chunks_inserted": (result["chunk_embed"]["stats"] or {}).get("chunks_inserted", 0),
"kb_ingest_summaries_inserted": (result["summarize"]["stats"] or {}).get("summarized", 0),
"kb_ingest_embed_skipped": 0 if result["ollama_up"] else 1,
"kb_ingest_mail_chunks_inserted":
(result["mail_body"]["stats"] or {}).get("chunks_inserted", 0),
"kb_ingest_mail_parse_errors":
_mail_parse_errors(result["mail_body"]["stats"]) if result["mail_body"]["stats"] else 0,
"kb_ingest_mail_skipped": 1 if result["mail_body"]["skipped"] else 0,
}
if result["embed_backlog"] is not None:
metrics["kb_ingest_embed_backlog"] = result["embed_backlog"]
@ -335,6 +445,17 @@ def main() -> None:
help=f"Ollama embedding model, chunks+summaries (default: {DEFAULT_EMBED_MODEL})")
parser.add_argument("--tags-vocab", type=Path, default=summarize.DEFAULT_TAGS_VOCAB_PATH,
help="Path to tags-vocab.yaml")
parser.add_argument("--mail-sources",
default=os.environ.get("KB_MAIL_SOURCES", ",".join(DEFAULT_MAIL_SOURCES)),
metavar="A,B",
help=f"Envelope sources the mail body stage drains "
f"(default: {','.join(DEFAULT_MAIL_SOURCES)}, or set KB_MAIL_SOURCES). "
f"Empty disables the stage.")
parser.add_argument("--mail-archive-root", type=Path,
default=Path(os.environ.get("MAIL_ARCHIVE_ROOT",
str(DEFAULT_MAIL_ARCHIVE_ROOT))),
help=f"Mail .eml archive root (default: {DEFAULT_MAIL_ARCHIVE_ROOT}). "
f"The stage is skipped if this directory does not exist.")
parser.add_argument("--prom-path", type=Path,
default=Path(os.environ.get("KB_INGEST_PROM_PATH", str(DEFAULT_PROM_PATH))),
help=f"Textfile-collector output path (default: {DEFAULT_PROM_PATH})")
@ -365,6 +486,8 @@ def main() -> None:
tags_vocab_path=args.tags_vocab,
apply=args.apply,
ollama_probe_timeout=args.ollama_probe_timeout,
mail_sources=tuple(s.strip() for s in args.mail_sources.split(",") if s.strip()),
mail_archive_root=args.mail_archive_root,
))
now_ts = time.time()
@ -382,6 +505,10 @@ def main() -> None:
chunk_embed=result["chunk_embed"]["stats"],
chunk_embed_skipped=result["chunk_embed"]["skipped"],
chunk_embed_error=result["chunk_embed"]["error"],
mail_body=result["mail_body"]["stats"],
mail_body_skipped=result["mail_body"]["skipped"],
mail_skip_reason=result["mail_skip_reason"],
mail_body_error=result["mail_body"]["error"],
summarize=result["summarize"]["stats"],
summarize_error=result["summarize"]["error"],
embed_summaries=result["embed_summaries"]["stats"],

30
jobs/documents-ingest/systemd/kb-ingest.timer
View file

@ -1,14 +1,32 @@
# kb-ingest.timer — module 5 phase 3 step 5 (kb/phases/kb-m5-faza3.md §7.1).
# Daily at 03:30, not hourly: the plan fixes this schedule explicitly (low-traffic window,
# and the underlying jobs are full-corpus re-scans each tick — cheap at 186 documents
# today, but daily keeps headroom as the corpus grows). Persistent=true catches up after a
# PIHA reboot instead of silently skipping the day's run.
#
# Every 2 hours since 2026-08-06. It used to be daily at 03:30, which was right when the only
# producer was Paperless: full-corpus re-scans, 186 documents, and a low-traffic window.
#
# Two facts made that schedule wrong (recon kb/audits/mail-sync-2026-08-06.md §3.3, operator
# Decyzja (d)):
# 1. SOLARIA is powered off ~16 h/day BY DESIGN, and 03:30 falls inside that window.
# Measured on PIHA 2026-08-06: `kb_ingest_embed_skipped 1` — that morning's tick
# skipped both embed stages because the Ollama probe got no answer.
# 2. From 2026-08-06 this tick also drains the mail queue that kb-mail-sync fills: roughly
# 60 new chunks a day. Skipped every single night, the embed backlog would grow
# monotonically and KbEmbedBacklogGrowing (`> 0` for 72h) would light up permanently —
# signalling nothing but a schedule that never coincides with the GPU being on. An
# alert that is always firing has stopped being an alert.
#
# Two hours, rather than a fixed hour chosen to match when SOLARIA is usually up: the Ollama
# probe already picks the working window on its own, so some tick lands in it whatever the
# operator's habits are that week. Ticks that miss it skip the embed stages and cost nothing;
# the write stages are idempotent and the queue is derived from the data.
#
# Persistent=true catches up after a PIHA reboot instead of silently skipping a run.
[Unit]
Description=Run kb-ingest.service daily (module 5 phase 3 step 5)
Description=Run kb-ingest.service every 2 hours (module 5 phase 3 step 5)
[Timer]
OnCalendar=*-*-* 03:30:00
OnCalendar=0/2:00:00
Persistent=true
RandomizedDelaySec=300
[Install]
WantedBy=timers.target

152
jobs/documents-ingest/tests/test_cyclic_ingest.py
View file

@ -166,11 +166,16 @@ SUMMARIZE_OK = {"documents_fetched": 1, "duplicates_skipped": 0, "no_active_chun
"already_summarized": 0, "summarized": 1, "llm_errors": 0,
"documents_mapreduce": 0, "tags_truncated": 0}
EMBED_SUMMARIES_OK = {"summaries_fetched": 1, "embedded": 1, "errors": 0}
MAIL_BODY_OK = {"mails_scanned": 1, "missing_file": 0, "read_errors": 0, "parse_errors": 0,
"body_empty": 0, "mails_chunked": 1, "chunks_total": 2, "chunks_inserted": 2,
"chunks_newsletter_flagged": 0, "chunks_already_embedded": 0,
"chunks_conflict_skipped": 0, "chunks_errors": 0}
def _patch_stages(monkeypatch, *, ollama_up=True, adapter=None, chunk=None, summarize=None,
embed_summaries=None, backlog=0):
calls = {"adapter": 0, "chunk_embed": 0, "summarize": 0, "embed_summaries": 0}
embed_summaries=None, mail_body=None, backlog=0, mail_archive=True):
calls = {"adapter": 0, "chunk_embed": 0, "summarize": 0, "embed_summaries": 0,
"mail_body": 0}
async def _fake_probe(url, timeout=5.0):
return ollama_up
@ -195,9 +200,22 @@ def _patch_stages(monkeypatch, *, ollama_up=True, adapter=None, chunk=None, summ
calls["embed_summaries"] += 1
return embed_summaries if embed_summaries is not None else dict(EMBED_SUMMARIES_OK)
async def _fake_mail_body_run(**kwargs):
calls["mail_body"] += 1
if mail_body is not None and isinstance(mail_body, Exception):
raise mail_body
return mail_body if mail_body is not None else dict(MAIL_BODY_OK)
async def _fake_backlog(dsn):
return backlog
class _FakeMailBodyModule:
run = staticmethod(_fake_mail_body_run)
monkeypatch.setattr(cyclic_ingest, "mail_body_ingest", _FakeMailBodyModule)
# The archive root check is a real filesystem probe; pin it so the suite does not depend
# on whether the host running the tests happens to have a mail archive.
monkeypatch.setattr(cyclic_ingest.Path, "is_dir", lambda self: mail_archive)
monkeypatch.setattr(cyclic_ingest, "probe_ollama", _fake_probe)
monkeypatch.setattr(cyclic_ingest.paperless_adapter, "run", _fake_adapter_run)
monkeypatch.setattr(cyclic_ingest.chunk_embed, "run", _fake_chunk_run)
@ -216,7 +234,8 @@ class TestRunCyclic:
)
assert result["failed"] is False
assert result["ollama_up"] is True
assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1}
assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1,
"mail_body": 1}
assert result["chunk_embed"]["skipped"] is False
assert result["embed_summaries"]["skipped"] is False
assert result["embed_backlog"] == 0
@ -238,6 +257,107 @@ class TestRunCyclic:
assert result["chunk_embed"]["stats"] is None
assert result["embed_summaries"]["skipped"] is True
async def test_mail_stage_drains_only_unchunked_envelopes(self, monkeypatch):
# The handoff from mail-imap-sync is the unchunked-envelope queue, not a timestamp.
captured = {}
async def _capture(**kwargs):
captured.update(kwargs)
return dict(MAIL_BODY_OK)
_patch_stages(monkeypatch, ollama_up=True)
monkeypatch.setattr(cyclic_ingest.mail_body_ingest, "run", staticmethod(_capture))
await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert captured["only_unchunked"] is True
assert captured["sources"] == ("gmail", "fastmail")
assert captured["apply"] is True
async def test_mail_stage_is_skipped_when_ollama_is_down(self, monkeypatch):
calls = _patch_stages(monkeypatch, ollama_up=False)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert calls["mail_body"] == 0
assert result["mail_body"]["skipped"] is True
assert result["mail_skip_reason"] == "ollama_unreachable"
assert result["failed"] is False
async def test_mail_stage_is_skipped_when_the_package_is_not_installed(self, monkeypatch):
# PIHA's venv may predate mail-body-ingest; that must not stop the whole wrapper.
calls = _patch_stages(monkeypatch, ollama_up=True)
monkeypatch.setattr(cyclic_ingest, "mail_body_ingest", None)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_skip_reason"] == "mail_body_ingest_not_installed"
assert result["failed"] is False
assert calls["adapter"] == 1
async def test_mail_stage_is_skipped_without_an_archive(self, monkeypatch):
_patch_stages(monkeypatch, ollama_up=True, mail_archive=False)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_skip_reason"] == "archive_root_missing"
assert result["failed"] is False
async def test_mail_stage_is_skipped_when_no_sources_are_configured(self, monkeypatch):
_patch_stages(monkeypatch, ollama_up=True)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True, mail_sources=(),
)
assert result["mail_skip_reason"] == "no_mail_sources_configured"
async def test_per_mail_parse_errors_do_not_fail_the_wrapper(self, monkeypatch):
# Looser than mail-body-ingest's own CLI predicate, on purpose: one permanently
# unparseable mail must not keep last_success frozen and light KbIngestStale forever.
flaky = dict(MAIL_BODY_OK, mails_scanned=2, parse_errors=1)
_patch_stages(monkeypatch, ollama_up=True, mail_body=flaky)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_body"]["failed"] is False
assert result["failed"] is False
async def test_unbalanced_mail_stats_do_fail_the_wrapper(self, monkeypatch):
_patch_stages(monkeypatch, ollama_up=True,
mail_body=dict(MAIL_BODY_OK, mails_scanned=9))
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_body"]["failed"] is True
assert result["failed"] is True
async def test_silent_chunk_conflict_fails_the_wrapper(self, monkeypatch):
# An insert that no-op'd is a defect in the code, not bad input.
broken = dict(MAIL_BODY_OK, chunks_inserted=1, chunks_conflict_skipped=1)
_patch_stages(monkeypatch, ollama_up=True, mail_body=broken)
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_body"]["failed"] is True
async def test_mail_stage_exception_is_isolated(self, monkeypatch):
calls = _patch_stages(monkeypatch, ollama_up=True,
mail_body=RuntimeError("archive vanished"))
result = await run_cyclic(
dsn="dsn", paperless_url="http://paperless", paperless_token="tok",
anthropic_api_key="key", apply=True,
)
assert result["mail_body"]["failed"] is True
assert result["failed"] is True
assert calls["summarize"] == 1 # later stages still ran
async def test_adapter_failure_does_not_skip_later_stages(self, monkeypatch):
broken_adapter = dict(ADAPTER_OK, errors=1)
calls = _patch_stages(monkeypatch, ollama_up=True, adapter=broken_adapter)
@ -248,7 +368,8 @@ class TestRunCyclic:
assert result["failed"] is True
assert result["adapter"]["failed"] is True
# every other stage still ran despite the adapter failing this tick.
assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1}
assert calls == {"adapter": 1, "chunk_embed": 1, "summarize": 1, "embed_summaries": 1,
"mail_body": 1}
assert result["chunk_embed"]["failed"] is False
assert result["summarize"]["failed"] is False
assert result["embed_summaries"]["failed"] is False
@ -322,10 +443,33 @@ class TestBuildMetrics:
"chunk_embed": {"stats": dict(CHUNK_OK), "failed": False, "skipped": False, "error": None},
"summarize": {"stats": dict(SUMMARIZE_OK), "failed": False, "skipped": False, "error": None},
"embed_summaries": {"stats": dict(EMBED_SUMMARIES_OK), "failed": False, "skipped": False, "error": None},
"mail_body": {"stats": dict(MAIL_BODY_OK), "failed": False, "skipped": False, "error": None},
"mail_skip_reason": None,
"embed_backlog": embed_backlog,
"failed": failed,
}
def test_mail_metrics_are_published(self, tmp_path):
metrics = build_metrics(self._result(failed=False), now_ts=1000.0,
prom_path=tmp_path / "kb-ingest.prom")
assert metrics["kb_ingest_mail_chunks_inserted"] == 2
assert metrics["kb_ingest_mail_parse_errors"] == 0
assert metrics["kb_ingest_mail_skipped"] == 0
def test_tolerated_mail_parse_errors_are_still_published(self, tmp_path):
# Tolerated by the exit code, but not invisible — a rising count is a real signal.
result = self._result(failed=False)
result["mail_body"]["stats"] = dict(MAIL_BODY_OK, parse_errors=2, missing_file=1)
metrics = build_metrics(result, now_ts=1000.0, prom_path=tmp_path / "kb-ingest.prom")
assert metrics["kb_ingest_mail_parse_errors"] == 3
def test_skipped_mail_stage_reports_zero_chunks_and_a_skip_flag(self, tmp_path):
result = self._result(failed=False)
result["mail_body"] = {"stats": None, "failed": False, "skipped": True, "error": None}
metrics = build_metrics(result, now_ts=1000.0, prom_path=tmp_path / "kb-ingest.prom")
assert metrics["kb_ingest_mail_skipped"] == 1
assert metrics["kb_ingest_mail_chunks_inserted"] == 0
def test_success_sets_last_success_to_now(self, tmp_path):
prom_path = tmp_path / "kb-ingest.prom"
metrics = build_metrics(self._result(failed=False), now_ts=1000.0, prom_path=prom_path)

18
jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh Executable file
View file

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# kb-mail-sync-run.sh — thin launcher for kb-mail-sync.service (faza mailowa Krok 7).
#
# All sequencing and exit-code logic lives in the Python job (mail_imap_sync/sync.py); this
# script only computes a date-stamped log path and redirects, per the repo convention that a
# job must never log only to a terminal or the journal — the lesson from the 2026-07 backfill
# that lost 4999 rows of diagnostics to a closed tmux.
#
# Installed at /opt/homelab/kb/kb-mail-sync-run.sh (copied from the repo at install time),
# invoked by kb-mail-sync.service. `exec` preserves the job's exit code as this script's own,
# so systemd — and `kb_mail_sync_last_exit_code` — see the real result.
set -euo pipefail
LOG_DIR="/opt/homelab/logs/kb-mail-sync"
mkdir -p "$LOG_DIR"
LOG_FILE="${LOG_DIR}/run-$(date +%Y%m%d).log"
exec /opt/homelab/kb/venv/bin/mail-imap-sync --apply >> "$LOG_FILE" 2>&1

22
jobs/mail-imap-sync/systemd/kb-mail-sync.service Normal file
View file

@ -0,0 +1,22 @@
# kb-mail-sync.service — faza mailowa Krok 7 (recon kb/audits/mail-sync-2026-08-06.md §3.3).
# Host-level oneshot on PIHA, 1:1 with the kb-ingest pattern: the job needs the canonical
# .eml archive (local path), kb-postgres (localhost:5433) and outbound TLS to two IMAP
# servers. Containerizing it would buy nothing.
#
# NOT ENABLED BY THE REPO. Installation and activation are the operator's steps — see
# kb/runbooks/mail-sync-run.md. Nothing here starts on its own after a `git pull`.
#
# EnvironmentFile is read by systemd AS ROOT before dropping to User=oskar, so the IMAP app
# passwords in /opt/homelab/kb/.env (root-owned 0600) reach the process without ever being
# readable by `oskar` at rest. Keep that property: do not "fix" the file's ownership.
[Unit]
Description=KB incremental IMAP mail sync (gmail + fastmail)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=oskar
EnvironmentFile=/opt/homelab/kb/.env
WorkingDirectory=/opt/homelab/kb
ExecStart=/opt/homelab/kb/kb-mail-sync-run.sh

22
jobs/mail-imap-sync/systemd/kb-mail-sync.timer Normal file
View file

@ -0,0 +1,22 @@
# kb-mail-sync.timer — faza mailowa Krok 7 (recon kb/audits/mail-sync-2026-08-06.md §3.3).
#
# Hourly. At the measured ~37 messages a day a tick pulls one or two mails — effectively
# free — while keeping the corpus fresh to within an hour, which is what a future dispatcher
# reading this KB will need. Fetching does not touch the GPU, so this cadence is independent
# of SOLARIA's daily power cycle; indexing catches up separately via kb-ingest.timer.
#
# Persistent=true so a PIHA reboot catches the missed tick up instead of silently skipping it.
# RandomizedDelaySec keeps two providers from being hit at exactly :00 every hour.
#
# NOT ENABLED BY THE REPO — `systemctl enable --now` is an operator step (kb/runbooks/
# mail-sync-run.md), deliberately after the first supervised --apply run.
[Unit]
Description=Run kb-mail-sync.service hourly (faza mailowa Krok 7)
[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=120
[Install]
WantedBy=timers.target

51
kb/phases/kb-m5-faza-mailowa.md
View file

@ -13,10 +13,12 @@ links: []
> retrieval, Etap A apply na żywej bazie, bramka jakościowa **PASS** — patrz §8).
> **Etap B (Krok 6) ZAMKNIĘTY 2026-08-06**: pełny korpus gmail jest zchunkowany
> i zembedowany (389 012 chunków, zero nie-excluded bez wektora) — patrz §9
> „Wynik Etapu B". **Krok 7 (recon przyrostówki) WYKONANY 2026-08-06**:
> `kb/audits/mail-sync-2026-08-06.md` — czeka na decyzje operatora (a)-(g),
> w tym rozstrzygnięcie IMAP vs JMAP dla Fastmaila (§10 niżej mówi JMAP,
> recon rekomenduje wspólny IMAP).
> „Wynik Etapu B". **Krok 7 (przyrostówka IMAP) — IN PROGRESS 2026-08-06**:
> recon `kb/audits/mail-sync-2026-08-06.md` wykonany, decyzje operatora (a)-(g)
> **zatwierdzone w całości 2026-08-06**, implementacja w repo (§10 „Zakres
> wdrożony"). **Kod nie łączył się z żywym kontem** — pierwszy sync, pomiar
> Fastmaila i aktywacja timera to kroki operatora wg
> `kb/runbooks/mail-sync-run.md`.
>
> Kontynuacja `05-faza4-plan.md` (faza 4: `packages/kb-retrieval` wydzielone,
> serwis `kb-query` z UI działa na PIHA — „KB po raz pierwszy odpowiada przez
@ -698,7 +700,9 @@ oznacza co innego: backend embed padł, trzeba wznowić plaster po naprawie Olla
## 10. Krok 7 — IMAP/JMAP przyrostówka (zarys; szczegóły = osobny recon)
> **Recon wykonany 2026-08-06: `kb/audits/mail-sync-2026-08-06.md`.** Zarys
> **Stan: IN PROGRESS (2026-08-06).** Recon:
> `kb/audits/mail-sync-2026-08-06.md`; decyzje (a)-(g) zatwierdzone przez
> operatora 2026-08-06 w całości, implementacja opisana niżej. Zarys
> poniżej pochodzi z 2026-07-22 i zachowuję go jako zapis intencji. Recon
> rozstrzyga inaczej dwa jego punkty: (1) **Fastmail przez IMAP, nie JMAP**
> (unifikacja adaptera — jeden `jobs/mail-imap-sync` zamiast
@ -707,6 +711,35 @@ oznacza co innego: backend embed padł, trzeba wznowić plaster po naprawie Olla
> IDLE, reuse `save_eml`/`insert_envelope`, sekrety w `/opt/homelab/config/`)
> się potwierdziła.
### Zakres wdrożony (2026-08-06)
| # | Element | Gdzie |
|---|---|---|
| 1 | Adapter IMAP, jeden na oba konta — EXAMINE + `BODY.PEEK[]` (nigdy nie ustawia `\Seen`), wybór folderu po atrybucie SPECIAL-USE | `packages/kb-mail/src/kb_mail/imap.py` |
| 2 | Model stanu synca: `plan_folder_sync` (pierwszy tick / przyrost / unieważnienie UIDVALIDITY) + `contiguous_last_uid` (kursor tylko po nieprzerwanym ciągu sukcesów) | `packages/kb-mail/src/kb_mail/sync_state.py` |
| 3 | Migracja `005_mail_sync_state.sql`, klucz `(account, folder)` — Decyzja (f) | `services/kb-postgres/init/` |
| 4 | Job przyrostówki: fetch → `save_eml` → `insert_envelope(entities=[headers, attachment…])` → kursor. **Nie chunkuje i nie embeduje** | `jobs/mail-imap-sync/` |
| 5 | Ekstrakcja wspólnego parse'u (`parse_headers`, `message_id`, `parse_date`, `parse_attachments`) do `kb-mail` — klucz dedup z jednej implementacji | `packages/kb-mail/src/kb_mail/{headers,message}.py` |
| 6 | `--sources` + `--only-unchunked` w `mail-body-ingest`; pre-fetch kluczy chunków zawężony do zbioru roboczego | `jobs/mail-body-ingest/` |
| 7 | `DEFAULT_SUMMARYLESS_SOURCES += "fastmail"` — Decyzja (g), w tym samym commicie co źródło | `packages/kb-retrieval/` |
| 8 | Etap mailowy w `kb-ingest` (kolejka „koperty bez chunków") + takt timera 03:30 → co 2 h — Decyzja (d) | `jobs/documents-ingest/` |
| 9 | Jednostki systemd (**nieaktywowane**) + deklaracja jednostek host-level | `jobs/mail-imap-sync/systemd/`, `hosts/piha/jobs.yaml` |
| 10 | Metryki `.prom` per konto + reguła `KbMailSyncStale` | `services/fleet-prometheus/rules/kb-mail-sync.yml` |
| 11 | `env.example` z placeholderami; poświadczenia wyłącznie ze środowiska — Decyzja (c) | `jobs/mail-imap-sync/env.example` |
| 12 | Runbook pierwszego uruchomienia + checklista punktów `[do weryfikacji na żywo]` z reconu | `kb/runbooks/mail-sync-run.md` |
Dokumentacja serwisu: `kb/services/job-mail-imap-sync.md`.
**Poza zakresem tej implementacji, świadomie:** pierwszy żywy sync, pomiar
`STATUS (MESSAGES)` na Fastmailu i wynikająca z niego **decyzja o historii
Fastmaila** (Decyzja (e) — recon celowo jej nie podejmuje, bo zależy od liczby,
której nikt jeszcze nie zna), oraz aktywacja timera. Wszystko to robi operator
wg runbooka. Alert „cisza w skrzynce" **odrzucony** (decyzja operatora, zgodna
z reconem §3.4): zero nowych maili to legalny stan skrzynki, a alert zapalający
się na zdrowym systemie zostaje wyciszony — i przestaje działać wtedy, gdy jest
potrzebny. Jedyny alert to `KbMailSyncStale` („czy poller w ogóle działa"),
który fałszywych trafień nie ma.
Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`,
`jobs/gmail-imap-poller`). Zarys decyzji do tamtego reconu:
@ -733,7 +766,10 @@ Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`,
| `mail_ui_url` (klikalny link do maila w UI) | kb-00 etap 6, pole zarezerwowane w kb-query | z modułem mail-UI |
| Graf wątków / entity_link z `entities[type=threading]` | ta faza tylko zapisuje surowiec (Decyzja 10) | przyszła faza „połączenia" |
| Streszczenia selektywne maili (hybryda Haiku) | Decyzja 6 — odłożona | po ocenie trybu hybrid w praktyce |
| IMAP/JMAP przyrostówka — implementacja | Krok 7 (zarys) | osobny recon + pakiet |
| ~~IMAP/JMAP przyrostówka — implementacja~~ | Krok 7 | **wykonane 2026-08-06** (§10 „Zakres wdrożony"); pierwszy żywy sync po stronie operatora |
| Historia Fastmaila (pełny zaciąg vs tylko przyrost) | Decyzja (e) reconu | po pomiarze `mail-imap-sync --measure` — runbook §5 |
| Alert per konto na wiek najnowszego maila (`kb_mail_sync_last_message_ts`) | recon §3.4 | po miesiącu obserwacji; próg z pomiaru, nie z góry |
| Etykiety Gmaila w `entities` (`X-GM-LABELS`) | recon §2.3 | odłożone — `X-GM-EXT-1` przywiązuje kod do Google, wprost wbrew „protokół, nie provider" |
## 12. Plan implementacji (kolejność = zależności)
@ -746,7 +782,8 @@ Zakotwiczone w kb-00 jako etapy 3–4 (`jobs/fastmail-poller`,
| 4 | rsync + Etap A (12 mies.) + kalibracja | 2 | 1 sesja | **WYKONANE** | §7 „Wynik Etapu A" (run na żywo 2026-07-23); potwierdzone na żywej bazie 2026-08-04: `document_chunk` gmail = 33 871 (6 398 z embeddingiem + 27 473 `newsletter`) — zgodne co do sztuki z tabelą §7 |
| 5 | Bramka jakościowa (eval mailowy + regresja) | 3, 4 + zapytania od operatora | 1 sesja | **WYKONANE** (PASS) | §8 „Wynik bramki"; `56f64e9` (eval + queries.yaml dla hybrid), `bce635c` (`mail_hit@3`, próg N2, werdykt PASS), `71eb264` (`--transport http`) |
| 6 | Etap B (pełne archiwum) + regresja + obserwacja PIHA | 5 = PASS | 1 sesja | **WYKONANE** (2026-08-06) | §9 „Wynik Etapu B"; żywa baza: 389 012 chunków, 0 nie-excluded bez embeddingu. Weryfikacja plastrami 0-4 (wszystkie EXIT 0) + fix NUL `4ec0b78`; `docs/sessions/2026-08-06-kb-etapb-backfill.md` |
| 7 | Recon przyrostówki IMAP/JMAP | — (po 6) | 1 sesja (poza DoD fazy) | **WYKONANE** (2026-08-06) | `kb/audits/mail-sync-2026-08-06.md`. Ustalenia: korpus urywa się 2026-06-19 (dziura 48 dni ≈ 1 800 maili), zero kodu IMAP w repo, brak modelu stanu synca; cały nowy kod to jeden `jobs/mail-imap-sync` + 4 drobne zmiany w istniejącym torze. Do decyzji operatora: (a)-(g) w §4 tamtego dokumentu |
| 7 | Recon przyrostówki IMAP/JMAP | — (po 6) | 1 sesja (poza DoD fazy) | **WYKONANE** (2026-08-06) | `kb/audits/mail-sync-2026-08-06.md`. Ustalenia: korpus urywa się 2026-06-19 (dziura 48 dni ≈ 1 800 maili), zero kodu IMAP w repo, brak modelu stanu synca; cały nowy kod to jeden `jobs/mail-imap-sync` + 4 drobne zmiany w istniejącym torze. Decyzje (a)-(g) zatwierdzone przez operatora 2026-08-06 |
| 8 | Implementacja przyrostówki IMAP | 7 + decyzje (a)-(g) | 2 sesje (poza DoD fazy) | **IN PROGRESS** (2026-08-06) | Zakres w §10 „Zakres wdrożony". Testy zielone: 80 (`mail-imap-sync`) + 111 (`kb-mail`) + 191 (`documents-ingest`) + 127 (`mail-body-ingest` / `kb-retrieval`). **Nie uruchomione na żywym koncie** — pierwszy sync, pomiar Fastmaila i aktywacja timera po stronie operatora (`kb/runbooks/mail-sync-run.md`) |
**Kryterium ukończenia fazy mailowej:** (a) pełny korpus gmail zchunkowany
(bilans domknięty, `parse_errors` na poziomie pojedynczych sztuk jak

15
kb/phases/kb-m5-faza3.md
View file

@ -560,10 +560,19 @@ Adapter i embed są już idempotentne — nowość to wyłącznie orkiestracja i
`/etc/systemd/system/`, `systemctl enable --now kb-ingest.timer`). Pierwszy
systemd-timer w repo — świadomie host-level, nie kontener (joby potrzebują jednocześnie
LAN, DB i plików hosta; konteneryzacja nic tu nie daje).
- **Harmonogram**: `OnCalendar=*-*-* 03:30`, `Persistent=true` (nadgania po reboocie).
- **Harmonogram**: ~~`OnCalendar=*-*-* 03:30`~~ → **`OnCalendar=0/2:00:00` (co 2 h) od
2026-08-06**, `Persistent=true` (nadgania po reboocie). Zmiana wynika z Decyzji (d) reconu
przyrostówki (`kb/audits/mail-sync-2026-08-06.md` §3.3): o 03:30 SOLARIA prawie na pewno
śpi (potwierdzone odczytem `kb_ingest_embed_skipped 1`), a od tej daty tick dostaje też
etap mailowy — ~60 nowych chunków na dobę pomijanych każdej nocy zapaliłyby
`KbEmbedBacklogGrowing` na stałe. Co 2 h zamiast stałej godziny dopasowanej do nawyków
operatora: probe Ollamy sam wybiera okno, więc któryś tick w nie trafi niezależnie od tego,
o której SOLARIA wstaje w danym tygodniu.
- **Sekwencja skryptu**: adapter `--apply` → chunk_embed `--apply`
(`OLLAMA_URL=http://solaria:11434`) → (po decyzji z pilota, rozszerzenie później:
summarize nowych dokumentów). Log do `/opt/homelab/logs/kb-ingest/run-YYYYMMDD.log`.
(`OLLAMA_URL=http://solaria:11434`) → **mail_body_ingest `--only-unchunked`** (dodane
2026-08-06 — konsument kolejki, którą wypełnia `jobs/mail-imap-sync`; import miękki, więc
venv bez tego pakietu pomija etap zamiast wywracać wrapper) → summarize → embed-summaries.
Log do `/opt/homelab/logs/kb-ingest/run-YYYYMMDD.log`.
- **Tolerancja na SOLARIĘ offline** (`availability_target: medium`): wrapper odróżnia
„Ollama nieosiągalna" (probe `GET /api/tags` przed embedem; brak → pomiń embed,
odnotuj, **to nie jest fail** — nadrobi następny run, bo embed jest idempotentny) od

352
kb/runbooks/mail-sync-run.md Normal file
View file

@ -0,0 +1,352 @@
---
okf: "0.1"
type: runbook
visibility: private
status: active
updated: 2026-08-06
links:
- ../services/job-mail-imap-sync.md
- ../audits/mail-sync-2026-08-06.md
---
# mail-imap-sync — pierwsze uruchomienie i eksploatacja
Wszystko poniżej wykonuje **operator na PIHA**. Kod nie łączył się nigdy z żadnym żywym
kontem — pierwszy sync jest nadzorowany i idzie krokami 1-9. Timer aktywujesz dopiero
w kroku 9, po tym jak ręczny run przejdzie czysto.
---
## 0. Punkty [do weryfikacji na żywo]
Recon (`kb/audits/mail-sync-2026-08-06.md`) oznaczył cztery twierdzenia o zewnętrznych
dostawcach jako **niesprawdzone** — stan wiedzy modelu sięga maja 2026 i **nie jest
wiarygodnym źródłem dla polityki dostawcy z sierpnia 2026**. Sprawdź je przy okazji kroków
1-3 i dopisz wynik tutaj. Nie zgaduj.
| # | Do sprawdzenia | Gdzie | Wynik |
|---|---|---|---|
| 1 | Czy Google nadal wydaje hasła aplikacji (i czy wymagają 2FA na koncie)? Gdyby metoda odpadła, przymusem staje się OAuth2 — reszta architektury się nie zmienia, wymianie podlega **wyłącznie** sposób uwierzytelnienia w `kb_mail.imap.ImapClient.connect`. | Krok 1 | *(wpisz)* |
| 2 | Czy Fastmail pozwala wydać hasło aplikacji **o zakresie tylko-IMAP** (bez dostępu do panelu)? Jeśli tak, jest to uprawnienie ściślejsze niż token JMAP — i to domyka argument za unifikacją adaptera. | Krok 2 | *(wpisz)* |
| 3 | Czy konto Fastmail wystawia jakiś folder wirtualny obejmujący całość (odpowiednik gmailowego `\All`)? Jeśli tak, zakres `INBOX+Archive+Sent` można uprościć. | Krok 4 (`--measure` wypisze wszystkie foldery po `LIST`) | *(wpisz)* |
| 4 | Dobowy limit transferu IMAP Gmaila (rzędu kilku GB). Przy ~1 800 zaległych maili (~225 MB) i ~4,6 MB/dobę bieżąco nie ma znaczenia — ma, gdyby ktoś kiedyś ciągnął IMAP-em pełną historię. | Krok 6, jeśli pierwszy sweep się urwie | *(wpisz)* |
---
## 1. Hasło aplikacji — Gmail
Wymaga włączonego 2FA na koncie (bez tego Google nie pokaże opcji haseł aplikacji).
Wygeneruj hasło aplikacji dla „Mail", zapisz je **od razu do menedżera haseł** — Google
pokazuje je jeden raz.
> **Nie wypisuj hasła na ekran i nie wklejaj go do żadnej sesji.** W tym repo są dwa
> udokumentowane przypadki rotacji sekretu po wycieku do transkryptu sesji
> (`docs/sessions/2026-07-15.md` §Krok 3, `docs/sessions/2026-07-21.md` incydent 3). To jedyna
> droga, którą sekrety w tym homelabie dotąd wyciekały.
Google wyświetla hasło w czterech grupach po cztery znaki — **wpisuj bez spacji**.
## 2. Hasło aplikacji — Fastmail
Wydaj hasło aplikacji o zakresie **IMAP** (nie „full access"), jeśli konto na to pozwala —
to punkt (2) z tabeli wyżej. Zapisz tak samo jak wyżej.
## 3. Wypełnienie `.env`
Sekrety dopisujesz do **istniejącego** `/opt/homelab/kb/.env` (już `root:root 0600`, już
trzyma `KB_DSN`, `PAPERLESS_API_TOKEN`, `ANTHROPIC_API_KEY`). Edytorem, nie `echo >>` —
`echo` zostawia hasło w historii powłoki.
```bash
sudo -e /opt/homelab/kb/.env
```
Wzorzec kluczy i komentarze: `jobs/mail-imap-sync/env.example`. Minimum:
```
MAIL_ACCOUNTS=gmail,fastmail
MAIL_GMAIL_USER=...
MAIL_GMAIL_APP_PASSWORD=...
MAIL_GMAIL_INITIAL_MODE=since
MAIL_GMAIL_INITIAL_SINCE=2026-06-15
MAIL_FASTMAIL_USER=...
MAIL_FASTMAIL_APP_PASSWORD=...
MAIL_FASTMAIL_INITIAL_MODE=new-only
```
Sprawdź uprawnienia po edycji — muszą zostać `600 root:root`:
```bash
sudo ls -l /opt/homelab/kb/.env
```
Dlaczego akurat tak: systemd czyta `EnvironmentFile` **jako root, przed zrzuceniem uprawnień
do `User=oskar`**. Sekret trafia do procesu, ale nie jest czytelny dla `oskar` w spoczynku.
To lepsza własność niż plik `600 oskar:oskar` — nie „napraw" jej.
### Skąd data dla `MAIL_GMAIL_INITIAL_SINCE`
Kilka dni **przed** najnowszą kopertą w bazie — nakładka jest darmowa (dedup po Message-ID),
a luka nie jest:
```bash
docker exec kb-postgres psql -U kb -d kb -c \
"SELECT source, count(*), max(ts) FROM envelope GROUP BY source ORDER BY source;"
```
Stan na 2026-08-06: gmail 225 030 kopert, `max(ts) = 2026-06-19 19:13:39+02`. Stąd
`2026-06-15` w przykładzie wyżej. `SINCE` filtruje po INTERNALDATE serwera (kiedy wiadomość
przyszła), nie po nagłówku `Date:` — czyli po właściwej stronie tego rozróżnienia.
## 4. Instalacja i migracja
```bash
cd ~/homelab-codex-ws && git pull
source /opt/homelab/kb/venv/bin/activate
pip install -e packages/kb-mail/ -e jobs/mail-imap-sync/
# mail-body-ingest jest potrzebny etapowi mailowemu w kb-ingest (patrz krok 8):
pip install -e packages/kb-retrieval/ -e jobs/mail-body-ingest/
```
Migracja `005` — `initdb` odpala skrypty z `init/` **tylko na świeżym wolumenie**, więc na
istniejącej bazie aplikujesz ją ręcznie:
```bash
docker exec -i kb-postgres psql -U kb -d kb \
< ~/homelab-codex-ws/services/kb-postgres/init/005_mail_sync_state.sql
docker exec kb-postgres psql -U kb -d kb -c "\d mail_sync_state"
```
Oczekiwane: kolumny `account, folder, uidvalidity, last_uid, last_sync_ts` i
`PRIMARY KEY (account, folder)`. DDL jest `IF NOT EXISTS` — powtórzenie jest bezpieczne.
## 5. Pomiar — ile waży Fastmail
**To jest moment, w którym zapada decyzja o historii Fastmaila.** Recon celowo jej nie
podjął: zależy od liczby, której nikt jeszcze nie znał (Decyzja (e), „najpierw pomiar, potem
decyzja"). Odłożona **do tego kroku**.
```bash
set -a; . /opt/homelab/kb/.env; set +a
mail-imap-sync --measure
```
Tryb jest read-only: `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, zero zapisów, zero
dotknięcia flag wiadomości. Weryfikuje przy okazji, że oba hasła aplikacji działają.
Odczytaj sumę `messages` dla `INBOX + Archive + Sent` i wybierz:
| Rozmiar | `MAIL_FASTMAIL_INITIAL_MODE` | Uwagi |
|---|---|---|
| kilka-kilkanaście tysięcy | `full` | Rzędu godzin transferu; mechanizm ten sam co przyrost, więc nie kosztuje osobnego kodu. To prawdopodobnie ta „sensowna poczta", o którą chodziło bardziej niż o gmailowe newslettery. |
| dziesiątki tysięcy+ | `full` **z `--limit`** | Pierwszy sweep w kawałkach, np. `--limit 2000` na tick; kursor idzie od najstarszych, reszta dochodzi kolejnymi tickami. |
| nieinteresująca | `new-only` | Historia zostaje poza KB. |
Wpisz wybór do `.env` **zanim** puścisz pierwszy `--apply` — po pierwszym runie istnieje
wiersz w `mail_sync_state` i `INITIAL_MODE` przestaje mieć jakikolwiek wpływ.
Przy okazji: `--measure` po drodze robi `LIST`, więc log pokaże wszystkie foldery konta —
stąd odpowiedź na punkt (3) z tabeli w §0.
## 6. Dry run, potem pierwszy sync
Dry run nie pobiera ani jednej wiadomości — robi EXAMINE + SEARCH i mówi, ile by ściągnął:
```bash
mail-imap-sync
```
Czytaj `candidates` per folder. Dla gmaila przy `since=2026-06-15` spodziewaj się rzędu
**~1 800** (48 dni × ~37/dobę). Rząd wielkości zupełnie inny niż oczekiwany = zatrzymaj się
i sprawdź konfigurację, nie puszczaj `--apply`.
Pierwszy prawdziwy run — z hamulcem, do pliku, na niskim priorytecie:
```bash
mkdir -p /opt/homelab/logs/kb-mail-sync
nice -n 10 ionice -c2 -n7 mail-imap-sync --apply --limit 200 \
> /opt/homelab/logs/kb-mail-sync/first-run.log 2>&1
echo "exit=$?"
```
Sprawdź w logu linię `summary`:
- `envelopes_inserted` > 0, `errors` = 0,
- `archived` + `archive_exists` = `processed`,
- `balance_problems` puste,
- `headers_fallback` — pojedyncze sztuki są normalne (8-bitowe nagłówki), dziesiątki nie.
Potem puszczaj bez `--limit`, aż `candidates` w dry runie spadnie do zera:
```bash
mail-imap-sync --apply >> /opt/homelab/logs/kb-mail-sync/first-run.log 2>&1
```
Kontrola w bazie:
```bash
docker exec kb-postgres psql -U kb -d kb -c \
"SELECT account, folder, uidvalidity, last_uid, last_sync_ts FROM mail_sync_state ORDER BY 1,2;"
docker exec kb-postgres psql -U kb -d kb -c \
"SELECT source, count(*), max(ts) FROM envelope GROUP BY source ORDER BY 1;"
```
`max(ts)` dla gmaila powinien być z ostatnich dni, a `source='fastmail'` pojawić się po raz
pierwszy w historii tej bazy.
## 7. Indeksowanie i weryfikacja end-to-end
Sam sync **nie chunkuje i nie embeduje** — celowo (recon §3.2). Nowe koperty czekają
w kolejce, którą jest brak chunków. Zdrenuj ją ręcznie raz, przy włączonej SOLARII:
```bash
mail-body-ingest --only-unchunked --apply \
>> /opt/homelab/logs/kb-mail-sync/first-ingest.log 2>&1
```
Pełny test od końca do końca — wyślij sobie maila z rozpoznawalną frazą, potem:
```bash
# 1. sync go pobiera
mail-imap-sync --apply
# 2. koperta jest w bazie
docker exec kb-postgres psql -U kb -d kb -c \
"SELECT id, source, ts FROM envelope ORDER BY ts DESC LIMIT 5;"
# 3. ingest robi chunki i embeddingi (SOLARIA musi być włączona)
mail-body-ingest --only-unchunked --apply
# 4. widać go w wyszukiwarce, trybem domyślnym (hybrid)
curl -s "http://localhost:8230/search?q=<fraza+z+maila>" | jq '.results[] | {envelope_id, dist}'
```
Krok 4 jest tym, co weryfikuje Decyzję (g): `fastmail` musi być w
`DEFAULT_SUMMARYLESS_SOURCES`, inaczej koperty zaindeksują się poprawnie i **nie pojawią się
w żadnym wyniku** — bez błędu gdziekolwiek. Jeśli mail z Fastmaila nie wychodzi w `/search`,
a z gmaila wychodzi, to jest pierwsze miejsce do sprawdzenia.
## 8. Instalacja jednostek systemd (jeszcze bez włączenia)
```bash
sudo install -m 0755 ~/homelab-codex-ws/jobs/mail-imap-sync/systemd/kb-mail-sync-run.sh \
/opt/homelab/kb/kb-mail-sync-run.sh
sudo cp ~/homelab-codex-ws/jobs/mail-imap-sync/systemd/kb-mail-sync.{service,timer} \
/etc/systemd/system/
sudo systemctl daemon-reload
```
**Równocześnie: zmiana taktu `kb-ingest`** (Decyzja (d), recon §3.3). Timer przechodzi
z 03:30 raz na dobę na **co 2 h**, a `kb-ingest` dostaje etap mailowy:
```bash
sudo cp ~/homelab-codex-ws/jobs/documents-ingest/systemd/kb-ingest.timer /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl restart kb-ingest.timer
systemctl list-timers kb-ingest.timer
```
Dlaczego to musi iść razem z uruchomieniem przyrostówki: o 03:30 SOLARIA prawie na pewno śpi
(potwierdzone 2026-08-06 odczytem `kb_ingest_embed_skipped 1`). Dopięcie ~60 nowych chunków
na dobę do ticku, który każdej nocy pomija embedowanie, sprawiłoby, że backlog rośnie
monotonicznie i `KbEmbedBacklogGrowing` (próg `> 0` przez 72 h) **zapala się na stałe** — nie
sygnalizując awarii, tylko rozjazd harmonogramu z dobowym cyklem SOLARII. Alert, który świeci
zawsze, przestaje być alertem.
Reguła alertowa dla samego synca (`services/fleet-prometheus/rules/kb-mail-sync.yml`) wchodzi
przy najbliższym deployu fleet-prometheus. Dopóki timer nie chodzi, metryka nie istnieje,
`time() - <brak serii>` nie daje wyniku i reguła jest bezczynna — kolejność nie ma znaczenia.
## 9. Aktywacja timera
Dopiero teraz, i tylko jeśli kroki 6-7 przeszły czysto:
```bash
sudo systemctl enable --now kb-mail-sync.timer
systemctl list-timers kb-mail-sync.timer
```
Po pierwszym automatycznym ticku:
```bash
journalctl -u kb-mail-sync.service -n 50
cat /opt/homelab/state/node-exporter/kb-mail-sync.prom
```
W `.prom` szukaj `kb_mail_sync_last_success_timestamp` (niezerowy) i
`kb_mail_sync_envelopes_inserted{account="..."}`.
---
## Eksploatacja
### Codzienne sprawdzenie
```bash
tail -n 20 /opt/homelab/logs/kb-mail-sync/run-$(date +%Y%m%d).log
docker exec kb-postgres psql -U kb -d kb -c \
"SELECT account, folder, last_uid, last_sync_ts FROM mail_sync_state ORDER BY 1,2;"
```
`last_sync_ts` ma się przesuwać co godzinę **nawet gdy nic nie przyszło** — to jest odpowiedź
na pytanie „czy poller w ogóle działa", niezależna od tego, czy ktoś do nas napisał.
### Folder stanął — jedna wiadomość blokuje kursor
Objaw: `errors` niezerowe co tick, `last_uid` w `mail_sync_state` stoi, w logu ten sam UID
przy `skip.fetch_error` / `skip.store_error`.
To jest zaprojektowane zachowanie: kursor nie przeskakuje wiadomości, której nie udało się
zapisać (przeskakiwanie = ciche gubienie poczty). Wiadomości *po* niej i tak lądują
w archiwum i w bazie — blokada dotyczy wyłącznie kursora.
Jeśli wiadomość jest trwale nie do pobrania, przestaw kursor ręcznie o jeden:
```bash
docker exec kb-postgres psql -U kb -d kb -c \
"UPDATE mail_sync_state SET last_uid = <zly_uid> WHERE account='gmail' AND folder='<folder>';"
```
Zanotuj w `kb/incidents/`, którego maila świadomie pominięto.
### Serwer unieważnił UIDVALIDITY
W logu `imap.uidvalidity_reset`, w metrykach `kb_mail_sync_uidvalidity_resets{account} = 1`.
Job sam robi pełne przemiecenie folderu i opiera się na dedupie po Message-ID — nie ma tu nic
do zrobienia ręcznie. Tick będzie dłuższy i `envelopes_skipped_dup` skoczy do rozmiaru
folderu; to jest poprawny przebieg, nie awaria.
### Rotacja hasła aplikacji
Odwołaj stare u dostawcy, wydaj nowe, podmień w `/opt/homelab/kb/.env`, `systemctl start
kb-mail-sync.service`. Restart timera nie jest potrzebny — `EnvironmentFile` czytany jest przy
każdym starcie serwisu. Nic w stanie synca nie zależy od poświadczeń.
### Dołożenie folderu
Dopisz go do `MAIL_<KONTO>_FOLDERS` w `.env`. Nowy folder nie ma wiersza w `mail_sync_state`,
więc obowiązuje go `INITIAL_MODE` tego konta — jeśli chcesz jego historię, przestaw na `full`
**na jeden run**, potem wróć do poprzedniej wartości (istniejące foldery mają już kursory
i zmiana ich nie dotyczy). Zawartość pokrywająca się z innym folderem niczego nie zduplikuje.
## Testy
```bash
pip install -e "jobs/mail-imap-sync[dev]"
cd jobs/mail-imap-sync && pytest
```
80 testów, bez sieci i bez bazy: `ImapClient` jest napędzany podstawionym połączeniem
`imaplib` (więc testowane jest prawdziwe parsowanie protokołu, nie mock adaptera), a
`envelope`/`mail_sync_state` są w pamięci. Pokrycie: nowe wiadomości, uniewaznienie
UIDVALIDITY, dedup na trzech warstwach, wznowienie po przerwaniu, kolizja Message-ID między
kontami, izolacja kont, wygasłe UID-y, dry-run bez sieci, przenoszenie
`last_success_timestamp` przez nieudany run.
W worktree bez gotowego venva:
```bash
python3 -m venv /tmp/kbvenv && /tmp/kbvenv/bin/pip install -q pytest pytest-asyncio \
-e packages/kb-mail/ -e jobs/mail-imap-sync/
/tmp/kbvenv/bin/python -m pytest jobs/mail-imap-sync/tests packages/kb-mail/tests -q
```

40
kb/services/job-mail-body-ingest.md
View file

@ -3,9 +3,10 @@ okf: "0.1"
type: service
visibility: private
status: active
updated: 2026-08-05
updated: 2026-08-06
links:
- ../runbooks/mail-body-ingest-run.md
- job-mail-imap-sync.md
---
# mail-body-ingest
@ -26,9 +27,16 @@ only thing genuinely shared is the chunker itself, which is why it was extracted
## Where it runs
**On SOLARIA** (needs Ollama on `localhost` for `/api/embed`), against `kb-postgres@PIHA` over
Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7, Krok 4) rather than
read live over the network — 225k small files over Tailscale would be slow and fragile.
**On SOLARIA** for backfill runs (needs Ollama on `localhost` for `/api/embed`), against
`kb-postgres@PIHA` over Tailscale. The `.eml` archive is rsync'd PIHA -> SOLARIA once (plan §7,
Krok 4) rather than read live over the network — 225k small files over Tailscale would be slow
and fragile.
**Also on PIHA since 2026-08-06**, as the mail stage of `kb-ingest` (see "Two modes" below).
There it reads the canonical archive locally and calls Ollama@SOLARIA over Tailscale — which
is fine at this scale: the daily increment is ~60 chunks at 8-18 ms each, measured at 8.2 ms
round-trip from PIHA. The argument against embedding from PIHA (recon §3.1) was about 271k
backfill chunks and does not carry over.
Install (from repo root, on SOLARIA):
@ -71,6 +79,30 @@ pip install -e jobs/mail-body-ingest/
`gmail-header-backfill` — done for every parsed envelope regardless of newsletter/
body_empty status, since it's the same read either way.
## Two modes: backfill slice vs cyclic queue
The job was written for supervised full-corpus slices (`--limit`/`--offset`/`--since`). Since
2026-08-06 it also runs as a stage of `kb-ingest` every 2 hours, draining what the incremental
IMAP sync fetched:
```bash
mail-body-ingest --only-unchunked --apply # cyclic mode
```
`--only-unchunked` restricts the set to envelopes with **no `document_chunk` rows at all** —
the queue that the data itself defines (recon `kb/audits/mail-sync-2026-08-06.md` §3.2). It
needs no separate state and is self-healing: an interrupted run, a mail fetched while Ollama
was down, a message inserted with a backdated header — each stays pending until it has chunks.
`--since` was explicitly rejected for that role. It filters on `envelope.ts`, the sender's
`Date:` header, so a mail delivered today carrying a month-old date would fall outside the
window and never be chunked at all. `--since` stays what it was: a tool for staging manual runs.
`--sources` defaults to `gmail,fastmail`. The pre-fetched chunk-key set is scoped to the
working set (`WHERE model = $1 AND envelope_id = ANY($2)`); unbounded it read all 389 012 keys
(~26 MB, ~1.0 s) on every run, which is nothing once for a 50k slice and wasteful every two
hours on a node with 2.4 GB available (recon §2.5 iv).
## Stats must balance
```

256
kb/services/job-mail-imap-sync.md Normal file
View file

@ -0,0 +1,256 @@
---
okf: "0.1"
type: service
visibility: private
status: active
updated: 2026-08-06
links:
- ../runbooks/mail-sync-run.md
- ../audits/mail-sync-2026-08-06.md
- ../phases/kb-m5-faza-mailowa.md
---
# mail-imap-sync
Module 5, faza mailowa, Krok 7 (`kb/phases/kb-m5-faza-mailowa.md` §10), built to the recon
`kb/audits/mail-sync-2026-08-06.md` after the operator approved decisions (a)-(g) on
2026-08-06. It is the first code in this repo that talks to a mail server.
Before it, the mail corpus was a photograph: 225 030 envelopes ending at **2026-06-19**,
imported once from a Gmail Takeout, with a gap that grew by roughly 37 messages a day. This
job closes the gap and keeps it closed.
## Where it runs
**On PIHA**, hourly, as a host-level systemd oneshot (`jobs/mail-imap-sync/systemd/`). The
choice is not close (recon §3.1): PIHA is up 24/7, holds the canonical `.eml` archive, and
runs kb-postgres locally. SOLARIA is powered off ~16 h a day by design, so a poller living
there would read the mailbox only when the desktop happened to be on.
Install (from repo root, on PIHA):
```bash
pip install -e packages/kb-mail/
pip install -e jobs/mail-imap-sync/
```
## What one tick does
Per account, per folder:
1. **EXAMINE** the folder — read-only. Never `SELECT`, and bodies come via `BODY.PEEK[]`, so
the job never sets `\Seen` on the operator's own mail.
2. **Plan** the tick from `mail_sync_state` and the server's UIDVALIDITY/UIDNEXT — first
contact, ordinary increment, or a UIDVALIDITY reset (see below).
3. **UID SEARCH** for the UIDs this tick owns.
4. Per UID, ascending: **FETCH** → `save_eml()` (append-only; `FileExistsError` just means
"already have it") → `insert_envelope()` with `entities = [headers, attachment…]`.
5. **UPDATE `mail_sync_state`** — but only over the contiguous prefix of messages that are
fully durable in both the archive and the DB.
## Folder scope (Decyzja (e))
| Account | Scope | Why |
|---|---|---|
| gmail | SPECIAL-USE `\All` | one fetch per message despite many labels; the same population as the Takeout the existing 225 030 envelopes came from, so the corpus stays continuous |
| fastmail | `INBOX` + `Archive` + `Sent` | a classic IMAP layout has no `\All` equivalent |
Spam, Trash and Drafts are excluded from both. Spam and Trash are defined noise and were not
in the Takeout either; drafts are not correspondence, have no stable Message-ID, and mutate.
The gmail folder is resolved **by SPECIAL-USE attribute, never by name**. Gmail localizes the
mailbox — with a Polish UI it is `[Gmail]/Wszystkie` — so a hardcoded `[Gmail]/All Mail` would
sync nothing while reporting a clean run every hour.
Scope is per-account configuration (`MAIL_<ACCOUNT>_FOLDERS` / `_SPECIAL_USE`), so adding a
folder later is a config change, not a code change; Message-ID dedup absorbs any overlap.
## The sync cursor
`mail_sync_state` (migration `005`), keyed `(account, folder)`, holding `uidvalidity`,
`last_uid` and `last_sync_ts`. A table rather than a file in `/opt/homelab/state/`
(Decyzja (f)) for one decisive reason: **the cursor and the envelopes it describes must
restore together or not at all.** A state file surviving a DB restore would make the poller
silently skip everything between the restored rows and the file's `last_uid` — a failure with
no symptom, noticed months later as missing mail.
Deriving the cursor from `max(envelope.ts)` instead was rejected outright: `ts` is the
sender's `Date:` header, not delivery time. 2 559 envelopes in the live corpus already sit at
epoch 1970.
### UIDVALIDITY
When the server changes UIDVALIDITY, every stored UID becomes meaningless. The job sweeps the
folder (`UID SEARCH ALL`), leans on Message-ID dedup to make that cheap, and stores the new
value. This is rare — and it is the failure that loses mail with no error anywhere if it is
not handled.
### Why the cursor only crosses a contiguous prefix
A message that fails is **not** stepped over. The cursor stops at the message before it, and
the next tick refetches from there — free, because the archive write is append-only and the
insert is `ON CONFLICT DO NOTHING`. Messages *after* the failure are still stored in the same
tick; they simply do not move the cursor.
The consequence is deliberate: a message that fails permanently **stalls its folder**. It
does so visibly — a non-zero error counter every tick, a cursor that stops moving, the
offending UID in the log — which is strictly better than the alternative of dropping mail
while reporting success. The runbook documents the manual `UPDATE mail_sync_state` escape
hatch for that case.
### The `n:*` trap
`UID SEARCH UID n:*` is a **range**, and when `n` exceeds the highest existing UID the server
resolves it as `highest:n` and returns the last message anyway. Without the client-side
filter, "new mails this tick" would never read zero on an idle mailbox, and every
observability claim built on that counter would be a small permanent lie.
## First contact with a folder
`MAIL_<ACCOUNT>_INITIAL_MODE` decides what the first-ever tick on a folder fetches. It has no
effect once a `mail_sync_state` row exists.
| Mode | Behaviour | When |
|---|---|---|
| `new-only` | record `UIDNEXT-1`, fetch nothing | start the corpus from now |
| `since` | `UID SEARCH SINCE <date>` (server INTERNALDATE, not the `Date:` header) | close a known gap |
| `full` | `UID SEARCH ALL` | pull a mailbox's whole history |
**gmail wants `since`.** A `new-only` first tick would leave the 48-day gap the recon measured
unfilled forever — and that gap is the entire reason this job exists. Point
`MAIL_GMAIL_INITIAL_SINCE` a few days before the newest envelope in the DB; dedup makes the
overlap free.
**fastmail is undecided by design.** The account is greenfield (zero rows, zero archive
files), and whether to pull its history depends on how big it is — a number nobody has yet.
Run `mail-imap-sync --measure` first (recon Decyzja (e) explicitly refuses to guess it).
## Message-ID collisions across accounts
`envelope.id` is a bare Message-ID and therefore **globally unique in the table**, with no
source prefix (unlike paperless, which uses `paperless:N`). A mail present in both mailboxes —
a mailing list, a forward, a CC to both addresses — has the same Message-ID in both, so
whoever inserts first sets `source` and the second is silently skipped by
`ON CONFLICT DO NOTHING`. `save_eml` still writes both `.eml` copies, because the archive path
contains the source; `raw_ref` points at one of them.
This is not corruption and it is the desired index behaviour: the content is indexed once.
It has two measurable effects worth knowing about **before** someone starts wondering about
the numbers — "new fastmail mails" reads systematically low by the shared part, and
`source` filtering in retrieval attributes such a mail to whichever account won the race.
The recon's resolution (§2.4) is to keep the behaviour and make the phenomenon visible:
`envelopes_conflict_other_source`, one extra query per suppressed insert. Re-keying to
`(source, message_id)` was rejected — it breaks the frozen envelope contract, requires
rewriting 225 030 ids and every `raw_ref`, and buys duplicates in the index.
## What this job does NOT do
**It does not chunk and does not embed.** That split is the architecture, not an omission
(recon §3.2): fetching is network-bound and belongs on the 24/7 node; chunking and embedding
need Ollama on SOLARIA. Coupling them would mean mail only arrives when the desktop is on.
The handoff needs no queue of its own — **an envelope with no `document_chunk` rows *is* the
queue**, drained by `mail-body-ingest --only-unchunked` from the `kb-ingest` timer. That is
self-healing in a way a timestamp cursor is not: an interrupted run, a mail fetched while the
GPU slept, a message inserted with a backdated header — each stays pending until it has
chunks, with nothing to reconcile.
`--since` was explicitly rejected as the glue: it filters on `envelope.ts`, so a mail
delivered today with a month-old header date would fall outside the window and never be
chunked at all.
## Headers at INSERT time
New envelopes carry `entities[type=headers]` immediately, via
`kb_mail.headers.parse_headers_resilient` (typed parse, compat32 fallback). Historical
envelopes only have headers because a separate backfill job walked all 225 030 of them later.
Without this, `mail_body_ingest.build_prefix` would produce
`Temat: (brak tematu) | Od: ? | Data: …` for every new mail — with no error, just permanently
worse retrieval (recon §2.5 i).
## Secrets
Environment only (Decyzja (c)). There is no `--password` or `--user` flag: `--dsn <secret>`
lands in `ps` output and shell history. Values go into the existing `/opt/homelab/kb/.env`
(root-owned `0600`), which systemd reads **as root before dropping to `User=oskar`** — so the
app passwords reach the process without being readable by `oskar` at rest. The repo ships only
`jobs/mail-imap-sync/env.example` with placeholders.
## Stats must balance
```
uids_seen = processed + vanished + errors
processed = archived + archive_exists
processed = envelopes_inserted + envelopes_skipped_dup + envelopes_conflict_other_source
```
`headers_fallback`, `uidvalidity_resets`, `folder_errors` and `account_errors` are labels and
diagnostics, deliberately outside the equations — a folder that failed on EXAMINE never
produced a UID to account for. An unbalanced run is a failure regardless of what else it
reports.
## Exit codes
| Code | Meaning |
|---|---|
| 0 | Balanced, zero errors at any level |
| 1 | Any message/folder/account error, or an unbalanced sum |
| 2 | Configuration error — missing/malformed env, no DSN, no archive directory |
Failures are isolated per account: a Gmail app password that stopped working must not also
stop Fastmail from syncing (the same stage isolation `cyclic_ingest` uses).
## Modes
| Invocation | Effect |
|---|---|
| *(default)* | Dry run: EXAMINE + SEARCH, report what a real tick would pull. No FETCH, no writes, no `.prom`. |
| `--apply` | The real thing, and the only mode that publishes metrics. |
| `--measure` | `STATUS (MESSAGES UIDNEXT UIDVALIDITY)` per folder, then exit. Read-only, no DB. |
| `--limit N` | At most N messages per folder per tick — a brake for a first big sweep; the rest follows next tick. |
## Metrics
`/opt/homelab/state/node-exporter/kb-mail-sync.prom`, written atomically (tmp + rename) and
scraped by node_exporter's textfile collector on PIHA.
| Metric | Notes |
|---|---|
| `kb_mail_sync_last_run_timestamp` | every run |
| `kb_mail_sync_last_success_timestamp` | carried forward across a failed run, so a blip does not reset the staleness clock |
| `kb_mail_sync_last_exit_code` | |
| `kb_mail_sync_envelopes_inserted{account}` | |
| `kb_mail_sync_envelopes_skipped_dup{account}` | |
| `kb_mail_sync_conflict_other_source{account}` | the cross-account overlap above |
| `kb_mail_sync_uidvalidity_resets{account}` | |
| `kb_mail_sync_errors{account}` | message + folder + account levels |
| `kb_mail_sync_last_message_ts{account}` | `max(envelope.ts)`; omitted for an account with no rows |
One alert, `KbMailSyncStale` (`services/fleet-prometheus/rules/kb-mail-sync.yml`): 6 h without
a successful tick. **"No new mail for X days" was considered and rejected** (recon §3.4) —
zero new mail is a legal state of a mailbox, an alert that fires on a healthy system gets
muted, and a muted alert is not there on the day it matters. A per-account warning on
`kb_mail_sync_last_message_ts` may be worth adding after a month of observation, with a
threshold taken from measured behaviour rather than a guess.
## Definition of Done
Per `CLAUDE.md`: `pytest` passes (80 for this job, 111 for `kb-mail` including the adapter and
the cursor planner) plus a CLI smoke run. The tests drive the real `ImapClient` over a fake
`imaplib` connection, so what is covered is the adapter's actual protocol parsing: new
messages, UIDVALIDITY invalidation, dedup, resumption after an interrupted run, cross-account
collisions, account isolation, expunged UIDs, a dry run that touches nothing, and the metric
carry-forward.
**Nothing here has ever connected to a live mailbox.** The first real sync is an operator
step, supervised, per `kb/runbooks/mail-sync-run.md`.
## Follow-ups
- `mail_imap_sync.prom` and `documents_ingest.cyclic_ingest`'s renderer are two copies of the
same idea (this one supports labels, that one does not). Worth folding into one shared
helper the next time either changes.
- `kb-mail-sync.timer` joins the "shadow-deploy family" — units installed outside GitOps drift
detection — listed in `hosts/piha/jobs.yaml`. That is open question 5 of
`kb/subsystems/recon-multiagent.md`; the list is now one item longer, knowingly.

4
kb/services/kb-postgres.md
View file

@ -102,6 +102,10 @@ The `envelope` table is the frozen cross-source envelope (see `kb/subsystems/kb-
Future migrations go in `init/` as `002_*.sql`, `003_*.sql`, …. Postgres runs `initdb` scripts only on a fresh volume — for existing instances apply migrations with `psql` directly.
Applied so far: `001` envelope, `002` document_chunk, `003` chunk model key + `excluded_reason`, `004` document_summary, `005` `mail_sync_state`.
`005_mail_sync_state.sql` (2026-08-06) adds the per-folder IMAP sync cursor for `jobs/mail-imap-sync`, keyed `(account, folder)`. A table rather than a file under `/opt/homelab/state/` for one decisive reason: the cursor and the envelopes it describes must restore together or not at all. A state file surviving a DB restore would make the poller silently skip everything between the restored rows and the file's `last_uid` — a failure with no symptom. Apply it by hand on the live instance (`kb/runbooks/mail-sync-run.md` §4); the DDL is `IF NOT EXISTS`, so repeating it is safe.
## Connection string
```

55
kb/services/pkg-kb-mail.md
View file

@ -3,15 +3,58 @@ okf: "0.1"
type: service
visibility: private
status: active
updated: 2026-08-03
stub: true
links: []
updated: 2026-08-06
links:
- job-mail-imap-sync.md
- job-mail-body-ingest.md
- job-gmail-bulk-import.md
---
# kb-mail
Biblioteka wspoldzielona: model koperty KB, helpery asyncpg do bazy, append-only archiwum .eml.
Shared library for the mail pillar. Everything that more than one mail job needs lives here,
by rule rather than by taste: the corpus rests on 225 030 envelopes whose ids, archive paths
and header entities were produced by one implementation, and a second copy of any of those
derivations would silently stop agreeing with the first.
> **Stub.** Katalog z kodem nie ma README — ten dokument powstal automatycznie w migracji do KB OKF (etap 2). Opis jednozdaniowy wyciagniety z: CLAUDE.md, sekcja `Shared Python Libraries (packages/)`. Pelna dokumentacja do napisania.
**Kod:** `packages/kb-mail/` (`src/kb_mail/`, `pyproject.toml`, `tests/`)
**Kod:** `packages/kb-mail/ (src/, pyproject.toml, tests/)`
Dependencies: `asyncpg`, `structlog`. Nothing else — the IMAP transport is stdlib `imaplib`.
## Modules
| Module | What it holds | Used by |
|---|---|---|
| `envelope` | The frozen cross-source `Envelope` dataclass (mirrors `envelope` in kb-postgres). Additive contract. | everything |
| `db` | asyncpg helpers: `insert_envelope` (returns the command tag), `get_envelope`, `rows_affected`, `envelope_source` | all jobs |
| `archive` | `save_eml` — append-only `.eml` writer, `{root}/{source}/{YYYY}/{MM}/{id}.eml`. `FileExistsError` is the "already have it" signal, never masked. | bulk-import, imap-sync |
| `text` | `sanitize_surrogates`, `strip_nul` — both mandatory before anything reaches `text`/`jsonb` | all parsing paths |
| `chunking` | `chunk_text` (2400/600 chars), extracted in Krok 0 | mail-body-ingest, documents-ingest |
| `headers` | `parse_headers`, `parse_headers_fallback`, `parse_headers_resilient` — the `entities[type=headers]` shape, typed parse with a compat32 fallback | header-backfill, imap-sync |
| `message` | `message_id`, `parse_date`, `parse_attachments`, `eml_ref`, `EPOCH` — envelope id, timestamp, attachment manifest, archive path | bulk-import, imap-sync |
| `imap` | `ImapAccount`, `ImapClient`, `FolderStatus` — read-only IMAP over stdlib `imaplib` | imap-sync |
| `sync_state` | `mail_sync_state` accessors + `plan_folder_sync` / `contiguous_last_uid` | imap-sync |
`headers` and `message` were extracted from `gmail-header-backfill` and `gmail-bulk-import`
respectively when `mail-imap-sync` needed the same derivations (2026-08-06). Both jobs
re-export them under their original names, so their CLIs and test suites are unchanged. This
is the same move as `chunking` in Krok 0 — extract, do not copy.
## Why the IMAP transport lives here and not in the job
The adapter is protocol code, and the sync-cursor semantics (UIDVALIDITY invalidation, how far
a cursor may advance) are protocol semantics — both are worth testing in isolation, and both
would otherwise be buried in a job's loop where a wrong branch is invisible. `kb_mail.imap` and
`kb_mail.sync_state` therefore carry the rules; `jobs/mail-imap-sync` carries the orchestration
and the policy. See `kb/services/job-mail-imap-sync.md` for the behaviour they add up to.
## Tests
```bash
pip install -e "packages/kb-mail[dev]"
cd packages/kb-mail && pytest
```
111 unit tests, no DB or network. `tests/test_db.py` is additionally marked `integration` and
needs a live kb-postgres (`KB_TEST_DSN`, default `localhost:5433`) — it is the only file that
does, and it errors rather than skips when the DB is absent.

35
kb/subsystems/kb-mail-pillar.md
View file

@ -3,7 +3,7 @@ okf: "0.1"
type: subsystem
visibility: private
status: active
updated: 2026-06-24
updated: 2026-08-06
links: []
---
@ -27,9 +27,22 @@ Realna potrzeba to najpierw **archiwum**, nie „RAG nad mailami". Dwie warstwy,
Gmail **nie jest porzucany** — zostaje jako konto śmieciowe / loginy / 2FA. Stąd maile mają dwa żywe wejścia:
- **Fastmail** — `source: fastmail`, adapter **JMAP** (read-only token). Primary: tu ląduje sensowna poczta na przyszłość.
- **Fastmail** — `source: fastmail`, adapter **IMAP** (hasło aplikacji). Primary: tu ląduje sensowna poczta na przyszłość.
- **Gmail** — `source: gmail`, adapter **IMAP** (protokół, nie Gmail API → przenośność). Ciągły sync żywej skrzynki.
> **Korekta 2026-08-06 — Fastmail przez IMAP, nie JMAP.** Do tej daty ten dokument (i §7, §9
> oraz §10 planu fazy mailowej) przewidywał dla Fastmaila **JMAP** i osobny `jobs/fastmail-poller`.
> Zapis historyczny: *„Fastmail — adapter JMAP (read-only token)"*, 2026-06-24.
>
> Decyzja z 2026-08-06 (recon `kb/audits/mail-sync-2026-08-06.md` Decyzja (b), zatwierdzona
> przez operatora) domyka otwartą od czerwca decyzję „unifikacja adaptera" z §9 na rzecz
> **jednego wspólnego IMAP-a dla obu kont**, w jednym jobie `jobs/mail-imap-sync`. Powody:
> JMAP synchronizuje po `state` — elegancko i niepotrzebnie przy jednym ticku na godzinę
> i ~37 mailach na dobę, skoro UIDVALIDITY/UIDNEXT rozwiązuje ten sam problem i tak trzeba go
> zaimplementować dla Gmaila; jeden adapter to jeden zestaw testów, jedna klasa błędów i jedna
> ścieżka hardeningu 8-bitowych nagłówków. JMAP nie jest zamknięty na zawsze — koperta
> i archiwum są protokołowo obojętne, więc wymiana transportu nie dotyka danych.
Plus jednorazowy **bulk historyczny Gmaila** (eksport „All Mail" / Takeout → surowy dump do archiwum). Operacja odwracalna i niezależna od reszty pipeline'u — robimy pierwsza. Urgency spadła (konto żyje), ale historia warta zassania od razu.
---
@ -81,7 +94,9 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki).
## 7. Deploy
- Wszystko w `homelab-codex`, przez Git na SATURN, konwencja override `hosts/<node>/runtime/<svc>/`.
- Usługi: `jmap-poller` (Fastmail), `imap-poller` (Gmail), `indexer`, embed (ollama na SOLARIA), `postgres+pgvector`, `mail-agent`; bulk importer jako one-shot job.
- Usługi: `mail-imap-sync` (Fastmail + Gmail, jeden job — korekta 2026-08-06; wcześniej
planowane jako osobne `jmap-poller` + `imap-poller`), `indexer`, embed (ollama na SOLARIA),
`postgres+pgvector`, `mail-agent`; bulk importer jako one-shot job.
- Deploy skryptem czytającym `inventory/topology.yaml`.
---
@ -90,8 +105,10 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki).
1. ✅ Zamroź kopertę + postaw Postgres+pgvector. *(2026-06-17)*
2. ✅ **Bulk Gmail historyczny → archiwum** — `jobs/gmail-bulk-import/` — **KOD GOTOWY** *(2026-06-24)*; nie uruchomiony (Takeout ~27 GB na SOLARIA, do transferu na PIHA).
3. Fastmail JMAP live ingest → archiwum.
4. Gmail IMAP live sync → archiwum.
3. ~~Fastmail JMAP live ingest~~ → **Fastmail IMAP live sync** → archiwum. *(korekta
2026-08-06; kod gotowy, pierwszy żywy run po stronie operatora —
`kb/runbooks/mail-sync-run.md`)*
4. Gmail IMAP live sync → archiwum. *(j.w. — ten sam job `jobs/mail-imap-sync`)*
5. Filtr archiwum→indeks.
6. Indexer (parse → chunk → embed bge-m3) → pgvector.
7. Cienki agent maili + tool dla warstwy 4.
@ -100,8 +117,12 @@ Załączniki: **II tura** (MVP = czysty tekst + nagłówki).
## 9. Decyzje otwarte (do przyklepania przed/w trakcie startu)
- **Sizing Gmaila** — ile realnie waży „All Mail"? (przesądza node/dysk archiwum).
- **Unifikacja adaptera** — jeden wspólny IMAP dla Fastmail + Gmail (mniej kodu) vs JMAP dla Fastmail + IMAP dla Gmail (JMAP bogatszy)?
- ✅ **Sizing Gmaila** — ZAMKNIĘTE: 225 030 kopert, archiwum ~27 GB na PIHA (Etap B, 2026-08-06).
- ✅ **Unifikacja adaptera** — ZAMKNIĘTE 2026-08-06 na rzecz **jednego IMAP-a** dla obu kont
(Decyzja (b) reconu, uzasadnienie w §2 wyżej).
- **Sizing Fastmaila** — OTWARTE, i celowo: przesądza o tym, czy ciągniemy historię konta czy
tylko przyrost. Rozstrzyga pomiar `mail-imap-sync --measure`, nie zgadywanie —
`kb/runbooks/mail-sync-run.md` §5.
- **Reguły filtra** — startowa lista blacklist domen/nagłówków.
- Vector store: pgvector **przyklepane** (spine).
- Embed model: bge-m3 **przyklepane**.

40
services/fleet-prometheus/rules/kb-mail-sync.yml Normal file
View file

@ -0,0 +1,40 @@
# fleet-prometheus kb-mail-sync rules — faza mailowa Krok 7
# (recon kb/audits/mail-sync-2026-08-06.md §3.4).
#
# Same delivery convention as kb-ingest.yml and liveness.yml: no Alertmanager. These rules
# only make alerts FIRING (visible at GET /api/v1/alerts on this Prometheus instance);
# brain-watchdog@PIHA polls that API and forwards to Telegram. Do NOT add Alertmanager here.
#
# Metrics come from jobs/mail-imap-sync's textfile output
# (/opt/homelab/state/node-exporter/kb-mail-sync.prom), scraped by node_exporter on PIHA —
# every series from that target already carries node="piha" (fleet-node job's static label).
#
# Until the operator installs and enables kb-mail-sync.timer the metric does not exist, and
# `time() - <missing series>` produces no result — so this file is inert on a fleet where the
# job has not been set up. It does not need to be deployed in lockstep with the job.
groups:
- name: kb-mail-sync
rules:
- alert: KbMailSyncStale
# 6 missed hourly ticks. This is the ONE alert this job gets, and deliberately so:
# it answers "is the poller working at all", which has no false positives — unlike
# any threshold on how much mail arrived.
#
# "No new mail for X days" was considered and REJECTED (recon §3.4). Zero new mail is
# a legal state of a mailbox — a holiday, a weekend, traffic moved to the other
# account. An alert that fires on a healthy system gets muted, and a muted alert is
# not there on the day it is needed.
expr: time() - kb_mail_sync_last_success_timestamp{node="piha"} > 21600
for: 5m
labels:
severity: critical
annotations:
summary: "kb-mail-sync stale on {{ $labels.node }}"
description: "kb-mail-sync.timer has not completed a successful run in over 6h on {{ $labels.node }} — the mail corpus has stopped growing. Check `journalctl -u kb-mail-sync.service` and the latest /opt/homelab/logs/kb-mail-sync/run-*.log; an expired or revoked IMAP app password is the most likely cause."
# Deliberately NOT defined here, pending a month of observation (recon §3.4):
# a per-account warning on kb_mail_sync_last_message_ts (age of the newest mail).
# It needs a per-account threshold — gmail is the junk account and days of silence
# there would be odd, while for fastmail it would not be — and that threshold should
# come from measured behaviour, not from a guess made before the first run.