oskar 0a8a668b05 feat(kb): SPLIT fleet-prometheus -> service + decision + runbook

kb/services/fleet-prometheus.md (Stack, Placement & exposure, Data, Next steps)
kb/decisions/fleet-prometheus-osobny-od-prom.md ("Why separate from the home prom")
kb/runbooks/fleet-prometheus-deploy.md (Configuration, Verify)

Wzajemne links. Tresc sekcji nietknieta; kontrola: multizbior niepustych
linii czesci == oryginal z HEAD.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-04 16:58:46 +02:00

2 KiB

Raw Blame History

okf

type

visibility

status

updated

links

0.1

service

private

active

2026-06-24

../decisions/fleet-prometheus-osobny-od-prom.md

../runbooks/fleet-prometheus-deploy.md

fleet-prometheus

Prometheus instance that is the source of truth for fleet liveness. Runs on the VPS (ingress / control-plane host) and is reachable only over Tailscale.

This is a clean scaffold: right now it scrapes only itself and the VPS-local node_exporter. The fleet targets, liveness rules, and alerting integration are deliberately separate, later steps (see Next steps below).

Stack

Container	Image	Purpose
`fleet-prometheus`	`prom/prometheus:v3.5.0`	Fleet liveness Prometheus (9090)

Placement & exposure

Node: VPS (ubuntu-4gb-hel1-1, Tailscale 100.95.58.48).
Exposure: tailscale-internal — exposed exactly like control-plane: a plain published 9090:9090 port with no public npm reverse-proxy entry. Reachability is constrained to the Tailscale mesh at the VPS firewall layer. Do not add this to npm or public DNS.
Memory: mem_limit: 512m. oom_score_adj: 200 (positive) — fleet monitoring is less critical than the control-plane (-900), so it is a sacrificial OOM victim before the control-plane, never after.
Retention: 15d / 2GB (whichever hits first) — tight on purpose.

Data

TSDB lives in the named volume fleet-prometheus_tsdb → /prometheus.

Next steps (NOT done in this scaffold)

Fleet targets: add the 100.x Tailscale node_exporters for SATURN, SOLARIA, PIHA, CHELSTY to the fleet-node job.
Liveness rules: rule_files with up == 0 for: 5m (placeholder is commented in prometheus.yml).
Alerting: Alertmanager wiring, and/or brain-watchdog querying the firing alerts API → Telegram (placeholder commented in prometheus.yml).

2 KiB Raw Blame History