homelab-codex-ws/services/fleet-prometheus

fleet-prometheus

Prometheus instance that is the source of truth for fleet liveness. Runs on the VPS (ingress / control-plane host) and is reachable only over Tailscale.

This is a clean scaffold: right now it scrapes only itself and the VPS-local node_exporter. The fleet targets, liveness rules, and alerting integration are deliberately separate, later steps (see Next steps below).

Why separate from the home prom

The home prom on PIHA is LAN-only and monitors the home network. This instance is intentionally a distinct Prometheus dedicated to the distributed fleet — hence the fleet- naming. The split is made explicit in every series:

• external_labels: { fleet: "homelab-codex" }
• job names prefixed/scoped for the fleet (prometheus, fleet-node)

It also avoids two anti-patterns the home prom has:

• No :latest — the image is pinned to prom/prometheus:v3.5.0 (LTS).
• No inline secrets — prometheus.yml is secret-free by contract (the home prom embeds a plaintext HAOS token; we do not).

Stack

Container	Image	Purpose
fleet-prometheus	prom/prometheus:v3.5.0	Fleet liveness Prometheus (9090)

Placement & exposure

• Node: VPS (ubuntu-4gb-hel1-1, Tailscale 100.95.58.48).
• Exposure: tailscale-internal — exposed exactly like control-plane: a plain published 9090:9090 port with no public npm reverse-proxy entry. Reachability is constrained to the Tailscale mesh at the VPS firewall layer. Do not add this to npm or public DNS.
• Memory: mem_limit: 512m. oom_score_adj: 200 (positive) — fleet monitoring is less critical than the control-plane (-900), so it is a sacrificial OOM victim before the control-plane, never after.
• Retention: 15d / 2GB (whichever hits first) — tight on purpose.

Data

TSDB lives in the named volume fleet-prometheus_tsdb → /prometheus.

Configuration

• prometheus.yml — committed, secret-free. Global config + two scrape jobs.
• env.example — no secrets / no required env (kept only for layout parity).

Scrape targets (scaffold)

Job	Target	Notes
prometheus	localhost:9090	self
fleet-node	host.docker.internal:9100	VPS node_exporter (runs network_mode: host, listens on host :9100); reached via host-gateway

Verify

# Config is syntactically valid (uses promtool inside the image):
docker run --rm -v "$PWD/services/fleet-prometheus/prometheus.yml":/etc/prometheus/prometheus.yml:ro \
  prom/prometheus:v3.5.0 promtool check config /etc/prometheus/prometheus.yml

# Compose renders:
docker compose -f services/fleet-prometheus/docker-compose.yml config

# After `up -d`:
curl -sf http://localhost:9090/-/healthy
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'
./services/fleet-prometheus/healthcheck.sh

Next steps (NOT done in this scaffold)

• Fleet targets: add the 100.x Tailscale node_exporters for SATURN, SOLARIA, PIHA, CHELSTY to the fleet-node job.
• Liveness rules: rule_files with up == 0 for: 5m (placeholder is commented in prometheus.yml).
• Alerting: Alertmanager wiring, and/or brain-watchdog querying the firing alerts API → Telegram (placeholder commented in prometheus.yml).