83 lines
3.6 KiB
Markdown
83 lines
3.6 KiB
Markdown
|
# fleet-prometheus
|
||
|
|
|
||
|
|
Prometheus instance that is the **source of truth for fleet liveness**. Runs on
|
||
|
|
the **VPS** (ingress / control-plane host) and is reachable only over Tailscale.
|
||
|
|
|
||
|
|
This is a **clean scaffold**: right now it scrapes only itself and the VPS-local
|
||
|
|
`node_exporter`. The fleet targets, liveness rules, and alerting integration are
|
||
|
|
deliberately separate, later steps (see *Next steps* below).
|
||
|
|
|
||
|
|
## Why separate from the home `prom`
|
||
|
|
|
||
|
|
The home `prom` on **PIHA** is LAN-only and monitors the home network. This
|
||
|
|
instance is intentionally a **distinct** Prometheus dedicated to the distributed
|
||
|
|
fleet — hence the `fleet-` naming. The split is made explicit in every series:
|
||
|
|
|
||
|
|
- `external_labels: { fleet: "homelab-codex" }`
|
||
|
|
- job names prefixed/scoped for the fleet (`prometheus`, `fleet-node`)
|
||
|
|
|
||
|
|
It also avoids two anti-patterns the home `prom` has:
|
||
|
|
|
||
|
|
- **No `:latest`** — the image is pinned to `prom/prometheus:v3.5.0` (LTS).
|
||
|
|
- **No inline secrets** — `prometheus.yml` is secret-free by contract (the home
|
||
|
|
prom embeds a plaintext HAOS token; we do not).
|
||
|
|
|
||
|
|
## Stack
|
||
|
|
|
||
|
|
| Container | Image | Purpose |
|
||
|
|
|--------------------|-------------------------|--------------------------------|
|
||
|
|
| `fleet-prometheus` | `prom/prometheus:v3.5.0`| Fleet liveness Prometheus (9090) |
|
||
|
|
|
||
|
|
## Placement & exposure
|
||
|
|
|
||
|
|
- **Node:** VPS (`ubuntu-4gb-hel1-1`, Tailscale `100.95.58.48`).
|
||
|
|
- **Exposure:** `tailscale-internal` — exposed exactly like `control-plane`: a
|
||
|
|
plain published `9090:9090` port with **no** public npm reverse-proxy entry.
|
||
|
|
Reachability is constrained to the Tailscale mesh at the VPS firewall layer.
|
||
|
|
Do **not** add this to npm or public DNS.
|
||
|
|
- **Memory:** `mem_limit: 512m`. `oom_score_adj: 200` (positive) — fleet
|
||
|
|
monitoring is less critical than the control-plane (`-900`), so it is a
|
||
|
|
sacrificial OOM victim *before* the control-plane, never after.
|
||
|
|
- **Retention:** `15d` / `2GB` (whichever hits first) — tight on purpose.
|
||
|
|
|
||
|
|
## Data
|
||
|
|
|
||
|
|
TSDB lives in the named volume `fleet-prometheus_tsdb` → `/prometheus`.
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
- `prometheus.yml` — committed, **secret-free**. Global config + two scrape jobs.
|
||
|
|
- `env.example` — no secrets / no required env (kept only for layout parity).
|
||
|
|
|
||
|
|
### Scrape targets (scaffold)
|
||
|
|
|
||
|
|
| Job | Target | Notes |
|
||
|
|
|--------------|-----------------------------|-----------------------------------------|
|
||
|
|
| `prometheus` | `localhost:9090` | self |
|
||
|
|
| `fleet-node` | `host.docker.internal:9100` | VPS `node_exporter` (runs `network_mode: host`, listens on host `:9100`); reached via `host-gateway` |
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Config is syntactically valid (uses promtool inside the image):
|
||
|
|
docker run --rm -v "$PWD/services/fleet-prometheus/prometheus.yml":/etc/prometheus/prometheus.yml:ro \
|
||
|
|
prom/prometheus:v3.5.0 promtool check config /etc/prometheus/prometheus.yml
|
||
|
|
|
||
|
|
# Compose renders:
|
||
|
|
docker compose -f services/fleet-prometheus/docker-compose.yml config
|
||
|
|
|
||
|
|
# After `up -d`:
|
||
|
|
curl -sf http://localhost:9090/-/healthy
|
||
|
|
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'
|
||
|
|
./services/fleet-prometheus/healthcheck.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
## Next steps (NOT done in this scaffold)
|
||
|
|
|
||
|
|
- **Fleet targets:** add the `100.x` Tailscale `node_exporter`s for SATURN,
|
||
|
|
SOLARIA, PIHA, CHELSTY to the `fleet-node` job.
|
||
|
|
- **Liveness rules:** `rule_files` with `up == 0 for: 5m` (placeholder is
|
||
|
|
commented in `prometheus.yml`).
|
||
|
|
- **Alerting:** Alertmanager wiring, and/or `brain-watchdog` querying the
|
||
|
|
firing alerts API → Telegram (placeholder commented in `prometheus.yml`).
|