Skip to content
BoringStack
GitHub

Observability

3 min read

Observability

WITH_OBSERVABILITY=1 drops a complete metrics-and-logs stack into your local environment. System, API, Postgres, and Valkey telemetry run on the same host.

Design choicesMetrics & Logs

0

Per-event cost

Grafana

Dashboard Hub

Loki

Log Storage

WITH_OBSERVABILITY=1 adds a full metrics-and-logs stack to the infra. It runs on the same host as the app; no SaaS, no per-event billing.

The default configuration covers debugging production incidents: app metrics, system metrics, container metrics, and all container logs queryable in one Grafana.

How signals flow

Section titled “How signals flow”
flowchart LR
  api["apps/api<br/>/metrics endpoint"]
  pg["postgres-exporter"]
  valkey_exp["valkey-exporter"]
  node["node-exporter<br/>host metrics"]
  prom["Prometheus<br/>scrape every 15s"]
  containers["every container<br/>stdout/stderr"]
  promtail["Promtail<br/>per-container labels"]
  loki["Loki<br/>log store"]
  grafana["Grafana<br/>dashboards + queries"]
  alert["Alertmanager<br/>routes alerts"]
  api --> prom
  pg --> prom
  valkey_exp --> prom
  node --> prom
  containers --> promtail --> loki
  prom --> grafana
  loki --> grafana
  prom --> alert

Design choices

Section titled “Design choices”
Self-hosted, not SaaS

Zero per-event cost; data stays on your infra.

Prometheus + Loki, not OpenTelemetry collector

Two well-known projects beat one unfamiliar abstraction.

Promtail labels logs per container

“Show me api-prod errors in the last hour” is one filter, not a regex.

Grafana auto-provisions datasources

Stack boots usable; no manual wiring after up -d.

Alertmanager included but not wired by default

Pager strategy is project-specific; the plumbing is there when you need it.

Bundled exporters: node, postgres, valkey

Standard metrics for incident response.

What you get out of the box

Section titled “What you get out of the box”
Grafana

Host :3010. Default admin / change-me (override via env).

Prometheus

Internal. Configurable retention (see compose/prometheus/prometheus.yml).

Loki

Internal. Per-container labels; configurable retention.

Promtail

Sidecar. Scrapes /var/lib/docker/containers/*.log.

node-exporter

Host metrics: CPU, memory, disk, network.

postgres-exporter

Postgres metrics: connections, transactions, table sizes.

valkey-exporter

Valkey metrics: hit ratio, evictions, memory.

Alertmanager

Internal. Routes defined in alertmanager/.

Drop dashboard JSON exports into compose/grafana/dashboards/ and Grafana auto-loads them within 30s. Datasources and the dashboard provider are already wired in compose/grafana/provisioning/.

Querying

Section titled “Querying”

Metrics (Prometheus, PromQL):

Terminal window
# 5-minute API request rate by route
rate(http_requests_total[5m]) by (route)


# Postgres connection count
pg_stat_database_numbackends{datname="app"}


# Worker job throughput
rate(bullmq_jobs_completed_total[1m])

See the infra/compose PromQL cheatsheet for more.

Logs (Loki, LogQL):

Terminal window
# All API errors in the last hour
{container="api-prod"} |= "level=error"


# Worker jobs by status
{container=~"api.*"} | json | event="email_delivery_completed"

See the LogQL cheatsheet.

Adding an alert

Section titled “Adding an alert”
  1. Drop a rule file in compose/prometheus/rules/ (e.g. api-error-rate.yml).
  2. Configure routing in compose/alertmanager/alertmanager.yml.
  3. docker compose restart prometheus alertmanager.

Alertmanager can route to Slack, email, PagerDuty, or a webhook. The template ships the structure; you fill in your receiver.

Adding a custom app metric

Section titled “Adding a custom app metric”

The API app’s /metrics endpoint uses the standard Prometheus client library. Define a counter / gauge / histogram in src/lib/metrics/, increment it from your code, and Prometheus picks it up on the next scrape. The convention is one file per metric domain (http-metrics.ts, queue-metrics.ts).

Cost

Section titled “Cost”

The overlay is light on memory at the default sizing; see Resource limits for the per-service knobs. Disk grows with retention windows; tune them in compose/prometheus/prometheus.yml and compose/docker-compose.observability.yml if storage matters.

Source

Section titled “Source”

compose/docker-compose.observability.yml · compose/prometheus/ · compose/grafana/ · compose/promtail/.

Section titled “Related”