Prometheus collects time-series metrics; Grafana visualizes them. Together they form the standard observability stack for infrastructure and application monitoring. This guide walks the setup, configuration, and the decisions that trip up first deployments.
The Architecture Model
Prometheus is a pull-based metrics database. It scrapes HTTP endpoints at regular intervals, stores the data locally, and exposes a query language (PromQL) to retrieve it. Grafana is a visualization layer that queries Prometheus and renders dashboards, alerts, and annotations. The separation matters: Prometheus handles retention and query logic; Grafana handles UI and multi-source aggregation.
The flow is linear: your application or node exporter exposes metrics → Prometheus scrapes and stores → Grafana queries and displays. No broker, no message queue. This simplicity is the stack's strength and its constraint.
Installation and Service Setup
Prometheus
Download the latest binary from prometheus.io. Extract and move the binary to /usr/local/bin. Create a system user and directories:
useradd --no-create-home --shell /bin/false prometheus
mkdir -p /etc/prometheus /var/lib/prometheus
chown prometheus:prometheus /var/lib/prometheus
Create /etc/prometheus/prometheus.yml with a basic scrape config:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
Set ownership: chown -R prometheus:prometheus /etc/prometheus. Create a systemd unit at /etc/systemd/system/prometheus.service:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus
Restart=always
[Install]
WantedBy=multi-user.target
Enable and start: systemctl daemon-reload && systemctl enable prometheus && systemctl start prometheus. Verify at http://localhost:9090/graph.
Grafana
Install via package manager or binary. On Debian/Ubuntu:
apt-get install -y software-properties-common
add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
apt-get update && apt-get install -y grafana-server
Enable and start: systemctl enable grafana-server && systemctl start grafana-server. Access at http://localhost:3000 (default credentials: admin/admin). Change the password immediately.
Configuring Scrape Targets
Prometheus scrapes metrics from targets. Each target is a job with a list of static or dynamically discovered endpoints. Add to prometheus.yml:
scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['localhost:9100'] # node_exporter
- job_name: 'app'
static_configs:
- targets: ['app-server:8080']
metrics_path: '/metrics'
The metrics_path defaults to /metrics; override if your app exposes them elsewhere. Scrape_interval can be per-job; shorter intervals increase cardinality and storage cost. 15 seconds is a safe default; production systems often use 30s or 60s.
Node Exporter
To monitor Linux hosts, deploy node_exporter. Download from prometheus.io, extract, and create a systemd unit:
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=always
[Install]
WantedBy=multi-user.target
Start it and add its endpoint to Prometheus. It exposes ~100 metrics by default: CPU, memory, disk, network, and process-level stats.
Wiring Grafana to Prometheus
In Grafana, add Prometheus as a data source: Settings → Data Sources → Add Data Source. Select Prometheus. Set the URL to http://localhost:9090 (or your Prometheus server). Save and test.
Create a dashboard: + → Dashboard → Add a Panel. In the query editor, write a PromQL expression. Start simple:
node_cpu_seconds_total{mode="idle"}
This returns raw CPU idle time. More useful is the rate of change:
rate(node_cpu_seconds_total{mode="user"}[5m])
This calculates the per-second increase over the last 5 minutes, giving you CPU usage as a percentage (multiply by 100 in the visualization). Set the panel type to Graph, configure axes, and save.
Storage and Retention
Prometheus stores metrics in a local time-series database (TSDB) at /var/lib/prometheus by default. It retains data for 15 days; change this with the --storage.tsdb.retention.time flag (e.g., 30d). Longer retention increases disk usage linearly with cardinality and scrape frequency.
Estimate storage: (cardinality × scrape_interval × retention_days) / 3600 ≈ GB. A typical setup with 5,000 metrics, 15-second scrapes, and 15-day retention uses ~30 GB. Monitor disk space; Prometheus will stop accepting new samples when the filesystem fills.
For long-term storage, use a remote write backend (Thanos, Cortex, or cloud providers). This is outside this guide but critical for production.
When This Breaks
Scrape targets missing or failing: Check the Targets page at http://localhost:9090/targets. Red endpoints mean Prometheus cannot reach them. Verify network connectivity, firewall rules, and that the exporter is running. A target in the "Down" state logs a reason; click it for details.
High cardinality explosion: If Prometheus memory or disk usage spikes, check for unbounded label values (e.g., user IDs or request paths in metrics). These create a new time series per value. Use relabel_configs to drop or rename labels:
metric_relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds_bucket'
action: drop
Grafana cannot query Prometheus: Check the data source URL and network routing. Verify the Prometheus service is listening on the correct interface (default: localhost:9090). If Grafana is in a container, use http://prometheus:9090 (service name) instead of localhost.
Out of disk space: Prometheus stops ingesting when the filesystem is full. Add storage, increase retention flags, or reduce scrape frequency. Monitor /var/lib/prometheus size regularly.
Trade-offs and Decisions
Prometheus is pull-based, not push-based. This means you control scrape timing and avoid network surprises, but you must expose an HTTP endpoint. For ephemeral workloads (batch jobs, serverless), use the Pushgateway, which adds a hop and single point of failure.
Grafana is stateless; all state lives in Prometheus. This simplifies deployment but means you cannot store computed metrics in Grafana itself. Complex transformations belong in Prometheus recording rules, not dashboard math. If you are choosing a language for the services feeding these metrics, the tradeoffs covered in this writeup on Python vs Go for backend services are worth reviewing before you commit to an instrumentation approach.
Local storage is simple but not distributed. For high availability, run multiple Prometheus instances behind a load balancer and query both, accepting some inconsistency. For true HA, use Thanos or a managed service. If your Prometheus instance runs on a self-hosted dev server, the comparison on devbox.id covers environment options that pair well with this kind of setup.
One-Line Takeaway
Prometheus + Grafana is the standard because it separates concerns cleanly: Prometheus owns metrics collection and storage, Grafana owns visualization, and you own the glue (scrape configs and PromQL).