systemary.io

Sysadmin tutorials, Linux server hardening, monitoring & observability

Prometheus Alertmanager Setup Guide

by Liam Foster
Prometheus Alertmanager Setup Guide

Alertmanager is the notification layer for Prometheus. It handles deduplication, grouping, and routing of alerts to your incident channels. Without it, you get raw alert noise; with it configured correctly, you get actionable incidents.

The Model

Prometheus fires alerts based on rules. Alertmanager receives those alerts and decides what to do with them:

  1. Receive – Prometheus sends firing/resolved alerts via HTTP push.
  2. Deduplicate & group – Alertmanager combines identical alerts across instances.
  3. Route – Match alert labels against a tree of routing rules.
  4. Inhibit – Suppress alerts based on other active alerts.
  5. Notify – Send grouped alerts to receivers (Slack, PagerDuty, email, webhook).

The core insight: Alertmanager is a state machine for alert lifecycle, not a monitoring engine. Prometheus decides when to alert; Alertmanager decides where and how often.

Installation & Startup

Download the binary from prometheus.io/download or install via package manager:

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
sudo mv alertmanager-0.26.0.linux-amd64/alertmanager /usr/local/bin/

Create a minimal config file at /etc/alertmanager/alertmanager.yml:

global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://localhost:5001/'

Start the daemon:

alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager

Alertmanager listens on :9093 by default. Verify with curl http://localhost:9093/-/healthy.

Connect Prometheus to Alertmanager

Edit your Prometheus config (prometheus.yml) to push alerts:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

rule_files:
  - 'alerts.yml'

Reload Prometheus. Check the Alerts page in the UI — you should see alert rules and their state. Alertmanager status appears under "Alertmanagers".

Gotcha: Alertmanager must be running before Prometheus starts firing alerts, or alerts will be queued and may timeout. If Prometheus can't reach Alertmanager, check firewall rules and DNS resolution. If you're also serving Alertmanager over HTTPS locally, this guide on local HTTPS development setup covers the certificate and proxy configuration you'll need.

Routing & Grouping

Routing is a tree. Each node matches labels and forwards to a receiver or sub-route:

route:
  receiver: 'default'
  group_by: ['alertname', 'instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h

  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      group_wait: 0s
      repeat_interval: 5m

    - match_re:
        service: '^(api|db)$'
      receiver: 'backend-team'
      group_by: ['alertname']

    - match:
        team: devops
      receiver: 'devops-slack'
      continue: true

Group by determines which alerts are bundled. If you group by ['alertname'], all instances of "HighCPU" fire as one notification. If you group by ['alertname', 'instance'], each instance sends separately.

Group wait is the hold time before sending the first notification (default 10s). Use 0s for critical pages, 30s–60s for routine alerts.

Group interval is the wait between updates to an open group (default 5m). Repeat interval is how often to re-send resolved groups (default 12h).

Continue: true allows an alert to match multiple routes. Useful for fan-out (send to both Slack and a webhook).

Inhibition Rules

Inhibit rules suppress alerts when a condition is met. Example: silence "DiskSpaceWarning" when "NodeDown" is firing:

inhibit_rules:
  - source_match:
      severity: critical
      alertname: NodeDown
    target_match_re:
      alertname: '.*'
    equal: ['instance']

  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'instance']

Read it as: "If a source alert matches these labels, and a target alert matches these labels, and they share these label values, suppress the target."

Common pattern: suppress warnings when the critical parent is firing. This cuts noise without losing signal.

Receivers: Slack Example

Add a Slack receiver:

receivers:
  - name: 'slack-prod'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#incidents'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Labels.instance }} - {{ .Annotations.summary }}\n{{ end }}'
        send_resolved: true

Get your webhook URL from Slack's app settings. Test with:

curl -X POST -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"TestAlert"}}]}' \
  http://localhost:9093/api/v1/alerts

You should see a message in Slack within seconds.

Persistence & High Availability

Alertmanager stores alert state on disk. The --storage.path flag sets the directory:

alertmanager --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager

For HA, run multiple Alertmanager instances with the same config and point Prometheus to all of them:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager-1:9093
            - alertmanager-2:9093
            - alertmanager-3:9093

Instances gossip with each other to sync state. Use --cluster.peer-address to manually bootstrap the cluster if auto-discovery fails.

When This Breaks

Alerts not firing: Check Prometheus rule syntax with promtool check rules alerts.yml. Verify alert conditions are actually met (query the metric directly).

Duplicate notifications: You're grouping too narrowly. Add more labels to group_by or increase group_wait.

Alerts disappear after resolve: Alertmanager forgets resolved alerts after resolve_timeout (default 5m). If Prometheus doesn't re-send the resolved status, the alert vanishes. Check Prometheus alert rule for duration.

Webhook receiver not firing: Verify the URL is reachable from Alertmanager's host. Check firewall and DNS. Alertmanager logs HTTP errors to stderr.

Inhibition not working: Ensure source and target labels match exactly. Use promtool check config to validate the config syntax.

Trade-offs

Grouping vs. noise: Tight grouping (few labels) reduces notifications but risks burying important instances. Loose grouping (many labels) is noisier but more precise. Start with ['alertname', 'instance'] and adjust.

Group wait vs. latency: Longer wait times batch alerts but delay notification. Critical pages need group_wait: 0s; background alerts can wait 30–60s.

Persistence vs. simplicity: Single-instance Alertmanager is simpler but loses state on restart. HA cluster is more resilient but harder to debug. For teams running Alertmanager alongside a WordPress-backed status page, the WordPress database optimization benchmarks on wpcompass.io are worth a look to ensure your status page stays responsive under alert load.

One-line Takeaway

Alertmanager turns Prometheus rule firings into grouped, routed, deduplicated notifications — configure grouping and routing first, add receivers second, and tune inhibition rules to cut noise.