Grafana alerting sends notifications when metrics breach thresholds. It bridges your dashboards and your on-call rotations. Three pieces must exist before any alert fires: a contact point (where alerts go), a notification policy (routing logic), and an alert rule (the condition).
The Alerting Model
Grafana's alert system works like a mail carrier with sorting rules. Contact points are addresses—Slack channels, email, PagerDuty, webhooks. Notification policies are routing rules—which alerts go where, how often, whether to group them. Alert rules are the triggers—"if CPU > 80% for 5 minutes, fire." An alert rule without a contact point is a silent alarm. A contact point without a policy is unreachable.
The flow: metric query → evaluation window → alert state change → notification policy → contact point → your team. Each step is independent. That independence is the design's strength and its gotcha.
Setting Up Contact Points
Contact points live under Alerting > Contact points. Click Create contact point.
Name it plainly: slack-oncall, pagerduty-prod, email-ops. The name is how you reference it in policies.
Choose an integration type. Slack is common: paste a webhook URL (generated in your Slack workspace under App Directory > Incoming Webhooks). Test it immediately—click the bell icon. If the test notification doesn't land in your channel, the URL is wrong or your Slack workspace has IP restrictions.
Email: enter the address. Grafana sends via its configured SMTP relay. If mail doesn't arrive, check your Grafana server logs for SMTP errors. Common culprit: authentication failure or relay rejection.
PagerDuty: generate an integration key in your PagerDuty service, paste it. Grafana will post incidents to that service.
Webhook: POST to any URL. Grafana sends a JSON payload with labels, annotations, and alert state. Use this for custom tooling or internal systems.
Gotcha: Contact points are workspace-scoped (in Grafana Cloud) or instance-scoped (self-hosted). You cannot mix them across organizations without careful RBAC setup.
Building a Notification Policy
Policies live under Alerting > Notification policies. The default policy routes all alerts to the first contact point you created. You almost always want to override it.
Policies are trees. The root policy catches everything. Child policies match on label conditions and override the parent.
Start with the root policy. Set a default contact point—the fallback for any alert that doesn't match a child rule. Set a group interval (how long to wait before grouping similar alerts) and a repeat interval (how often to re-notify if an alert stays firing).
Group interval = 5m means: if 10 CPU alerts fire within 5 minutes, send one notification with all 10. Repeat interval = 4h means: if the alert is still firing after 4 hours, send another notification.
Add a child policy. Click Add nested policy. Set a matcher—for example, environment = prod. Set a contact point—perhaps slack-oncall. Now all alerts with the label environment: prod will route to that Slack channel, overriding the root.
You can nest deeper. A child of a child will inherit and override its parent's settings. Matchers use label names and values from your alert rules.
Gotcha: Matchers are AND logic within a policy, OR logic across siblings. If you have two child policies with matchers env=prod and service=api, an alert with both labels will match both. The first match wins (top-to-bottom order). Reorder policies by dragging.
Creating an Alert Rule
Alert rules live under Alerting > Alert rules. Click Create alert rule.
Step 1: Define the query. Select a data source (Prometheus, Loki, Graphite, etc.). Write a query. For Prometheus: rate(http_requests_total[5m]). For Loki: {job="api"} | json | level="error". The query must return numeric data (for threshold rules) or log lines (for log-based rules).
You can have multiple queries. Label them A, B, C. Use them in expressions—for example, B / A to compute a ratio.
Step 2: Set the condition. Choose a threshold or expression. For a simple threshold: select query A, set operator (>, <, ==), set value (80). For an expression: use math—$A > 80 and $B < 20. The condition must evaluate to true or false.
Step 3: Set the evaluation window. How long must the condition be true before firing? 5m means the condition is true for 5 consecutive minutes. 1m is more sensitive; 15m is slower to alert. Choose based on noise tolerance.
Step 4: Add labels and annotations. Labels are metadata for routing (environment, service, severity). Annotations are human text (summary, runbook, description). You can use template variables: {{ $labels.instance }} interpolates the instance label from the metric.
Example labels:
environment: prod
service: api
severity: critical
Example annotations:
summary: "API response time > 500ms on {{ $labels.instance }}"
runbook: "https://wiki.internal/api-latency-runbook"
Step 5: Assign a folder and evaluation group. Folders organize rules. Evaluation groups determine how often the rule is evaluated (usually 1m or 5m). All rules in a group are checked on that cadence.
Click Save rule. Grafana will evaluate it on the next cycle.
Gotcha: A newly created rule may not fire immediately if the metric has no recent data. Grafana treats "no data" as a separate alert state. You must configure what happens when a query returns no data: fire an alert, resolve it, or do nothing. Set this in the rule's Configure no data and error handling section.
Testing and Debugging
After creating a rule, check Alerting > Alert rules. Find your rule. The State column shows Normal, Pending, or Firing. Click the rule to see its query result and current value.
If the rule is Pending and not firing, the evaluation window hasn't elapsed. Wait.
If the rule fires but you don't receive a notification, check:
- The rule's labels match a notification policy matcher.
- The matched policy has a contact point assigned.
- The contact point was tested and works.
- Check Alerting > Alert instances to see if the alert instance exists.
If the alert instance exists but no notification was sent, check Grafana's logs for policy evaluation errors. Self-hosted: docker logs <grafana-container> or tail /var/log/grafana/grafana.log. Grafana Cloud: check the audit log.
When This Breaks
Alert storms: if a rule matches thousands of series (e.g., per-instance CPU), you'll get thousands of notifications. Scope the query with {job="api"} or use recording rules to pre-aggregate.
Label cardinality: avoid labels with unbounded values (timestamps, request IDs). Use only discrete labels (environment, service, region).
Evaluation lag: if your evaluation group is 5m and your condition window is 5m, the earliest a new alert can fire is 10m after the metric breach. Reduce both to 1m if latency matters. If you're also evaluating hosting performance as part of your infrastructure decisions, the comparison on tinjauhost.biz.id covers server speed benchmarks worth reviewing.
Notification deduplication: Grafana groups alerts by their label set. Two alerts with identical labels are grouped into one notification. If you want separate notifications, add a unique label (not recommended).
One-Line Takeaway
Grafana alerting is three independent pieces—contact points, policies, rules—each must be correct for alerts to reach humans; test the contact point first, then the policy routing, then the rule query.