systemary.io

Sysadmin tutorials, Linux server hardening, monitoring & observability

Kubernetes Node Monitoring with Prometheus

by Liam Foster
Kubernetes Node Monitoring with Prometheus

Kubernetes node monitoring with Prometheus means collecting CPU, memory, disk, and network metrics from every node in your cluster. You do this by running node exporter on each node, configuring Prometheus to scrape it, and building dashboards on the resulting time series. The mental model: each node is an independent target; Prometheus polls each one every 15 seconds (by default); you aggregate and alert on what you see.

The Architecture

Three pieces work together:

  1. Node exporter — a binary that runs as a DaemonSet on every node, exposes metrics on port 9100.
  2. Prometheus scrape config — tells Prometheus where to find each node exporter, how often to scrape, and what labels to attach.
  3. Service discovery — Kubernetes API tells Prometheus which nodes exist and their IPs; you don't maintain a static list.

Node exporter reads /proc and /sys on the host and converts kernel stats into Prometheus metrics. It doesn't need cluster permissions; it only needs access to the node filesystem. Prometheus then ingests these metrics every 15 seconds and stores them as time series.

Setup Checklist

Install node exporter as a DaemonSet:

Use the Prometheus community Helm chart or deploy a minimal DaemonSet manifest. The exporter must run on every node, including control-plane nodes if you want to monitor them.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

The hostNetwork: true and hostPID: true flags let the exporter read host metrics. Mount /proc and /sys read-only.

Create a Service to expose the exporter:

Node exporter listens on port 9100. Create a headless Service so Prometheus can discover all instances via Kubernetes DNS.

apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  clusterIP: None
  selector:
    app: node-exporter
  ports:
  - port: 9100
    targetPort: 9100

Configure Prometheus scrape job:

Add this to your Prometheus config. It uses the Kubernetes API to discover all nodes, then scrapes each one.

scrape_configs:
- job_name: 'kubernetes-nodes'
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    regex: '([^:]+)(?::\d+)?'
    replacement: '${1}:9100'
    target_label: __address__
  - source_labels: [__meta_kubernetes_node_name]
    target_label: node

The first relabel rule rewrites the address to point to port 9100 on the node. The second labels each series with the node name. This is critical: without the node label, you cannot distinguish which node a metric came from.

Labeling Strategy

Every metric from node exporter arrives with labels like instance (the IP:port it came from) and job (always "kubernetes-nodes"). You should add more:

  • node — the Kubernetes node name (added by relabel above).
  • cluster — your cluster name, if you run multiple clusters.
  • zone — the availability zone or region, pulled from node labels.

Add zone labeling via Kubernetes node labels:

relabel_configs:
- source_labels: [__meta_kubernetes_node_label_topology_kubernetes_io_zone]
  target_label: zone

This extracts the topology.kubernetes.io/zone label from the node object and attaches it to every metric. When you query, you can then group by zone without parsing instance IPs.

Key Metrics to Collect

Node exporter exports hundreds of metrics. Focus on these:

  • node_cpu_seconds_total — CPU time per core and mode (user, system, iowait). Use rate() over 5 minutes to get CPU utilization.
  • node_memory_MemAvailable_bytes — available RAM. Compare to node_memory_MemTotal_bytes to get memory pressure.
  • node_disk_io_reads_completed_total — disk read count. Use rate() to get reads per second.
  • node_disk_io_writes_completed_total — disk write count.
  • node_filesystem_avail_bytes — free space per filesystem. Alert if it drops below 10% of total.
  • node_network_receive_bytes_total — bytes received per interface. Use rate() for throughput.
  • node_network_transmit_bytes_total — bytes sent per interface.
  • node_load1 — 1-minute load average. Useful for context, but don't alert on it alone.

All of these are gauges or counters. Use rate() on counters to get rates; use gauges directly or with avg_over_time() for trends.

When This Breaks

Node exporter crashes or stops responding. Prometheus marks the target as down. You lose metrics for that node. Set up an alert: up{job="kubernetes-nodes"} == 0. This fires when any node exporter is unreachable for 2 minutes.

Node exporter runs out of memory. It reads /proc and /sys on every scrape. On a node with thousands of processes or network connections, this can use 100+ MB. If memory is tight, increase the DaemonSet resource request or disable unused collectors with --collector.disable-defaults and --collector.enable=<name>.

Metric cardinality explosion. Node exporter exports metrics for every network interface, disk, and mounted filesystem. A node with 20 network interfaces and 50 mounted filesystems generates 1000+ series. Multiply by 100 nodes and you have 100k series. If your Prometheus instance has limited memory, this hurts. Use metric relabeling to drop unwanted series:

metric_relabel_configs:
- source_labels: [__name__]
  regex: 'node_network_.*'
  action: drop

This drops all network metrics. Be intentional: keep what you alert on or dashboard, drop the rest.

Control-plane nodes are not monitored. By default, DaemonSets don't run on control-plane nodes (they have a taint). Add a toleration to the node exporter DaemonSet:

tolerations:
- key: node-role.kubernetes.io/control-plane
  operator: Exists
  effect: NoSchedule

Scrape interval is too high or too low. The default 15 seconds is fine for most clusters. If you scrape every 5 seconds, you generate 3x more data and stress Prometheus. If you scrape every 60 seconds, you miss short spikes. Stick with 15 seconds unless you have a specific reason to change it.

Trade-offs

Node exporter vs. kubelet metrics. The kubelet exposes node metrics directly on port 10250. You can scrape those instead of running node exporter. But kubelet metrics are less complete and less stable across Kubernetes versions. Node exporter is the standard; use it.

Scrape interval vs. cardinality. Faster scrapes mean more data points, more storage, more CPU in Prometheus. Slower scrapes mean you miss transient events. 15 seconds is the sweet spot for most on-call teams. If you run WordPress alongside your infrastructure tooling, the comparison on wpcompass.io on Redis vs Memcached for WordPress object cache is a useful reference for thinking through similar storage and latency trade-offs.

Host vs. container metrics. Node exporter gives you host-level metrics: CPU, memory, disk. Container metrics (from kubelet or cAdvisor) give you per-pod metrics. You need both. Node exporter catches host problems; container metrics catch runaway workloads.

One-line Takeaway

Run node exporter as a DaemonSet, configure Prometheus to discover nodes via the Kubernetes API, label each metric with the node name and zone, and drop metrics you don't use to control cardinality.