systemary.io

Sysadmin tutorials, Linux server hardening, monitoring & observability

OpenTelemetry Collector Configuration Guide

by Liam Foster
OpenTelemetry Collector Configuration Guide

OpenTelemetry Collector is a vendor-neutral agent that receives, processes, and exports telemetry data—metrics, traces, and logs. Configuration is YAML-based and declarative; the collector reads one file and runs exactly what you specify, nothing more. Get the config right and you have a flexible observability pipeline. Get it wrong and data either disappears silently or floods your backend.

The Configuration Model

The collector config has four main sections: receivers, processors, exporters, and service. Think of it as a DAG (directed acyclic graph) where data flows from receivers through pipelines to exporters.

Receivers → Processors → Exporters
   ↓           ↓            ↓
  OTLP       Batch        Jaeger
  Prometheus Sampling     Datadog
  Syslog     Attributes   Prometheus

The service section wires these together. A pipeline defines which receivers feed into which processors, which then feed into which exporters. You can have multiple pipelines running in parallel—one for traces, another for metrics, a third for logs.

Receivers: Where Data Enters

Receivers listen on ports or read from files. Common ones:

  • otlp: gRPC and HTTP endpoints for OpenTelemetry protocol (port 4317 gRPC, 4318 HTTP by default).
  • prometheus: Scrapes Prometheus targets. Requires a config block with scrape_configs.
  • syslog: Parses syslog messages (RFC 3164 or RFC 5424).
  • jaeger: Accepts Jaeger spans (gRPC, Thrift, HTTP).
  • fluentforward: Listens for Fluent Forward protocol.

Example receiver block:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'my-app'
          static_configs:
            - targets: ['localhost:8080']

Gotcha: If you don't specify an endpoint, the receiver uses a default (4317 for otlp gRPC). If that port is already bound, startup fails silently in some distributions. Always be explicit about ports.

Processors: Transform and Filter

Processors sit between receivers and exporters. They modify, drop, or batch telemetry. Run them in sequence; order matters.

  • batch: Groups telemetry into batches before export (reduces network calls). Essential for production.
  • memory_limiter: Prevents memory runaway; drops data if heap exceeds a threshold.
  • attributes: Adds, removes, or modifies attributes on spans, metrics, or logs.
  • sampling: Drops spans probabilistically or by trace ID (tail-based sampling available).
  • resource_detection: Adds resource attributes from environment (AWS, GCP, Kubernetes, etc.).
  • span: Renames spans, removes attributes, updates status.

Example processor block:

processors:
  batch:
    send_batch_size: 1024
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  attributes:
    actions:
      - key: environment
        value: production
        action: insert
  sampling:
    sampling_percentage: 10

Trade-off: The batch processor adds latency (up to timeout) to reduce resource use. For high-volume systems, increase send_batch_size to flush faster. For low-volume, increase timeout to batch more efficiently.

Exporters: Where Data Leaves

Exporters send telemetry to backends. Most run over gRPC or HTTP with configurable retry and timeout logic.

  • otlp: Sends to any OTLP-compatible backend (Jaeger, Tempo, Datadog, New Relic, etc.).
  • prometheus: Exposes metrics on an HTTP endpoint for Prometheus to scrape.
  • jaeger: Native Jaeger exporter (gRPC, Thrift, HTTP).
  • logging: Writes to stdout (debug only).
  • zipkin: Sends to Zipkin.
  • splunk: Sends to Splunk HEC.

Example exporter block:

exporters:
  otlp:
    endpoint: localhost:4317
    timeout: 30s
  prometheus:
    endpoint: 0.0.0.0:8889
  logging:
    loglevel: debug

Gotcha: The prometheus exporter exposes metrics; it doesn't scrape. Prometheus scrapes the collector's metrics endpoint. Don't confuse this with the prometheus receiver, which scrapes targets.

Service: Wiring Pipelines

The service section defines which receivers, processors, and exporters form a pipeline. Each pipeline is independent.

service:
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [syslog]
      processors: [attributes]
      exporters: [otlp]

This config:

  • Accepts traces from OTLP and Jaeger, processes them, exports to OTLP.
  • Scrapes Prometheus targets, batches metrics, exposes them.
  • Receives syslog, adds attributes, exports as OTLP.

Critical: A receiver, processor, or exporter must be referenced in a pipeline to run. Unused blocks are ignored. If your receiver isn't in the service section, it won't listen.

Minimal Production Example

Here's a realistic setup for a Kubernetes cluster:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
  resource_detection:
    detectors: [k8s, env, system]
  batch:
    send_batch_size: 512
    timeout: 5s

exporters:
  otlp:
    endpoint: tempo.observability.svc.cluster.local:4317
    timeout: 30s
  logging:
    loglevel: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection, batch]
      exporters: [otlp, logging]

This setup:

  • Listens on standard OTLP ports.
  • Detects Kubernetes resource attributes automatically.
  • Limits memory to 1 GB; batches spans for efficiency.
  • Exports to Tempo and logs (for debugging).

When This Breaks

Data loss with no error: Exporters have retry logic and a queue. If the backend is unreachable, the queue fills, and old telemetry is dropped. Monitor exporter queue depth and backend latency.

Memory runaway: The memory_limiter processor is a safety valve, not a solution. If you're hitting the limit frequently, either increase it (if you have headroom) or reduce sample rate or batch size.

Duplicate attributes: If you use resource_detection and also add attributes manually, you may end up with conflicting or redundant fields. Check your backend to confirm.

Processor ordering: Sampling before resource_detection means you drop spans before enriching them. Reverse the order if you want all spans enriched equally.

Port conflicts: Receivers bind to ports at startup. If a port is in use, the collector fails. Use netstat or ss to verify ports are free before deploying. If you're running the collector alongside other services on a shared host, the comparison on tinjauhost.biz.id on shared hosting versus VPS trade-offs is worth a read before deciding where to deploy.

One-Line Takeaway

OpenTelemetry Collector config is a declarative pipeline: receivers → processors → exporters, wired in the service section. Get the YAML right, and you have a reusable observability backbone; get it wrong, and telemetry vanishes or your backend drowns in noise. For teams also thinking about how their instrumentation choices affect end-user experience, this guide on ecommerce platform conversion rates by tier illustrates how backend performance and observability gaps can surface in real business metrics.