How to Set Up Log Aggregation with Loki

Log aggregation with Loki centralizes logs from distributed systems without indexing every field—a deliberate trade-off that keeps resource costs low while keeping query latency predictable.

Unlike Elasticsearch (which indexes every field), Loki indexes only labels—metadata you attach to log streams. The logs themselves stay compressed. This means Loki is cheap to run at scale, but you pay for that efficiency with a different mental model: you must think about what you'll query before you ship logs.

The Loki Architecture Model

Three components matter:

1. Promtail (or another agent): Scrapes logs from disk or journald, adds labels, ships to Loki.
2. Loki: Receives logs, compresses them, stores them in object storage (S3, GCS, or local filesystem), and keeps an index of labels.
3. Grafana: Queries Loki using LogQL, displays results.

Think of Loki as a write-once, read-many log store. You label streams at ingest time. Queries filter by label, then scan the compressed logs inside each matching stream.

Setup: The Runbook

Step 1: Deploy Loki

Start with the Loki binary or Helm chart. A minimal single-binary config for testing:

auth_enabled: false
ingester:
  chunk_idle_period: 3m
  max_chunk_age: 1h
  max_streams_per_user: 10000
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
server:
  http_listen_port: 3100

This stores everything locally. For production, replace filesystem with S3 or GCS. Start Loki: loki -config.file=loki-config.yaml.

Step 2: Configure Promtail

Promtail runs on each host and ships logs to Loki. A basic config:

clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: prod-01
    pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            message: message
            level: level
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - labels:
          level:
          job:

This scrapes /var/log/syslog, parses JSON, extracts a timestamp, and adds labels. The labels stage promotes fields into queryable metadata.

Gotcha: Every unique label combination creates a new stream. Too many cardinality (e.g., adding request IDs as labels) will explode your index and memory. Limit labels to low-cardinality metadata: job, host, service, env, level. Put high-cardinality data in the log line itself.

Step 3: Query with LogQL

LogQL is Loki's query language. Start simple:

{job="syslog", host="prod-01"}

This returns all logs from that stream. Add filters:

{job="syslog"} |= "error"

The |= operator matches the literal string "error" in the log line. Use !=, |~ (regex), !~ (inverse regex) to refine.

For aggregation, use metric queries:

rate({job="syslog"} |= "error" [5m])

This counts error logs per second over the last 5 minutes. Pipe into Grafana dashboards for alerting.

When This Breaks

High cardinality labels: If you add a field with thousands of unique values (user ID, request ID, trace ID) as a label, Loki's index will grow unbounded. The fix: keep that data in the log line and use | json to extract it at query time. You'll scan more logs, but you won't blow up the index.

Label mismatch: Promtail's config defines which labels it sends. If you query for a label that doesn't exist in your Promtail config, you get no results—and it's silent. Always verify labels are in the pipeline. Use {job=""} in Grafana to discover what labels exist.

Timestamp parsing: Loki requires a valid timestamp. If your logs lack one or the format doesn't match the pipeline config, Loki will reject them or assign the current time. Check Promtail logs for parse errors: docker logs promtail | grep -i error.

Retention and disk space: By default, Loki keeps logs forever. In production, set retention_enabled: true and retention_period: 720h (30 days) in limits_config. Without this, your storage will fill.

Query performance on large time ranges: Loki scans all matching streams. A 30-day query across 100 streams is slow. Use label filters to narrow the scope before querying. Avoid queries like {} |= "pattern" (no label filter)—this scans every stream.

Trade-Offs

Cheap to store, expensive to query: Loki compresses logs, so storage is 10× cheaper than Elasticsearch. But queries scan compressed data, so complex filters are slower. Use Loki for operational logs (syslog, access logs, application errors). Use Elasticsearch for high-cardinality event data (clickstreams, user behavior). If you're also evaluating how styling choices affect your frontend observability stack, the CSS in JavaScript frameworks debate is worth a read for context on how tooling trade-offs compound.

Label discipline required: You must decide what to label before logs arrive. Changing labels later requires re-ingesting logs. Elasticsearch lets you index any field retroactively.

No full-text search: Loki doesn't index the log body. You can only search by label and then filter the line text. If you need to search across all logs for a rare string, Elasticsearch is faster.

One-Line Takeaway

Loki is a low-cost log store that trades query flexibility for storage efficiency—use labels to define what you'll search, keep cardinality low, and accept that you're scanning, not indexing.