systemary.io

Sysadmin tutorials, Linux server hardening, monitoring & observability

Nginx Reverse Proxy Setup Checklist

by Liam Foster
Nginx Reverse Proxy Setup Checklist

Nginx reverse proxy setup is a series of discrete decisions, each with a failure mode. This checklist captures the model: upstream definition → request routing → response handling → observability → failure recovery. Follow it in order; each step unlocks the next.

The Model

A reverse proxy sits between clients and backend servers. Nginx receives a request, forwards it to one or more upstreams, collects the response, and sends it back. The proxy must decide:

  1. Which upstream gets this request (load balancing algorithm)?
  2. How to modify the request before forwarding (headers, path rewriting)?
  3. How to handle timeouts, retries, and failures?
  4. What to log and expose for monitoring?

Each decision is independent; you can implement them incrementally.

Upstream Definition Checklist

Define upstream blocks first. This is your backend pool.

upstream backend {
  server 10.0.1.10:8080 weight=1 max_fails=2 fail_timeout=10s;
  server 10.0.1.11:8080 weight=1 max_fails=2 fail_timeout=10s;
  keepalive 32;
}
  • List all backend servers by IP and port (not hostname; DNS lookup adds latency).
  • Set weight equal if backends are identical capacity; adjust if one is larger.
  • Set max_fails and fail_timeout based on your SLA. Two failures in 10 seconds is conservative; adjust upward if backends are flaky.
  • Enable keepalive to reuse TCP connections to upstreams (default 0 means no reuse). Use 32 per worker as a baseline.
  • If using health checks, add check directive (requires nginx_http_upstream_module or use passive health checks via fail_timeout).

Gotcha: Nginx does not automatically remove dead backends. A backend marked down by max_fails will recover after fail_timeout expires. If a backend is permanently dead, remove it from the config and reload.

Request Routing Checklist

Route requests to the upstream and set load balancing algorithm.

server {
  listen 80;
  server_name example.com;

  location / {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}
  • Use proxy_pass http://upstream_name; to route to the upstream block.
  • Set proxy_http_version 1.1 to enable connection reuse (1.0 does not support keep-alive).
  • Forward the original Host header; backends often need it for virtual hosting.
  • Forward X-Real-IP (client IP) and X-Forwarded-For (chain of IPs) so backends can log the true client.
  • Forward X-Forwarded-Proto so backends know if the original request was HTTP or HTTPS.
  • Choose a load balancing algorithm: round_robin (default), least_conn, ip_hash, or random. Use least_conn if requests have variable processing time.

Gotcha: If you do not forward the Host header, backends may reject the request or serve the wrong virtual host.

Timeout and Retry Checklist

Set timeouts to prevent hanging requests and silent failures.

proxy_connect_timeout 5s;
proxy_send_timeout 10s;
proxy_read_timeout 30s;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
proxy_next_upstream_tries 2;
  • proxy_connect_timeout: time to establish a TCP connection to the upstream. Default 60s is too long; use 5s.
  • proxy_send_timeout: time to send the full request to the upstream. Default 60s; use 10s for typical APIs.
  • proxy_read_timeout: time to receive the full response from the upstream. Default 60s; adjust based on your longest-running endpoint.
  • Set proxy_next_upstream to list conditions that trigger a retry on another upstream (e.g., error, timeout, http_5xx).
  • Set proxy_next_upstream_tries to the number of upstreams to try before failing. Use 2 for redundancy; higher values mask problems.

Gotcha: proxy_next_upstream retries are transparent to the client; if the first upstream times out and the second succeeds, the client sees only success. This can hide a flaky backend. Monitor retry rates.

SSL/TLS Checklist

Encrypt traffic to clients and optionally to upstreams.

server {
  listen 443 ssl http2;
  server_name example.com;
  ssl_certificate /etc/nginx/certs/example.com.crt;
  ssl_certificate_key /etc/nginx/certs/example.com.key;
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!aNULL:!MD5;
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 10m;
}
  • Use listen 443 ssl http2; to enable HTTPS and HTTP/2 (faster than HTTP/1.1).
  • Point ssl_certificate and ssl_certificate_key to your cert files (use absolute paths).
  • Restrict ssl_protocols to TLSv1.2 and TLSv1.3; disable older versions.
  • Use a strong cipher suite; the example above is Mozilla's "Intermediate" recommendation.
  • Enable ssl_session_cache to reuse session keys and reduce handshake overhead.
  • Redirect HTTP to HTTPS: add a separate server block on port 80 with return 301 https://$host$request_uri;.

Gotcha: Certificate paths must be readable by the nginx worker process (usually www-data or nginx user). Check file permissions.

Observability Checklist

Log requests and expose metrics for monitoring.

access_log /var/log/nginx/access.log combined buffer=32k flush=5s;
error_log /var/log/nginx/error.log warn;

upstream backend {
  server 10.0.1.10:8080;
  ...
}

map $upstream_addr $upstream_name {
  default "unknown";
  "10.0.1.10:8080" "backend-1";
  "10.0.1.11:8080" "backend-2";
}
  • Use a custom log format that includes $upstream_addr, $upstream_response_time, and $status to track which backend handled each request.
  • Set buffer and flush on access_log to batch writes and reduce I/O.
  • Use a map block to convert upstream IP addresses to human-readable names in logs.
  • Monitor error_log for connection failures, timeouts, and SSL errors.
  • Export nginx metrics (requests per second, response times, upstream health) via a metrics exporter (e.g., prometheus-nginx-exporter).

Gotcha: Default logging is synchronous and blocks the worker thread. Buffering mitigates this but adds latency. For high-traffic proxies, consider shipping logs to a remote syslog server.

Health Check Checklist

Detect and remove failing backends automatically.

  • Use passive health checks (built-in): max_fails and fail_timeout mark a backend down after N failures in M seconds.
  • Use active health checks (requires third-party module or external script): nginx periodically sends a probe request to each upstream and removes it if it fails.
  • If using passive checks, set fail_timeout to a value that matches your SLA (e.g., 10s means a backend is unavailable for 10s after two failures).
  • If using active checks, probe every 5–10 seconds; too frequent probes add overhead, too infrequent probes delay detection.
  • Log health check results (failures, recoveries) to a separate file for debugging.

Gotcha: Passive health checks only work if requests are flowing. A backend can fail silently if no requests reach it. Use active health checks in production.

Reload and Test Checklist

Validate config and apply changes without downtime.

  • Run nginx -t to check syntax.
  • Run nginx -T to dump the full config (useful for debugging).
  • Run nginx -s reload to reload config without dropping connections.
  • Verify upstream health: curl -s http://localhost:8080/health on each backend.
  • Test the proxy: curl -v http://example.com/ and check X-Real-IP and Host headers in the backend log.
  • Monitor error_log for connection errors during reload.

Gotcha: reload does not restart workers; old workers continue serving existing connections until they close. New connections go to new workers. If you change worker_processes, you must do a full restart.

When This Breaks

502 Bad Gateway: The upstream is unreachable or rejected the request. Check backend logs and error_log for "upstream timed out" or "connection refused".

Slow responses: Check $upstream_response_time in access logs. If high, the backend is slow, not the proxy. If low but client sees slow responses, check network latency or client-side issues.

Uneven load distribution: Verify weight values and load balancing algorithm. If using ip_hash, clients from the same IP always hit the same backend (useful for session affinity, but can cause imbalance). If you're also evaluating infrastructure costs at this stage, the comparison on techjournaler.com offers a useful perspective on whether cloud-hosted backends justify the overhead.

SSL certificate errors: Check cert expiry (openssl x509 -in cert.crt -noout -dates) and that the cert matches the hostname.

One-Line Takeaway

Reverse proxy setup is upstream definition, request routing, timeout handling, and observability—implement them in that order, test each step, and monitor retry rates to catch flaky backends early. If your backends run as devbox self-hosted development tools, the same checklist applies with minimal adjustment.