nginx access and error logs, field by field

The combined format

nginx predefines one format, combined. Its definition in ngx_http_log_module is:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

A real line, with two timing variables appended (see the next section):

192.0.2.15 - - [18/Oct/2026:10:02:09 +0000] "GET /api/items?page=3 HTTP/1.1" 200 790 "-" "okhttp/4.12.0" 2.912 0.140

With the default escaping, nginx writes ", \ and bytes below 32 or above 126 inside variables as \xXX, and logs a single dash when a variable has no value. That is why a request line normally has exactly three words and the status lands in $9. A garbage request line (a TLS handshake sent to a plain HTTP port, for example) breaks that assumption, so expect a few odd values in $9 on internet-facing servers.

$request_time vs $upstream_response_time

These two are the reason to extend the format. Add them at the end, so no combined field moves and parsers that read combined by position keep working:

log_format timed '$remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent" '
                 '$request_time $upstream_response_time';
access_log /var/log/nginx/access.log timed;

How to read the pair

• Both high and close to each other: the backend is slow. Look at the application, its database, its queue.
• $request_time much larger than $upstream_response_time: the time went on the client side — a slow upload, or a large response draining to a slow or distant client. The example line above is this case: 2.912 s total, 0.140 s upstream.
• $upstream_response_time is -: nginx never talked to an upstream for this request (static file, redirect, cached response, request rejected before proxying).
• Several values, like 0.120, 1.702: nginx tried more than one upstream server for the same request. Values are separated by commas, like the addresses in $upstream_addr; a colon separates values from different upstream groups after an internal redirect. Check $upstream_status to see what the first server answered.

Because the second value can contain ", ", it can turn into two awk fields. The one-liners below avoid that by splitting on the double quote and reading the text after the User-Agent.

The error log

The error log has its own fixed layout: date and time, level in brackets, pid#tid:, then *N (the connection number) and the message. For requests, nginx appends the context: client, server, request, upstream, host.

2026/10/18 10:02:03 [error] 2817#2817: *91512 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 198.51.100.7, server: shop.example.com, request: "POST /api/upload HTTP/1.1", upstream: "http://10.0.2.12:8080/api/upload", host: "shop.example.com"

Levels, from most to least severe: emerg, alert, crit, error, warn, notice, info, debug. The connection number *91512 is the same for every line about one connection, so grep '\*91512 ' error.log collects the whole story.

The text before "while" is the error, the text after it is what nginx was doing at that moment: connecting to upstream, sending request to upstream, reading response header from upstream, reading upstream, sending to client.

What the board shows for nginx

• Status codes, with 5xx and 4xx kept apart, and the share of server errors.
• Where errors happen: paths ranked by 5xx and 4xx count.
• Top paths, top client IPs, who is calling (User-Agent classes) and bytes by path.
• Latency distribution and slowest paths from $request_time, with p50/p90/p99.
• When traffic happens: a per-minute timeline you can drag to narrow every card.
• Scanner probes: requests for wp-login.php, .env, .git/ and similar.
• Look first: rules such as "82% of server errors hit /api/upload between 10:02 and 10:06", or "request time is 20× the upstream time — the slow side is the client link".

Shell one-liners for nginx logs

All of these were run against a sample combined log with the two timing fields appended. They assume the default combined layout; if your log_format differs, adjust the field numbers. For rotated logs, replace the file name with zcat -f access.log* piped into the command.

• How many requests are in the file?

  wc -l < access.log

  Copy

• Status code distribution, most frequent first.

  awk '{print $9}' access.log | sort | uniq -c | sort -rn

  Copy

• How many server errors?

  awk '$9 ~ /^5/' access.log | wc -l

  Copy

• Which paths return 5xx?

  awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head

  Copy

• Which clients send the most requests?

  cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head

  Copy

• How many distinct client addresses?

  cut -d' ' -f1 access.log | sort -u | wc -l

  Copy

• Requests per minute — the spike shows up as a big count (works because the log is in time order).

  cut -d[ -f2 access.log | cut -d: -f1-3 | uniq -c

  Copy

• Slowest requests by $request_time, with the request line.

  awk -F'"' '{split($NF, t, " "); print t[1], $2}' access.log | sort -rn | head

  Copy

• p99 of $request_time (nearest-rank).

  awk -F'"' '{split($NF, t, " "); print t[1]}' access.log | sort -n | awk '{a[NR]=$1} END {i=int(NR*0.99); if (i<NR*0.99) i++; print a[i]}'

  Copy

• Requests where nginx spent over a second more than the upstream did — slow clients or slow uploads.

  awk -F'"' '{n = split($NF, t, " "); u = 0; for (i = 2; i <= n; i++) u += t[i]; if (t[2] != "-" && t[1] - u > 1) printf "%s %.3f %s\n", t[1], u, $2}' access.log

  Copy

• Which paths send the most bytes?

  awk '{b[$7]+=$10} END {for (p in b) print b[p], p}' access.log | sort -rn | head

  Copy

• Most common User-Agent strings.

  awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head

  Copy

• How many requests are scanners probing for wp-login, .env or .git?

  grep -cE 'wp-login|\.env|\.git/' access.log

  Copy

• Mask the last two octets of client IPv4 addresses before sharing lines.

  sed -E 's/^([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/\1.x.x/' access.log | head -3

  Copy

• Error log: lines per level.

  awk '{print $3}' error.log | sort | uniq -c | sort -rn

  Copy

• Error log: which upstream failures, how often.

  grep -oE 'upstream timed out|connect\(\) failed|no live upstreams|upstream prematurely closed connection' error.log | sort | uniq -c | sort -rn

  Copy

• Error log: which upstream address the failures point at.

  grep -oE 'upstream: "[^"]+"' error.log | sort | uniq -c | sort -rn

  Copy