syslog: headers, priorities and kernel events

Two header formats

RFC 3164 (BSD syslog)

The older, informational format, still what most files on disk look like:

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

Because the day can be one or two characters with padding, awk field numbers stay the same ($1 month, $2 day, $3 time, $4 host, $5 tag) but fixed character positions do not line up for days 1–9 unless you account for the padding. The missing year matters when a file spans New Year.

RFC 5424

The current standard adds a version, a full timestamp and structured data:

<165>1 2026-10-18T10:02:11.003Z web-1 app 4242 ID47 [exampleSDID@32473 iut="3"] request timed out

Facility and severity

PRI packs two numbers: PRI = facility × 8 + severity. So facility is int(PRI / 8) and severity is PRI % 8. <0> is kern.emerg; <165> is local4.notice.

Kernel and system events worth finding

The OOM killer writes two lines that matter: which task triggered it and which process it killed. From the kernel source (6.x):

Oct 18 10:02:11 web-1 kernel: java invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Oct 18 10:02:11 web-1 kernel: Out of memory: Killed process 4242 (java) total-vm:8123456kB, anon-rss:6012344kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:12345kB oom_score_adj:0
Oct 18 10:02:12 web-1 systemd[1]: app.service: Main process exited, code=killed, status=9/KILL

• <task> invoked oom-killer: — the task whose memory allocation failed. Not necessarily the one killed.
• Out of memory: Killed process <pid> (<name>) — the victim, with its virtual size and resident memory (anon-rss, file-rss, shmem-rss) at that moment. When the limit was a cgroup, the text begins with "Memory cgroup out of memory" instead.
• The next lines are usually the service manager noticing: code=killed, status=9/KILL from systemd is the same event seen from the service side.

Other lines worth a timeline: segmentation faults (segfault at), block I/O errors (I/O error, dev sdb), file system errors, and NIC link changes. Their exact wording depends on the kernel version and driver, so search with a short, case-insensitive pattern rather than a full sentence.

Shell one-liners for syslog files

Run against a sample /var/log/messages in RFC 3164 style (no PRI) and a small RFC 5424 file. On systemd hosts, journalctl -k gives kernel messages and journalctl -o short prints lines in the same layout, so the same pipes work.

• Which programs write the most lines (pid stripped).

  awk '{print $5}' messages | sed -E 's/\[[0-9]+\]//; s/:$//' | sort | uniq -c | sort -rn | head

  Copy

• Lines per host, when several servers log to one file.

  awk '{print $4}' messages | sort | uniq -c | sort -rn

  Copy

• How many processes the OOM killer killed.

  grep -cE '(Out of memory|Memory cgroup out of memory): Killed process' messages

  Copy

• Which process names were killed.

  grep -oE 'Killed process [0-9]+ \([^)]+\)' messages | awk '{print $4}' | tr -d '()' | sort | uniq -c | sort -rn

  Copy

• Which tasks triggered the OOM killer.

  grep -oE '[^ ]+ invoked oom-killer' messages | sort | uniq -c

  Copy

• Which programs crashed with a segfault.

  grep -oE '[^ ]+\[[0-9]+\]: segfault at' messages | cut -d[ -f1 | sort | uniq -c | sort -rn

  Copy

• How many lines mention an I/O error.

  grep -ciE 'i/o error' messages

  Copy

• Lines per minute — bursts stand out.

  cut -c1-12 messages | uniq -c

  Copy

• Lines per host per hour.

  awk '{split($3, t, ":"); print $4, t[1] ":00"}' messages | sort | uniq -c

  Copy

• Which programs write lines with error, fail or timed out.

  grep -iE 'error|fail|timed out' messages | awk '{print $5}' | sed -E 's/\[[0-9]+\]//; s/:$//' | sort | uniq -c | sort -rn

  Copy

• RFC 5424: decode PRI into facility and severity.

  grep -oE '^<[0-9]+>' syslog5424.log | tr -d '<>' | awk '{print "facility", int($1/8), "severity", $1%8}' | sort | uniq -c

  Copy

• RFC 5424: lines per host and app name.

  awk '{print $3, $4}' syslog5424.log | sort | uniq -c | sort -rn

  Copy