Syslog as Evidence: Provenance and Integrity of Network Logs

01 /What syslog is

Syslog is the message format and delivery convention that network devices and Unix-like servers use to report events. A link going down, a login failing: each becomes one syslog message sent to a collector. RFC 3164 (2001) describes the BSD syslog protocol as found in the field and calls itself informational. RFC 5424 (2009) replaced it with a specified layout: a version digit, an RFC 3339 timestamp with zone, hostname, application name, process ID, message ID, optional [structured data] elements and the free text message.

Both begin with PRI, one number: facility multiplied by eight plus severity. Facilities 0 to 23 name the subsystem; severities 0 to 7 run from Emergency to Debug, so <34> is facility 4 at severity 2. For evidence the timestamp difference matters most: a 5424 line says when in absolute terms, a 3164 line gives local time with no year and no zone.

02 /Transport: UDP 514, TCP and TLS on 6514

RFC 3164 describes syslog over UDP port 514 with a ceiling of 1024 bytes per packet; RFC 5426 specifies the same carriage for 5424 messages. UDP has no acknowledgement, no ordering and no sender authentication: a datagram claiming to come from 203.0.113.7 could have come from anywhere on the LAN, and drops under load are silent.

TCP transport, framed per RFC 6587, fixes loss and ordering, but the sender is still only an address. RFC 5425 defines syslog over TLS on port 6514: the stream is encrypted, the collector presents a certificate, and the sender can be required to present one. That turns "a message from an address" into "a message from a device holding this key".

03 /Provenance: which device, which clock, which path

Provenance is the record of where a line came from, made of three facts. The device: the hostname in the header plus, with TLS, the certificate presented. The clock: a timestamp is an assertion by the sender's clock, and a switch that has drifted or booted to a 1970 epoch produces lines that fit no other record. The path: every relay can drop, reorder or re-stamp a message, and some overwrite the hostname with their own.

NIST SP 800-92, the Guide to Computer Security Log Management, treats time synchronisation and a documented log architecture as baseline controls: know which systems log to where and who owns each tier. The same reasoning applies to HTTP records, covered in the access log evidence procedure for edge proxies.

04 /Integrity: can the line be altered after the fact

A log the administrator can open in an editor is a statement by that administrator, not a record. The cheap end is an append-only attribute on the storage host. The stronger end is a hash chain: at a fixed interval, hash the closed segment with the previous chain value and store the result where the collector cannot write. Any change to an earlier segment breaks every hash after it. Signing the daily chain head with a key the logging team does not hold makes it attributable too.

05 /What each transport and storage choice proves

06 /Building the syslog pipeline a reviewer can trust

Every source keeps time from the same NTP servers and sends over TLS where it can, or by UDP to a relay on its own subnet where it cannot. The collector writes hourly files that are never reopened once closed; a separate job hashes each into the chain.

# rsyslog collector, TLS listener on 6514
module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="x509/name")
global(DefaultNetstreamDriverCAFile="/etc/rsyslog/ca.pem")
input(type="imtcp" port="6514")
template(name="perhost" type="string" string="/var/log/remote/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log")
action(type="omfile" dynaFile="perhost" fileCreateMode="0440")

# switch side, generic IronWare-style syntax
switch(config)# ntp server 192.0.2.10
switch(config)# logging host 192.0.2.20
switch(config)# logging facility local6

# one RFC 5424 message as received
<182>1 2026-09-23T14:07:41.220Z sw-core-1.example.net switch 1201 LINK [origin ip="198.51.100.4"] Interface ethernet 1/3, state down

In the sample, <182> decodes to facility 22 (local6) at severity 6. The structured data element records the sender's own address, so a hostname rewrite upstream is detectable. The IronWare logging and clock commands give exact syntax for Foundry gear, which was UDP-only.

1. 01

   Synchronise every clock

   Point every device and the collector at the same NTP servers. Confirm offset is under a second before enabling remote logging.

2. 02

   Choose the transport per device

   Devices that support RFC 5425 send TLS with a client certificate to 6514. Others send UDP 514 to a relay on the same VLAN, which forwards over TLS. Never let raw UDP cross a routed boundary.

3. 03

   Write per host and per hour

   One file per source hostname per hour, never reopened once closed. Relays must preserve the original HOSTNAME.

4. 04

   Hash closed segments into a chain

   A scheduled job computes SHA-256 over each closed file plus the previous chain value and appends it to a chain on another host. Sign the daily head.

5. 05

   Define retention

   Write down a retention period per source class, following SP 800-92 on matching retention to need. Delete only by policy, after the chain entry is verified.

6. 06

   Document every export

   Cut incident extracts with a tool that reports SHA-256; record requester, range, reason and digest in an export ledger. That ledger is the chain of custody.

07 /Worked example: one incident on one switch

An access switch, sw-edge-14.example.net, is suspected of having a port moved between VLANs without a change ticket on 22 September. It logs by UDP only, to a relay that forwards over TLS. The investigator needs every line from 17:00 to 23:00 UTC in a form that survives a challenge.

First, show clock on the switch, captured to a timestamped file, shows NTP synchronised with a 40 millisecond offset. Second, six hourly files are located under /var/log/remote/sw-edge-14.example.net/. Third, each file's SHA-256 is recomputed and matches the chain. Fourth, the six files are concatenated into one export, its digest recorded and a ledger entry written. The export shows a vlan 30 change on ethernet 1/17 at 19:42:08Z by user admin over SSH from 198.51.100.77. Fifth, the switch's local buffer still holds the same line.

What this proves: at 19:42:08Z by a synced clock, that device emitted that line, it arrived over an authenticated last hop, and the stored copy is unchanged. It does not prove who was at the keyboard; correlating the SSH source with sFlow samples taken on the uplink is a separate exercise.

08 /Pitfalls: drift, loss, local-only logs and in-place edits

• Clock drift on the sender. A switch without NTP drifts minutes per month and boots to an arbitrary epoch. Sync first, log second, alarm on NTP loss.
• Silent UDP loss. Under a storm of link flaps datagrams vanish without trace, so gaps look like quiet periods. Keep any UDP hop to one subnet and watch relay receive counters.
• Local-only logging. The switch buffer is a ring: a busy device overwrites it in hours and a reboot clears it. It corroborates; it never replaces sending.
• Editing files in place. Opening a sealed segment to remove a leaked password breaks the chain. Redact in a copy, note it in the ledger, leave the original sealed.

Two neighbouring pages in the provenance section depend on this one: the NTP hierarchy that gives every log line a defensible timestamp comes first, because an unsynced clock spoils the record before it is written, and the audit trail that names who changed a configuration and when is built from these same syslog and accounting lines.

Syslog was designed to tell an operator that something happened. Making it tell a reviewer the same thing months later takes synced clocks, an authenticated path, storage nobody can quietly edit, and a record of every export. That is why the wider provenance lane on this site treats logging as an integrity problem.