WARC Files: Archiving Web Pages as Evidence With Provenance

01 /What a WARC file is

A WARC file (Web ARChive) is a sequence of records, each wrapping one HTTP request, one HTTP response, or a note about a capture. The format is ISO 28500; the text most tools follow is the IIPC WARC 1.1 specification. Every record starts with the line WARC/1.1, named headers, a blank line, then content whose length is fixed by Content-Length. Records are concatenated, usually gzip-compressed one at a time, so each can be read alone.

The format grew out of the Internet Archive's crawler and was first published by ISO around 2009. For an operator the point is simple: a WARC file preserves the exchange itself rather than a rendering of it. The evidence-grade network records lane treats it as the file-level counterpart of an access log.

02 /Record types and the headers that matter

The specification defines eight record types; four carry the evidential weight. warcinfo opens the file and states who made it: software, operator, hostname, date. request holds the exact HTTP request sent. response holds the exact response received, status line and headers included. revisit notes that a URI was fetched again and the payload matched an earlier capture by digest, so the bytes are not stored twice. The other four cover notes, non-HTTP content, derived formats and split records.

Four headers are mandatory: WARC-Record-ID (in practice a urn:uuid), Content-Length, WARC-Date (UTC, W3C ISO 8601 form) and WARC-Type. WARC-Target-URI names the resource fetched; WARC-IP-Address records the address contacted. Two optional headers do the integrity work: WARC-Block-Digest covers the whole block, WARC-Payload-Digest the HTTP body alone. The specification recommends no algorithm; wget writes sha1: plus a Base32 value, weak against deliberate collision but sound against accidental change.

03 /Why a WARC file is stronger evidence than a screenshot

A screenshot proves that pixels were on a screen at some moment. It carries no request, no response headers, no server timestamp, nothing a reviewer can recompute. Save As writes the DOM after scripts ran, rewrites links and drops headers. A PDF print has the same gaps. A WARC response record keeps the Date, Last-Modified, ETag and Cache-Control headers the origin sent, the status code and the body as transferred. The paired request record shows which Host and Cookie headers went out, so a reader can judge whether the answer was personalised.

As with digest headers on an HTTP response, proof attaches to bytes rather than appearances.

04 /Making a WARC file

GNU wget has written WARC since version 1.14, around 2012. The manual lists the options: --warc-file=file names the output (wget appends .warc.gz), --warc-header=string adds a line to the warcinfo record, --warc-cdx writes a CDX index, --warc-max-size splits large captures, and --no-warc-digests turns digests off, which an evidential capture must never set. wget also stores its own log as a resource record; wget2 keeps the same options. The warcio project supplies a Python library and a capture_http context manager that wraps any requests call. For pages assembled by JavaScript, only a browser-based writer that records every fetch the page makes captures what a person saw.

$ wget --warc-file=example-net-20260924 --warc-header="operator: noc@example.net" \
       --warc-cdx --page-requisites --no-verbose https://www.example.net/status/
2026-09-24 14:02:11 URL:https://www.example.net/status/ [4812/4812] -> "www.example.net/status/index.html" [1]
$ zcat example-net-20260924.warc.gz | grep -a -m 10 -E '^(WARC/|WARC-)'
WARC/1.0
WARC-Type: warcinfo
WARC-Date: 2026-09-24T14:02:11Z
WARC-Record-ID: <urn:uuid:2f1c8a4e-9b3d-4c7e-8a51-0d6e3b2f9a10>
WARC/1.0
WARC-Type: response
WARC-Record-ID: <urn:uuid:7a0d5e12-33c4-4f8b-9e2a-b41c6d0f8e77>
WARC-Target-URI: https://www.example.net/status/
WARC-Date: 2026-09-24T14:02:11Z
WARC-IP-Address: 203.0.113.42

wget writes WARC/1.0, the ISO 28500:2009 dialect, which every 1.1 reader also handles. --page-requisites fetches the images, stylesheets and scripts the page references, each as its own record pair.

05 /Verifying a WARC file

Verification means recomputing what the writer wrote. The warcio documentation describes warcio check, which recalculates every payload and block digest and exits 1 on any mismatch; -v prints a line per record. warcio index lists the records as JSON lines. Check three things: the warcinfo record names the software, operator and host you expect; every WARC-Date falls inside the capture window; the digests pass. That proves internal consistency. Authenticity depends on the custody record around it.

06 /Worked example: capture, verify and preserve one page

An operator needs a record of a vendor's support-lifecycle page before a site migration removes it. The page, https://www.example.net/support/lifecycle/, is static HTML with two stylesheets and one image. The capture runs from a jump host with an NTP-synchronised clock.

1. 01

   Prepare the host

   Confirm the clock with chronyc tracking or ntpq -p: synchronised, offset under a second. Confirm wget --version is 1.14 or later.

2. 02

   Capture with digests and page requisites

   Run wget --warc-file=lifecycle-20260924 --warc-header="operator: J. Ortiz" --warc-cdx --page-requisites https://www.example.net/support/lifecycle/. Leave digests on; send no cookies.

3. 03

   Check the records

   Run warcio check -v lifecycle-20260924.warc.gz; every record must pass with exit status 0. Run warcio index and read the warcinfo line, the four target URIs and the dates.

4. 04

   Hash the file and the index

   Run sha256sum lifecycle-20260924.warc.gz lifecycle-20260924.cdx > lifecycle-20260924.sha256. Per-record SHA-1 guards against corruption; the file-level SHA-256 is what you compare years later.

5. 05

   Store read-only with a custody note

   Copy the three files to write-once or immutable storage, set them read-only, and write a note alongside: who ran the capture, from which host and address, when, on whose request, and the SHA-256 values. Sign it if the team uses signing keys.

6. 06

   Ask a third party to witness it

   Submit the same URL to Save Page Now. The Internet Archive's help page states it saves the page with images and CSS but without outlinks, and returns a permanent URL. Record it in the note.

07 /What a site operator uses this for

Three uses recur. The first is archiving vendor documentation before it disappears. The page on keeping copies of discontinued vendor manuals exists because so much Foundry material vanished after the 2008 acquisition in the Foundry Networks corporate timeline; a WARC capture is the form those copies should take. The second is capturing a page before a takedown or dispute. The third is recording what your own edge served: a scheduled fetch of key pages from outside the network proves what a visitor received and shows whether serving cached originals to crawler traffic matched the origin at that hour. Retention follows hash-chained retention for edge logs: hash on close, store apart, log every handoff.

One paragraph on CDX. A CDX index has one line per record: canonicalised URI key, timestamp, original URI, MIME type, status, digest, byte offset and WARC filename. wget's --warc-cdx writes one; warcio and replay tools read it. It lets a reader jump to one record among thousands, and its digest column can itself be hashed and kept with the note.

08 /Pitfalls: scripted pages, cookies, clocks and missing parts

• JavaScript-rendered pages. wget stores the HTML the server sent. If content is assembled by script after load, the response record holds a shell. Use a browser-based writer and say so in the note.
• Cookies and personalisation. A response fetched with a session cookie may be a page only that user sees. Capture anonymously where possible; where not, the request record keeps the cookies and the note explains their effect.
• Missing subresources. Without --page-requisites the archive holds the document and nothing it references, so replay shows a broken page. Check the CDX for the count you expect.
• An unsynchronised clock. WARC-Date comes from the capturing host's clock. If the host is minutes out, every record is, and the only cross-check is the origin's Date header.

A WARC is the archive-side counterpart of two operator records here: the syslog record of what the edge did and the corroboration of captures, flows and logs during an incident. Where those prove what your own devices saw, a WARC with digests proves what a server on the other side of the wire said.

A WARC file is the cheapest form of web evidence a stranger can check: request, response, time, address and digest in one container public archives have used for over a decade. Capture with digests on, verify before filing, hash the file, store it read-only with a note saying who and when, and let a third-party archive corroborate it.