Answer Engine Referral Traffic: Measuring Visits From AI Answers

01 /Three streams, not one

An answer engine touches a site in up to three ways. It may have crawled the page weeks earlier for its index. It may fetch the page at question time to quote it. And, if the answer shows a citation and the reader clicks, a human visit arrives. The first two are covered by identifying AI crawler traffic; this page, in the network monitoring section, is about the third, which is the only one that brings a reader.

02 /What the Referer header carries

The browser sends a Referer header (the historical spelling) on the request that follows a link. Its content is governed by the page's referrer policy. The default in modern browsers is strict-origin-when-cross-origin: a cross-site click sends only the origin, such as https://answers.example, with no path or query. So you can usually tell which service sent a visit, but not which answer or prompt. Some services set a stricter policy and send nothing; native apps often open links without any referrer at all.

03 /Building the report from access logs

1. 01

   Keep the Referer field

   Make sure the balancer or web server log format records Referer and the request path; many trimmed formats drop it.

2. 02

   Classify referring origins

   Maintain a short table mapping referring hosts to answer-engine operators, reviewed monthly as services change domains.

3. 03

   Find first requests

   Take the first HTML request of each visit (no same-site referrer before it) as the landing page.

4. 04

   Pair with fetches

   For each landing page, count the engine's own fetches of it in the same week. A page fetched often but rarely visited is being quoted without clicks.

5. 05

   Publish weekly

   Report visits by operator, the top landing pages and the fetch-to-visit ratio, with the blind spots stated under the table.

04 /A worked example: AI referral traffic in one week of logs

The transcript below runs the report steps against one week of a combined-format access log. Hostnames are invented and the counts are synthetic; the shape of the output is what a real log produces. The Referer field is the fourth quoted string in the common combined format, which is why the first command splits on quotation marks.

$ awk -F'"' '{print $4}' access.log | awk -F/ '{print $3}' | sort | uniq -c | sort -rn | head -5
  4812 www.search.example
   911 answers.example
   407 assistant.example
    58 chat.example
    31 m.assistant.example
$ grep '"https://answers.example' access.log | awk '{print $7}' | grep -v '\.\(css\|js\|png\|svg\)' | sort | uniq -c | sort -rn | head -3
   233 /network-monitoring/sflow/
   140 /switching/vlans/
    97 /routing/vrrp-and-vsrp/
$ grep -c 'answers-fetcher/' access.log
1670
$ grep 'answers-fetcher/' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -3
  1120 /network-monitoring/sflow/
   310 /switching/vlans/
    61 /routing/vrrp-and-vsrp/

Note the fifth line of the first block: m.assistant.example is the same operator as assistant.example on a mobile hostname. The origin-to-operator table has to fold those together, and it has to fold http and https variants too, or one operator appears as three. The fetch count uses the operator's declared fetcher token; that token must be confirmed by reverse DNS, address range or signature before it is trusted, because anyone can send it.

The third row is the interesting one. A ratio below one means the engine is citing a copy it fetched earlier, perhaps weeks earlier, so a change to that page will take time to reach answers. A ratio far above one, as on the first row, means readers are satisfied by the quotation; if that page is meant to bring people to the site, the part being quoted is doing the whole job and the page may need something the quote cannot carry, such as a table or a worked configuration.

05 /Reading the numbers

A high fetch-to-visit ratio means the page answers the question well enough that readers do not need to click; that is a content decision, not a network fault. A sudden fall in referral visits with steady fetches often means the service changed its referrer policy or began opening links in an in-app browser. A rise in fetches with no visits and no citation may be a crawler misreporting itself, which the signed-request verification method can settle. Caching matters too: quoted pages are fetched repeatedly, and the caching approach for crawler-heavy pages keeps those fetches off the origin.

Agents that browse on a user's behalf are a separate case again: their visits carry no citation click at all and are covered in what AI agent sessions look like.

06 /Pitfalls in AI referral traffic reports

• Counting the engine's own fetch as a visit. A fetcher that sends a Referer, or a browser-like string, lands in the referral count. Split fetchers out first by the verification ladder, and only then count what is left as people.
• Prefetches. Some applications fetch cited links before the reader taps them. A landing request with no asset requests and no second page within a minute is a prefetch, not a visit; count it separately or drop it.
• Step changes that are not yours. A referral count that halves in one week while fetches hold steady usually means the operator changed its referrer policy, moved links into an in-app browser or changed hostname. Check the origin list before touching the site.
• Origin variants. Scheme, www, mobile and regional hostnames of one operator must map to one row. Review the mapping monthly.
• Sampled logs. Small operators vanish at one-in-ten sampling. If the balancer must sample, exclude requests that carry a cross-site Referer from the sample.
• Attribution by landing page alone. A page that only answer engines link to is a hint that a visit came from an answer, not proof; label such rows as inferred.

07 /Practical tips for the weekly report

• Keep thirteen months of the fields the report needs (timestamp, path, Referer, verified client class) so year-on-year comparison is possible; the retention and integrity rules in log retention that survives an audit cover the how.
• Record a second signal for each referral visit, such as a second page or a beacon after thirty seconds, and report the proportion. A referral that leaves in two seconds is a different thing from one that reads.
• Publish one canonical URL per page and keep old paths redirecting to it, so citations to three spellings of the same page land in one row.
• Print the operator-to-origin mapping, its review date and the blind-spot list under every table. A reader six months later must be able to tell what the numbers meant.
• When a row looks like people but behaves like a script (identical paths, even timing, one origin range), route it through separating scripted visits from readers before reporting it as referral traffic.