What First-Party Trade-Site Logs Can and Cannot Tell Us About AI Crawlers

Research question

Which observations are direct, and which are proxies?

Request-level exports can count response statuses within the exported windows. A user-agent-family parser can identify strings that resemble named crawlers or automation, but the strings are self-reported and do not authenticate a requester. A separate crawl map stores cumulative counts by URL and its latest label; it cannot reconstruct each request. The distinction matters when discussing AI discovery or potential paid access.

Sample design

Four selected 1,000-row windows

The audited first-party inventory contains two TurfBurn windows from July 8 and July 10, one ResidentialFilm window from July 11, and one CoolDeck window from August 2, 2026. Each export contains exactly 1,000 request rows. The windows were selected from available exports, not randomly sampled, and do not represent a full day, the full site network, or ongoing traffic.

That is 3,845 successful HTTP 200 responses in this sample and 155 other status outcomes. An HTTP 200 alone says nothing about who requested the page, whether it was used in an AI answer, or whether any payment was accepted.

Classifier comparison

A 1,015-row classification gap

The existing application detector marked 2,865 of the 4,000 requests as bots. A separate family parser, applied to unverified request headers, labeled 3,880 as crawler or automation requests. The difference is 1,015 detector misses relative to that parser: 988 labeled GoogleOther, 26 Chrome Lighthouse, and one labeled python-httpx.

These are family labels, not authenticated identities. GoogleOther is not by itself proof of AI model crawling; Lighthouse and python-httpx describe automation or a client library. The comparison identifies a rule-coverage issue worth investigating. It cannot establish 1,015 verified AI crawlers, human-to-bot conversion, or a change in demand.

Different evidence layer

Twenty-two crawl snapshots across five sites

The inventory also includes 22 cumulative crawl-map snapshots across BuildingTint, ResidentialFilm, CoolDeck, TurfBurn, and GarageCoatings. They span July 6 through September 15, 2026. To avoid adding the same mutable entry across exports, the audit grouped each URL lifetime by site, URL, and first-seen time and retained its largest observed hit count. That produces an observed lower bound of 697,685 hits across 575,101 URL lifecycles.

This lower bound is not an exact request total. The map can evict older URLs, and each retained URL combines hits from different visitors while keeping only the latest family label. The snapshots have no response-status or response-size field. They therefore cannot validate the four request windows, identify a crawler for every hit, measure transferred data, or support revenue calculations.

Method and disclosure

How the aggregates were calculated

1. Count request rows and HTTP statuses within each of the four fixed-size traffic exports, then sum by site.
2. Compare the application bot flag with a separate family parser over the same 4,000 rows; group disagreements by family label.
3. For crawl-map exports, deduplicate repeated cumulative entries by site, URL, and first-seen time, retaining the maximum observed hits per lifetime.
4. Keep the request-window findings and crawl-map lower bound separate. Do not extrapolate a selected window to all traffic.

The underlying request-level files are withheld because they contain client-level metadata. This page publishes aggregates and calculation rules only. The analysis is first-party work, not an independent measurement of the broader web.

Limits and implications

What this sample cannot prove

The sample cannot authenticate bot operators from header strings; show that any page was cited by an AI assistant; establish how many unique people or companies searched; or demonstrate payment, settlement, bandwidth, or revenue. Its selected windows are not representative of all dates, trades, sites, or responses. The data also do not support a causal claim that a product change, publication, or search intervention caused the observed traffic.

A better future measurement design would separate verified edge identity, application classification, response outcome, and commercial settlement in distinct records with clear collection windows. Public summaries should carry sample sizes, exclusions, and uncertainty alongside any trend.