Research · First-party data
We pulled thirty-one days of raw access logs off a B2B site we run, to answer a question that should have been boring: how many people actually read this thing? It took three attempts, because the first two answers were wrong.
Nothing unusual about the site — a few dozen pages, a blog, a contact form. Here is what the logs said once we stopped fooling ourselves, and why the naive version of this analysis overstates your human audience by roughly seven times.
Of the traffic that can be identified with any confidence:
AI systems fetched the site roughly nine times more often than people did. One agent on its own made more requests in the month than every human visitor combined — on a site nobody would describe as famous.
Our first pass produced a flattering human number. It was nonsense, for three reasons that apply to most measurement setups too.
Nearly a quarter of all requests were for /wp-admin, /.env, /.git/config and similar — paths this site has never had, on a site that has never run WordPress. Most arrived with an ordinary Chrome user-agent. Count requests instead of real URLs and a quarter of your "traffic" is somebody hunting for your credentials file.
Tens of thousands of requests sent no user-agent string at all. Browsers do not do that. Your analytics tool never sees this traffic, which is precisely the reason to go and read the server logs yourself.
Most of what survived the first two filters came from a handful of cloud addresses running thousands of requests each behind a copied browser user-agent. A crawler that claims to be Chrome is still a crawler.
Apply all three and the human figure drops by about seven-eighths. That is not a rounding error. It is the difference between believing you have an audience and knowing you have one.
This was the genuinely surprising finding. AI crawling is not spread evenly across a site — it is concentrated almost entirely on the home page. On ours, AI agents made tens of thousands of requests to the front page and fewer than two hundred to every article combined.
More than a dozen agents that hit the home page thousands of times each have never fetched a single article. They arrive, take the summary of who you are, and leave. Whatever depth you have written is, to most of them, invisible.
That probe traffic is not only a counting nuisance. It is a continuous, automated census of your attack surface. Every hour, something is checking whether you left an environment file lying around, an exposed Git directory, or an admin panel where the scanners expect to find one.
On this site every one of those requests returned a 404, because none of those paths exist — and that is the only reason this paragraph is a footnote rather than an incident report. The scanning is constant and entirely indiscriminate. Being small is not a defence, because nothing in that traffic knew or cared what the site was.
Ten minutes in the log file will tell you more about who reads you than a year of dashboards, because JavaScript analytics can only count visitors that run JavaScript — and most of your traffic does not.
Filter to pages that actually exist in your sitemap before drawing any conclusion at all. Everything else is probes, noise and false comfort.
It is the page the crawlers actually take. If the first thing on it is a slogan rather than what you do, that is what gets summarised back to your buyers.
It is the file answer engines read most literally, so anything wrong in it gets repeated verbatim. Ours had drifted to a phone number that was not ours, which is exactly what hand-maintained files do.
A 404 on /.env is fine. A 200 is an incident. It is a five-minute check and almost nobody runs it.
This is a single site over thirty-one days, not an industry study, and we would not pretend otherwise. But the ratio is lopsided enough that even a much milder version of it is worth knowing about — and unlike most published statistics, you can check this one against your own logs before lunch.
Next step
A 30-minute call, then a scope document with what is in, what is out and what it costs. No obligation, and no charge for the conversation.
