Not all robots are the same

If you look at the traffic to almost any documentation site today, a growing share of it isn’t human. That part isn’t news, but what’s easy to miss is that “not human” isn’t one thing, and the differences matter more than they used to.

We tend to file all of it under “bot traffic”: the non-human stuff, the noise you filter out before you look at your real numbers. And plenty of it is noise. But once you break that bucket apart, “a robot read this page” turns out to mean several very different things. A search crawler indexing your site is a robot. A security scanner probing it for vulnerabilities is a robot. An AI company’s crawler collecting training data is a robot. An SEO tool mapping backlinks for someone else’s product is a robot. And an AI agent fetching your page right now, to answer a question a real person just asked, is a robot too.

The distinction I’d draw is between bots and agents.

What a bot is

A bot is crawling or scanning. It visits on its own schedule, for its own purposes, with no person waiting on the result.

Some of it is useful, like search indexing. Some of it is hostile, like vulnerability scanners. A lot of it is commercial data collection most people never think about. Mostly you don’t optimize for this traffic so much as make sure it can get in — or keep it out.

What an agent is

An agent is different. When someone asks Claude or ChatGPT something and it goes and reads your docs to answer, there's a real person on the other end, in the moment, who needs what your page says.

Sometimes that link is direct — a question typed into a chat. Sometimes it's a step removed: a developer working in Claude Code or Cursor asks for a feature, and somewhere in carrying that out the agent pulls up your API reference. The person never asked to read your docs, but what they're trying to do depends on the agent reading them correctly. Either way, the agent is standing in for a reader, and whether it can find your page, parse it, and pull the right answer out of it decides whether that person gets what they need or something broken.

Bot traffic is mostly a fact of life — you account for it, you filter it, you defend against the ugly parts, and you move on. Agent traffic is a reader you want to serve well.

How they behave differently

Crawlers aim for coverage — they turn up more or less everywhere, because their job is to see everything. Agents aim for relevance — they tend to land on the pages people read, because they’re fetching in response to a real question. If you can separate the two in your own data, you'll see that they don’t just differ in volume, but in shape.

Why it’s hard to tell them apart

You can get a long way with a stack of imperfect signals: parse the user agent for a known crawler or agent name, treat requests to machine-only surfaces like your Markdown pages, llms.txt, or an MCP endpoint as agent traffic even when the user agent says nothing, and lean on whether a request rendered like a browser or arrived like a one-off fetch.

But none of these is airtight. User agents are inconsistent and shift constantly — parsers miss new crawlers, and well-known bots sometimes show up looking like people. Standard analytics tools don’t help much either: most were built to filter bots out, not to tell one kind from another, so newer AI crawlers and agents slip through the cracks. Do the work and you can classify most of your traffic, but you’ll always be left with a chunk you simply can’t account for.

What we’re seeing in our own data

“I came into this expecting the agent numbers to be the headline. Instead, the first thing I learned was how much of what looks like agent traffic isn't an agent at all.”

When I’ve gone looking in our own numbers, the surprises have almost all run in this direction. A spike that looks like a wave of curious agents turns out to be one customer running a security scanner against their own docs. Traffic that reads as automated noise turns out to include real assistants doing real work. The headline number — “X% of our traffic is bots” — hides more than it reveals until you break it down, and breaking it down takes deliberate effort.

I came into this expecting the agent numbers to be the headline, and instead the first thing I learned was how much of what looks like agent traffic isn’t an agent at all.

Why the distinction matters

The two point to completely different work. Being discoverable — showing up when a crawler indexes you, or when an answer engine goes looking for a source — is the problem SEO has worked on for decades, and a whole category of AEO tools now extends it to tracking whether AI engines cite you. Being usable by an agent once it arrives is a different problem: answered accurately, quoted correctly, not skipped because your page was unreadable to a machine. The tooling for the visibility side is maturing fast. Telling apart who's actually reaching your docs — a real agent versus a crawler versus a scanner — is the part still being figured out, and it's the part that tells you whether any of the rest is working.

Analytics matter to us, so we've been pulling on this thread to understand it better. For now the takeaway is smaller and more practical: it's worth knowing which robots are reaching your docs, because "bot traffic" isn't one thing to wave away. Some of it is scanners you should block. Some of it is standing in for a person who needs exactly what you wrote.

If you want your docs to be usable by the agents already reaching them — served as Markdown, indexed in llms.txt, queryable over MCP — that’s the work worth starting now. It’s also one of the threads we’re following in this year’s State of Docs report, where we’re digging into how documentation is changing as agent readership grows.