Which AI crawlers read my pages, and are they real?

Why the snippet is not enough

Crawlers fetch raw HTML and never execute a script, so a browser-side tracker does not see them at all. The server log does, one line per request, with the user agent and the address.

Step 1: send the log

Upload on the settings page or post it daily with the API token. nginx and Apache combined format, plain or gzipped, whole files; lines older than the newest already imported are skipped.

curl -sS -X POST https://agenttracking.co/api/logs/example.com \
  -H "Authorization: Bearer wmt_your_token" \
  -H "Content-Type: text/plain" --data-binary @/var/log/nginx/access.log

Step 2: verification against published ranges

OpenAI, Anthropic's partners, Perplexity, Microsoft, Google and Apple publish the address ranges their crawlers use. Every line claiming one of them is checked; a match counts, a miss is listed as unverified. The ranges are refreshed nightly.

Unverified does not mean malicious, but it does mean the request did not come from where the vendor says it would, which is worth knowing before you trust a robots.txt rule to hold.

Step 3: bursts

One agent, several pages, a few seconds: that is an assistant answering a question about you, not a crawl. The Agents view counts bursts per agent and lists the recent ones with their pages, so you can see which of your pages an assistant pulled together to answer.

In short