Which AI crawlers read my pages, and are they real?
Some crawler user agents can be checked against fresh published IP ranges in origin logs. Google-Extended is a robots policy token, not a crawler identity. Missing or unsupported verification remains explicit.
By Gorden Wübbe · Updated 2026-09-25
Why the snippet is not enough
Many crawlers fetch HTML without executing browser scripts. An origin access log can record their requests with the claimed user agent and address when that information is available.
Step 1: send the log
Upload an append-only log snapshot in Settings or use the collector with stable source, generation and record positions. nginx and Apache combined format are supported, plain or gzipped. Repeated positions are skipped; rotated or reordered logs need source metadata.
curl -sS -X POST https://agenttracking.co/api/logs/example.com \ -H "Authorization: Bearer wmt_your_token" \ -H "Content-Type: text/plain" --data-binary @/var/log/nginx/access.log
Step 2: verification against published ranges
For supported operators and crawler identities, the importer compares origin addresses with current published ranges. Some identities cannot be verified this way; they remain unknown instead of becoming verified by user-agent text alone.
Unverified does not necessarily mean malicious. Check log provenance, list freshness and the operator's documentation before drawing a conclusion.
Step 3: bursts
A burst is at least three distinct relevant paths with short gaps inside one imported batch. It is a request pattern, not evidence of a user question or assistant intent.
In short
Which crawlers are on the list?
The list is public in the repository and versioned; the docs page prints the current version. Missing one? Open a pull request with the vendor's documentation.
Is the address stored?
No. It is used while the upload is processed, to group one agent's fetches and to check the range, and discarded when the request ends. The log file is not kept.