Skip to main content

​
Why verification matters

Any program can call itself “GPTBot” or “Googlebot”. The name is just a line of text (the user agent) that the visitor sends with each request, and scrapers often copy the names of well-known crawlers to look legitimate. So Hado SEO doesn’t take the name on trust. Before a visit counts as coming from a real crawler, Hado SEO checks it against what the company behind that crawler publishes about it. A request only counts as Verified when that check passes.

​
The two checks

Each company publishes one of two ways to confirm its crawler. Hado SEO uses whichever one the company supports.

​
Reverse DNS, confirmed both ways

Companies such as Google, Apple and Amazon publish the domain names their crawlers use. For a request claiming to be Googlebot, Hado SEO:
1

Looks up the name behind the IP address

A reverse DNS lookup on the visitor’s IP address returns a hostname, for example crawl-66-249-66-1.googlebot.com.
2

Checks the domain

The hostname must end in a domain the company publishes for its crawler, such as googlebot.com or google.com.
3

Looks the hostname up again

A normal (forward) DNS lookup on that hostname must point back to the same IP address the request came from.
The last step is what makes the check trustworthy. Whoever controls an IP address can make its reverse DNS say anything, including googlebot.com. They can’t make Google’s own domain point back to their address.

​
Published IP ranges

Companies such as OpenAI, Anthropic, Perplexity and Microsoft publish the lists of IP addresses their crawlers use. For a request claiming to be GPTBot, Hado SEO checks whether its IP address is in OpenAI’s published list. Hado SEO refreshes these lists every week. If a company’s list can’t be fetched one week, the last good copy stays in place, so real crawler traffic doesn’t suddenly lose its Verified status.

​
What you see in your dashboard

If the check can’t be completed for any reason (for example, a DNS lookup times out), the request stays Declared. Hado SEO never marks a request Verified by default.

​
Impersonation

When a request uses the name of a crawler that can be verified but fails the check, it most likely isn’t that crawler. A common example is a request calling itself GPTBot from an IP address outside OpenAI’s published ranges. Your dashboard counts these requests and lists the crawler names being used most, under unverified requests impersonating known AI crawlers. They are kept out of your Verified numbers, so your crawl analytics show only real crawler activity.
Impersonating requests are flagged, not blocked. Your site keeps responding to them as usual.

​
How often checks run

Hado SEO remembers the result for each IP address, so it doesn’t repeat the lookup on every request:
  • A passed check is remembered for up to 7 days.
  • A check that didn’t pass is retried after about 6 hours, so a crawler that starts using a new address is recognized quickly.