Googlebot’s list of IP addresses to verify the crawler’s scans

Put us to the test
Put us to the test!
Analyze your site
Select the database

It’s a topic that can worry site owners, webmasters and SEOs: how to distinguish reliable crawler passes from accesses performed by spammers or other troublemakers claiming to be Googlebots? In addition to the two classic methods of verifying crawls, for some time now Google has decided to officially publish the list of IP addresses used by Googlebots for its views, thus allowing interested parties to have a definitive way to verify that crawler accesses to the site match those claimed.

The list of Googlebot IP addresses

To reveal this coveted list is the new page added to Google’s official documentation, in which it reads that you can now “check whether a web crawler that accesses the server is really a Google crawler, like Googlebot”, so as to prevent any spammers or other disturbers from accessing the site claiming to be Googlebot and doing damage with their scans.

Thanks to this intervention, in fact, it is now possible to “identify Googlebot based on the IP address, matching the IP address of the crawler to the list of Googlebot IP addresses”.

What is the list of both Googlebot and Google IP addresses

We can view the complete list of Googlebot IP addresses in a JSON file published at this link. It should be noted, however, that Google may update this list, and so it would be useful to check the file periodically.

We can also find out all the other Google crawlers by checking whether the IP address found on the server corresponds to one of those present in the complete list of Google IP addresses. Again, the list of addresses may vary over time.

How to verify Google crawlers

So far, the only way to do this was to reverse DNS the addresses that had accessed the server, but now there are two roads available, as the Google guide, again, says.

  • Manual method: suitable for one-off searches, based on the use of command line tools and sufficient for most use cases.
  • Automatic method: useful for large-scale searches, uses an automatic solution to compare the IP address of a crawler with the list of published Googlebot IP addresses.

The manual process

To perform a manual verification we must then use the command line tools, performing a reverse DNS search on the IP address of access from our logs, using the host command.

The second step is to check that the domain name is googlebot.com or google.com, followed by a Forward DNS lookup on the domain name recovered in the first step, using the host command on the recovered domain name. Finally, we must find that the one obtained is the same as the original access IP address from our logs.

Come verificare le scansioni di Google