What is Googlebot?
Googlebot is Google’s primary bot that is used to crawl web pages.
Generally speaking there are actually two different crawlers, Googlebot Desktop and Googlebot Smartphone.
Google Desktop
The Google Desktop crawler uses the Googlebot user agent token.
The following is are examples of full Google Desktop user agent strings, as per the Google documentation:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Googlebot/2.1 (+http://www.google.com/bot.html)Google Smartphone
The Google Smartphone crawler also uses the Googlebot user agent token.
The following is an example of a full Google Smartphone user agent string:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)HTTP/2
Since November 2020 Google may crawl websites using HTTP/2. It is possible to opt of this by responding to HTTP/2 crawls with a 421 HTTP status code.