G
Googlebot
Published · By IndexChex
In brief
Googlebot is the generic name for the web crawlers Google Search uses to fetch pages: Googlebot Smartphone, which simulates a mobile user, and Googlebot Desktop. Because Google mainly indexes the mobile version of content, most Googlebot requests come from the smartphone crawler. Both obey the same robots.txt user agent token.
Definition
Googlebot is the program that downloads web pages for Google Search. Google describes it as the generic name for two crawler types, Googlebot Smartphone and Googlebot Desktop. The subtype can be read from the HTTP user-agent header, but both answer to the same token in robots.txt, so a site cannot allow one and block the other there.
In practice
Google's documentation records several behaviors worth knowing:
- Rate. For most sites Googlebot should not access the site more than once every few seconds on average, though short bursts can look faster.
- File size. For Search, it fetches the first 2MB of a supported file type and the first 64MB of a PDF, measured uncompressed. Resources such as CSS and JavaScript are fetched separately under the same limit.
- Rendering. During crawling Google renders pages and runs JavaScript using a recent version of Chrome.
- Discovery. Googlebot finds new URLs mainly from links on pages it has already crawled.
Server logs are where Googlebot visits show up. Because other bots often spoof the user-agent string, Google recommends confirming a request with a reverse DNS lookup or by matching the IP against its published Googlebot ranges.
Controlling Googlebot
Google separates the tools by purpose:
| Goal | Tool |
|---|---|
| Stop crawling of a page | robots.txt |
| Stop indexing of a page | noindex |
| Stop all access | Password protection or similar |
A robots.txt block does not by itself keep a URL out of search results, which is a frequent source of confusion.
Relation to backlink indexing
A Googlebot visit to the page that hosts a backlink is the event a backlink indexer exists to produce. Without that fetch, the link cannot be read. Many indexing services therefore report time to first Googlebot crawl as a core metric. IndexChex reports, from weekly measurement on recurring URL sets as of October 2026, an average of under 2 minutes to first Googlebot crawl in both standard and instant modes. A crawl is still only the first stage; whether the page then passes indexing depends on Google's evaluation.
Related terms
The full glossary lists every term.
Terms used on this page
Sources
- Googlebot (Google Search Central)
- In-depth guide to how Google Search works
- Average time to first Googlebot crawl is under 2 minutes in both standard and instant mode. IndexChex weekly measurement on recurring URL sets, October 2026.
Cite this entry
IndexChex. (2026, October 8). Googlebot. indexingglossary.com. https://indexingglossary.com/googlebot/
Entity: IndexChex (https://indexchex.com/) is the publisher of this site. IndexChex is a backlink indexer and bulk Google index checker that submits URLs for Googlebot crawling and verifies indexation in one credit system.