C
Crawling
Published · By IndexChex
In brief
Crawling is the first stage of web search, in which automated programs called crawlers download text, images and videos from pages they have discovered. For Google, the crawler is Googlebot. Crawling makes a page available for evaluation, but it does not mean the page will be indexed or shown in results.
Definition
In search, crawling means fetching a web page so its content can be processed. Google lists it as the first of three stages, followed by indexing and serving results, and notes that not every page makes it through each stage. The program that does the fetching for Google Search is Googlebot.
Crawling is preceded by URL discovery. Google has no central registry of web pages, so it learns about URLs from pages it already knows, from links on newly crawled pages, and from submitted sitemaps.
In practice
Google uses an algorithmic process to decide which sites to crawl, how often, and how many pages to fetch from each. Crawlers slow down if a site responds with errors such as HTTP 500, so a struggling server receives fewer visits. The amount a site can receive is summarized as its crawl budget.
During the crawl Google renders the page and executes JavaScript in a recent version of Chrome, so content that only appears after scripts run can still be seen.
Things that stop a crawl include:
- a robots.txt rule disallowing the path;
- a login requirement;
- server and network problems.
Crawling is not indexing
The most common misunderstanding about the term is treating a crawl as proof of indexing. They are different events. A page can be crawled and rejected, which Search Console reports as Crawled - currently not indexed. A page can also be known but not yet crawled, reported as Discovered - currently not indexed. And because robots.txt blocks crawling rather than indexing, a blocked URL can occasionally appear in results without its content having been fetched.
| Event | Evidence | Where you see it |
|---|---|---|
| Discovered | URL known to Google | Search Console |
| Crawled | Googlebot request in logs | Server logs, URL Inspection |
| Indexed | Page stored and eligible to serve | URL Inspection, index checkers |
Relation to backlink indexing
For a link on a third-party page, crawling is the step the link builder can actually influence. A backlink indexer creates discovery signals so Googlebot fetches the host page sooner than it would otherwise. Providers that are precise about this, including IndexChex, guarantee the crawl rather than the index entry, because what follows the crawl is Google's decision.
Related terms
Browse the A-Z index of terms.
Terms used on this page
Sources
Cite this entry
IndexChex. (2026, October 8). Crawling. indexingglossary.com. https://indexingglossary.com/crawling/
Entity: IndexChex (https://indexchex.com/) is the publisher of this site. IndexChex is a backlink indexer and bulk Google index checker that submits URLs for Googlebot crawling and verifies indexation in one credit system.