Crawling
crawler / Googlebot / spider / bot
Crawling is a search engine fetching your pages to see what is on them. MDN describes a crawler as a program that systematically browses the web to collect data. Google is blunt about the limit: it does not guarantee that it will crawl, index, or serve your page, even when everything is done correctly.
Start with the neutral definition, because the search engines describe their own version. MDN puts it plainly: a web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages, and typically search engines use crawlers to build indexes. Fetching pages and building an index are two different jobs, and crawling is only the first.
Google describes three stages, and the framing matters more than the names: Google Search works in three stages, and not all pages make it through each stage. Crawling is the first of them. The sentence printed above all of it is the one worth remembering, because it applies to every stage: Google does not guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.
Discovery is stranger than most people expect. There is no central registry of all web pages, so Google must constantly look for new and updated pages, finding them from pages it already knows, from links on those pages, and from sitemaps you submit. Then comes the part that surprises owners: Googlebot does not crawl all the pages it discovered. Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site, and it tries not to crawl the site too fast to avoid overloading it.
Two practical details sit underneath. Google renders the page during the crawl and runs any JavaScript it finds using a recent version of Chrome, which is why a page built entirely in the browser can be fetched and still look empty. And crawling depends on whether the crawlers can access the site at all: some pages may be disallowed for crawling by the site owner, and other pages may not be accessible without logging in.
In practice
When a page is not showing up, the first question is which stage it failed at, because the fixes are unrelated. Not crawled is an access or discovery problem. Crawled and not indexed is a judgment the engine made about the page. Treating them as one problem is how a site ends up with a new sitemap it did not need.
Not the same as
- Indexing
- Crawling fetches the page. Indexing is the separate stage where the engine analyzes it and decides whether to store it. A page can be crawled every week and never indexed.
- Ranking
- Both earlier stages happen before any question of position arises, and Google says a page has to be eligible to be shown at all before that question exists.
Why it matters to you
Most arguments about visibility are really arguments about which stage failed, and that is knowable rather than mysterious. It also sets a realistic expectation for anyone selling you work: nothing done to a site obliges an engine to fetch it, store it, or show it, and Google says so in a sentence it prints above its own instructions.
What to ask or check
- 01Which stage is this page stuck at, and what is the evidence?
- 02Is anything on our side preventing access, such as a login or a blocked path?
- 03Does this page render its content in the browser, and does it survive being fetched without one?
What people get wrong
That a discovered page will be fetched. Google states that Googlebot does not crawl all the pages it discovered, and that an algorithmic process decides which sites to crawl, how often, and how many pages to fetch from each.
Red flags
- A promise that a page will be crawled or indexed by a given date.
- A fix applied to the wrong stage, such as a sitemap for a page that is crawled and rejected.
- Content that exists only after JavaScript runs, with no check that it survives a plain fetch.
Who owns it
The search engine decides. What you own is access: whether the page can be reached, and whether its content is there when it is fetched.
Where you will see it
In server logs, in Search Console's crawl reports, and in the gap between publishing something and seeing it anywhere.