robots.txt
robots file / disallow / blocking crawlers / robots exclusion protocol
A robots.txt file tells crawlers which URLs they may fetch on your site. Google is explicit that it is not a way to keep a page out of search: a disallowed URL can still be indexed if other sites link to it. The convention dates to 1994 and became RFC 9309 in 2022.
Google's description is one sentence and worth holding onto: a robots.txt file tells search engine crawlers which URLs the crawler can access on your site, and it is used mainly to avoid overloading your site with requests. Access and load. That is the job.
The convention is old and only recently written down properly. RFC 9309 specifies and extends the Robots Exclusion Protocol originally defined by Martijn Koster in 1994, describing it as a way for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Twenty-eight years of everyone following a convention before anyone standardized it.
Now the sentence that matters most, in Google's own words: it is not a mechanism for keeping a web page out of Google. To keep a page out, Google says to block indexing with noindex or password-protect the page. A page disallowed in robots.txt can still be indexed if linked to from other sites. Google will not crawl the blocked content, but it says it might still find and index a disallowed URL if it is linked from elsewhere on the web.
Which sets up a trap worth knowing before you touch either setting. For a noindex rule to be effective, Google says the page must not be blocked by robots.txt and has to be otherwise accessible to the crawler. If it is blocked, the crawler never sees the noindex, and the page can still appear in results. The two tools people reach for together actively cancel each other.
In practice
This is how a staging site ends up in search results. Someone blocks it in robots.txt, believing that hides it. Another site links to it, and the URL appears in results with no description, because Google found the address without being allowed to read the page. Adding a noindex at that point does nothing until the robots.txt block comes off.
Not the same as
- noindex
- That removes a page from results, and Google says it works regardless of who links to it. It requires the crawler to be able to read the page.
- Password protection
- Google names that as the other way to keep a page out. It is the only one of the three that stops a human too.
Why it matters to you
The file is trivial to edit and the consequences are not proportional to that. Blocking the wrong path can remove a section from search, and blocking a page to hide it can leave the URL visible while making the actual fix impossible. Google also notes that while most crawlers follow the rules, each one might interpret them differently, so this is a request rather than a lock.
What to ask or check
- 01What exactly does our robots.txt disallow right now, and who last changed it?
- 02Are we trying to reduce crawling, or to keep something out of results, which needs a different tool?
- 03Is anything marked noindex also blocked in robots.txt, which would stop the noindex working?
What people get wrong
That disallowing a page hides it from search. Google says the opposite in as many words: robots.txt is not a mechanism for keeping a web page out of Google, and a disallowed URL can still be indexed if linked from elsewhere.
Red flags
- A page blocked in robots.txt and marked noindex at the same time, which prevents the noindex from ever being read.
- A staging or private area protected only by a robots.txt disallow.
- Nobody able to say what the current file disallows or who last edited it.
Who owns it
Whoever can write to the site root. Often the developer rather than the marketer, which is why changes get made without the people watching search knowing.
Where you will see it
At /robots.txt on any site, and in every conversation about why a page is or is not showing up.