Skip to content
Adrythm
Book 20 minutes
AI search

GPTBot

GPTBot user agent / OpenAI crawler / block GPTBot / GPTBot robots.txt

In short

GPTBot is the web crawler OpenAI uses to collect pages that may be used to train its AI models. You allow or block it with a line in your robots.txt file. Blocking it is a separate decision from appearing in ChatGPT search, which a different crawler handles.

OpenAI runs several crawlers and documents each one by the name that shows up in your server logs. GPTBot is the one tied to training. OpenAI says it crawls content that may be used in training its generative AI foundation models, and that disallowing it signals a site's content should stay out of that training.

Search runs on a different bot. OpenAI uses OAI-SearchBot to surface websites in ChatGPT's search features, and it describes the two settings as independent. Its own example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot. A site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show up as a navigational link.

A third name, ChatGPT-User, visits a page when a person asks ChatGPT to open it. OpenAI says robots.txt rules may not apply to those visits, because a user started them.

Two limits apply to all of this. OpenAI says a robots.txt change can take about 24 hours to reach its search systems. And robots.txt is a request. RFC 9309, the standard that defines the file, describes rules that crawlers are requested to honor and says they are not a form of access authorization.

In practice

A law firm wants its articles to appear when someone asks ChatGPT about estate planning, and wants them kept out of model training. Its robots.txt gets two groups. The first reads User-agent: GPTBot followed by Disallow: /. The second reads User-agent: OAI-SearchBot followed by Allow: /. Search visibility stays on and training is declined, in two lines each.

Not the same as

OAI-SearchBot
OpenAI's search crawler. It decides whether your pages can appear in ChatGPT search answers, and OpenAI treats its setting independently of GPTBot.
Google-Extended
Google's robots.txt token for its own AI models. A rule written for GPTBot has no effect on Google.

Why it matters to you

A blanket block on every AI crawler can take a business out of ChatGPT search along with training. Allowing everything hands pages to training by default. Each crawler is a separate line, so each decision can be made on purpose and written down.

What to ask or check

  1. 01What does this site's robots.txt currently say for GPTBot and for OAI-SearchBot?
  2. 02Has the decision about AI training been made separately from the decision about ChatGPT search?
  3. 03Do the server logs show GPTBot requests coming from the IP ranges OpenAI publishes?

What people get wrong

That blocking GPTBot hides a site from ChatGPT. GPTBot governs training. Whether pages can appear in ChatGPT search answers is set by OAI-SearchBot, a separate line in the same file.

Keep reading

robots.txt

A robots.txt file tells crawlers which URLs they may fetch on your site. Google is explicit that it is not a way to keep a page out of search: a disallowed URL can still be indexed if other sites link to it. The convention dates to 1994 and became RFC 9309 in 2022.

Google-Extended

Google-Extended is a robots.txt token that controls whether content Google crawls from your site may be used to train Gemini models and ground Gemini apps. Google states it does not affect your inclusion in Search or your ranking. It is a training control, not a way to stay out of search results.

Crawling

Crawling is a search engine fetching your pages to see what is on them. MDN describes a crawler as a program that systematically browses the web to collect data. Google is blunt about the limit: it does not guarantee that it will crawl, index, or serve your page, even when everything is done correctly.

AI Overviews

An AI Overview is the summary Google sometimes places above the results, with links to sources. Google says they are only shown when its systems judge them additive to classic Search, so they often do not trigger. There is no special file or markup that gets you into them.

AI performance report

An AI performance report shows how often AI search features show or cite your site. Google's version counts impressions in AI Overviews and AI Mode. Bing's counts citations and states its numbers do not indicate ranking, authority or placement. Neither publishes a position inside an AI answer.

AI content disclosure

AI content disclosure is labeling material as machine-made where a platform requires it. Google's product listings require AI-generated images to carry a specific IPTC metadata code, and IPTC defines three different codes: created with generative AI, edited with it, and edited by a person.

Questions people ask about this

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes