OpenAI and Google Will Honor a Shared Robots.txt Signal for AI Training Opt-Outs
A common robots.txt token built on IETF work would let websites tell multiple AI providers 'don't train on this' in one step. Neither company has switched it on yet.
Published
Last updated / Fact-checked on
OpenAI said on October 30, 2026, that it will honor a common signal in robots.txt that lets websites opt out of having their content used for foundational generative AI training. Google announced the same day that it plans to support the shared control. The signal would let publishers state one preference to every participating AI provider instead of writing separate rules for each crawler.
Image: Illustrative photo of a white humanoid robot. Alex Knight via Unsplash / Wikimedia Commons — CC0 1.0
What was announced
Robots.txt is the file websites use to give instructions to automated crawlers. Until now, OpenAI said, website operators had to identify individual crawlers and maintain separate rules for each provider. A shared signal across providers reduces that work.
OpenAI called the effort voluntary. It said it aims to implement the signal "in the coming months" and will share implementation details when it does. The control is not live yet.
The signal draws on the work of the Internet Engineering Task Force's (IETF) AI Preferences (AIPREF) working group. OpenAI said it is joining other working group participants in announcing the signal, and it encouraged other AI providers to honor it.
Google's Search Central blog described how the mechanism will work. Website owners will use a shared "policy token" in robots.txt to set broad preferences across participating providers. Google said it aims to support the control, along with other AI providers, in 2027.
| OpenAI | ||
|---|---|---|
| Timing | "In the coming months" | Aims for 2027 |
| Existing control | GPTBot | Google-Extended (offered since 2023) |
| How the shared opt-out is read | Implementation details to come | Treated as disallowing Google-Extended, unless Google-Extended is set explicitly |
| Existing controls kept? | Yes | Yes |
Existing controls stay in place
Both companies said the shared signal adds to their current controls rather than replacing them.
According to OpenAI's crawler documentation, disallowing GPTBot indicates that a site's content should not be used to train OpenAI's generative AI foundation models. OAI-SearchBot is separate and is used to show websites in ChatGPT's search features. The settings are independent, so a publisher can block training while staying visible in ChatGPT search.
Google said website owners can set a broad choice with the industry signal and still tailor instructions for individual developers through controls such as Google-Extended. It noted that sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.
Google also said the proposal reflects feedback from publishers, civil society groups and regulators. It described the proposal as consistent with the EU's Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a primary way for publishers to express machine-readable preferences.
Why it matters
For publishers, the appeal is simpler upkeep. Managing separate rules for training, search and other crawlers takes ongoing technical attention, and one shared token could cover several AI companies at once.
There are limits. The signal is voluntary, and its reach depends on how many providers adopt it. A robots.txt line states a preference; it does not technically block a crawler. Neither announcement says the signal will affect content already collected for training.
What website owners can do now:
- Keep current rules in place. GPTBot and Google-Extended entries continue to work.
- Separate training from search. Blocking a training crawler doesn't have to remove a site from AI search features.
- Watch for implementation notes from OpenAI and Google before relying on the shared token.
OpenAI called the signal "a practical step toward an open web" where website owners have clearer ways to communicate how their content is used. More coverage is in our Artificial Intelligence hub.
How we reported this
We read OpenAI's announcement and crawler documentation and Google's Search Central post in full. We removed leftover favicon links and repetitive analysis, and added Google's description of how the shared token interacts with Google-Extended.
Published and fact-checked October 30, 2026.
Sources
- OpenAI: Making it easier for websites to express AI training preferences, Oct 30, 2026.
- Google Search Central Blog: Supporting a shared AI training control for websites, Oct 30, 2026.
- OpenAI Developers: Overview of OpenAI Crawlers, accessed Oct 30, 2026.
DailyMech used AI tools to help research and draft this article; it was fact-checked by our editors.