Can you stop AI companies from training on your website? What robots.txt can do now
Can you stop AI companies from training on your website?
Yes, in part. Website owners can ask the major AI companies not to train on their content by adding rules to robots.txt, the file that tells automated crawlers what they may do on a site. On Oct. 30, 2026, OpenAI, Google and Anthropic backed a shared signal that does it for several providers at once. OpenAI said in its announcement that it will honor the signal, and Anthropic says it has begun recognizing it. The system is voluntary, and each company is adopting it on its own timeline.
The longer answer
Until now, a site owner had to find each company's training crawler and write a separate rule for it, OpenAI noted in its Oct. 30 post. Those company-specific rules still work:
How to opt out of AI training by company, as of Nov. 6, 2026
| Company | Company-specific robots.txt name | Shared __usage_ai-training signal |
|---|---|---|
| OpenAI | GPTBot | Will honor it; aims to implement in the coming months |
| Google-Extended | Aims to support it in 2027 | |
| Anthropic | ClaudeBot | Began recognizing it Oct. 30, 2026, rolling out over time |
What is the new shared signal?
It is a robots.txt group with a special name, __usage_ai-training, defined in an Internet-Draft filed on Oct. 30, 2026, by Kevin Kelley of Anthropic. A site that writes the following expresses a preference against AI training on every page except its press kit:
`` User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ``
The draft adds no new commands to robots.txt. It applies the definition of AI training from the Internet Engineering Task Force's (IETF) AI Preferences working group: "Using an asset to modify the learned parameters of a generative AI model."
Who honors it, and when?
- Anthropic says that "as of October 30, 2026," it has begun recognizing the signal, rolling it out "across all of our services over time," according to its help article on crawling. A rule that names ClaudeBot directly takes priority over the shared signal.
- OpenAI aims to implement the signal in the coming months and says its existing robots.txt controls will keep working.
- Google aims to support it "in 2027," according to its Search Central post. When a site opts out with the shared token and does not name Google-Extended, Google plans to treat it as if the site had disallowed Google-Extended.
What does an opt-out not cover?
A training opt-out is not a search opt-out. OpenAI's crawler documentation lists separate agents for ChatGPT search results (OAI-SearchBot) and for pages fetched when a user asks ChatGPT something (ChatGPT-User), and says that for the latter "robots.txt rules may not apply." Google says sites that allow Google-Extended can appear in responses and receive clicks through grounding in Gemini Apps.
What we don't know
The draft is an individual Internet-Draft, "not endorsed by the IETF," and has no formal standing yet. It says the preferences that apply are those in the robots.txt file in effect when the content was retrieved, so a new opt-out does not reach content collected before it. And because the signal is voluntary, it binds only the companies that choose to honor it; the draft also allows private agreements to override it.
FAQ
How do I block GPTBot?
Add a robots.txt group for the user agent GPTBot with Disallow rules. OpenAI says disallowing GPTBot signals that your content should not be used to train its generative AI foundation models.
What is __usage_ai-training?
A shared robots.txt name proposed in an Oct. 30, 2026, Internet-Draft. A Disallow rule under it expresses a preference against AI training to every provider that honors it.
Does blocking AI training remove my site from ChatGPT or Google results?
Not by itself. OpenAI and Google use separate controls for search, such as OAI-SearchBot, and Google says sites that allow Google-Extended can be shown through grounding in Gemini Apps.
Does opting out now remove my content from models already trained?
No. Under the draft, the preferences that apply are those in effect when the content was retrieved.
Sources
- OpenAI, Making it easier for websites to express AI training preferences (Oct. 30, 2026) · archived copy
- IETF, Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00 (Oct. 30, 2026) · archived copy
- Google Search Central Blog, Supporting a shared AI training control for websites (Oct. 30, 2026) · archived copy
- Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler? · archived copy
- OpenAI, Overview of OpenAI crawlers · archived copy
Cite this
Can you stop AI companies from training on your website? What robots.txt can do now, New Mind Museum, November 6, 2026. https://newmindmuseum.com/tools/2026/11/stop-ai-companies-training-on-your-website/
New Mind Museum. (2026, November 6). Can you stop AI companies from training on your website? What robots.txt can do now. https://newmindmuseum.com/tools/2026/11/stop-ai-companies-training-on-your-website/
“Can you stop AI companies from training on your website? What robots.txt can do now.” New Mind Museum, 6 Nov. 2026, https://newmindmuseum.com/tools/2026/11/stop-ai-companies-training-on-your-website/.
@misc{new-mind-museum-2026-can-you-stop-ai-companies-from-training-,
author = {{New Mind Museum}},
title = {Can you stop AI companies from training on your website? What robots.txt can do now},
year = {2026},
url = {https://newmindmuseum.com/tools/2026/11/stop-ai-companies-training-on-your-website/}
}Free to quote and cite with a link to this page.
Keep reading
Sources
- OpenAI, Making it easier for websites to express AI training preferences (Oct. 30, 2026)
- IETF, Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00 (Oct. 30, 2026)
- Google Search Central Blog, Supporting a shared AI training control for websites (Oct. 30, 2026)
- Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- OpenAI, Overview of OpenAI crawlers