🔍 Read the full analysis: OpenAI, Google, Microsoft And Anthropic Back One Opt-Out From AI Training. What It Covers, What It Doesn’t, How To Set It on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal asking that website content not be used to train generative AI models. The signal is voluntary, does not block all crawling or AI use of webpages, and has different implementation timelines; Meta expressed support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said they will honor a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The proposed rule gives publishers one way to express that preference across participating companies, but it is voluntary, not an enforcement mechanism, and the companies plan to adopt it on different schedules.
The signal appears in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published October 30 by Kevin Kelley, an Anthropic standards lead. A site would add a rule group using User-Agent: __usage_ai-training and Disallow: /. It is intended to express a preference against using that site’s content for generative AI training.
The companies have not announced a common start date. Anthropic says it has already started implementing support, while acknowledging that adding it across its services will take time. OpenAI says it aims to implement the signal in the coming months. Microsoft says it will honor it from February 1, 2027, as it crawls. Google says it aims to support the control in 2027; its crawler documentation specifies early 2027.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a common signal an important step. He did not say whether Meta will honor this particular rule. The source material reports no commitment from Apple, Amazon, xAI or Perplexity.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Several Company Timelines
A shared rule could reduce the work publishers face when expressing a preference to multiple AI companies. Until now, site operators have generally had to account for company-specific crawler names and controls, including GPTBot, Google-Extended and ClaudeBot. A common token could make the preference easier to publish and interpret across participating services.
But the practical effect depends on each company’s implementation and adherence. The rule is a request, not a technical lock: it does not guarantee that a company will comply, and it does not itself create a right to payment. Microsoft has said that neither choice on its controls guarantees inclusion, traffic or payment. Publishers will also need to keep existing company-specific rules in place while providers move to the shared signal.
The distinction matters to anyone who publishes material online. A site may want to permit ordinary search indexing while opting out of training, or may have a different view about content used to generate AI answers. The shared signal addresses only one of those choices, so it should not be read as a general opt-out from all AI-related use.
Products Worth Considering
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
What the Robots Rule Covers
The proposal builds on work by the AI Preferences working group at the Internet Engineering Task Force, which develops internet standards. The document is an individual Internet-Draft, not an adopted standard. At a working-group meeting on October 21, one of the chairs said the group had not declared consensus on anything yet.
Under the companies’ announced plans, the new rule concerns training use. It does not stop a crawler from accessing a page, undo use of content in models that have already been trained, or govern AI answers assembled from live web content. Existing crawler rules are not immediately being replaced: OpenAI says its current controls will continue to apply, and Google says the new preference will be interpreted as a Google-Extended disallow if a site has not set a separate Google-Extended rule.
Microsoft has announced a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says its shared training signal and grounding preference will take effect February 1, 2027. Its existing NoArchive and NoCache tags will stop counting as training or grounding preferences after March 1, 2027. Microsoft says these preferences do not change eligibility for Bing search results; Google says Google-Extended does not affect Google Search.
““has already started implementing support for this draft””
— Anthropic
Compliance and Scope Remain Open
The draft has no formal standing in the IETF standards process at this stage, and the working group has not declared consensus. The companies’ public commitments also leave open how broadly and consistently the rule will be applied across their products and services. Anthropic says implementation has begun, but its crawler help page still described only per-crawler controls as of October 31.
It is also unclear whether Meta or other AI companies will honor the signal, and how publishers could respond if a company ignores it. The rule does not address live-web grounding except where a company offers a separate control, as Microsoft has said it will. It also does not establish compensation for content use or remove material from existing models.
Watch the 2027 Rollout
Publishers will need to track each provider’s implementation notices and decide whether to add the shared training rule alongside existing crawler-specific controls. The announced milestones include OpenAI’s target of the coming months, Google’s early-2027 aim, and Microsoft’s February 1, 2027 start date. Anthropic says support is underway but has not given a completion date.
For sites using Microsoft’s NoArchive or NoCache tags as training or grounding preferences, Microsoft’s stated March 1, 2027 cutoff makes a review of those settings relevant before then. The proposal’s status in the IETF process, additional company commitments and the details of each rollout remain developments to watch.
Key Questions
What does the shared AI-training opt-out do?
It is a proposed robots.txt preference asking participating companies not to use a website’s content to train generative AI models. It does not block crawling by itself.
Which companies say they will honor it?
OpenAI, Google, Microsoft and Anthropic have announced plans to support the signal. Meta expressed support for the idea but did not commit to honoring this rule.
Does the rule stop AI companies from using content in answers?
Not generally. The signal covers training, not answers generated using live web content. Microsoft has announced a separate AI Grounding preference for certain Microsoft products.
When will the signal take effect?
There is no single date. Anthropic says implementation has started, OpenAI is targeting the coming months, Google aims for early 2027, and Microsoft says it will honor the signal from February 1, 2027.
Is this an official internet standard?
No. The signal is described in an individual Internet-Draft, and the working group has not declared consensus. The rule is voluntary and does not provide a technical guarantee of compliance.
Source: ThorstenMeyerAI.com
