On 30 October 2026, the four biggest AI labs did something they have never done before: they agreed on a single, shared way for websites to say “do not train your models on my content.” For anyone running a blog, this is the most practical AI training opt-out development of the year — a two-line addition to a site’s robots.txt file, designed to work across OpenAI, Google, Microsoft and Anthropic at once. In short: one token, added once, understood by all four. But it comes with real limits: it does not stop crawling, does not cover AI-generated answers, and does not remove content already used for training. This article explains how the token works, what it does and does not do, when each company starts honouring it, and what publishers should do this week.
What the four AI labs just agreed
OpenAI, Google, Microsoft and Anthropic announced on 30 October 2026 that they will honour a single shared robots.txt policy token, so a website can express its AI training preference to every participating company in one place. OpenAI described it as a voluntary effort on its official blog the same day. Google’s Search Central blog, in a post by John Mueller, said the company intends to support the control in 2027. The technical specification is published as an Internet-Draft in the IETF datatracker, authored by Kevin Kelley of Anthropic.
It is an individual draft, not an endorsed IETF standard, dated 30 October 2026 and expiring 3 May 2027. The draft is publicly available in the IETF datatracker for anyone to read. For background on AI regulation elsewhere, see our coverage of Japan AI Regulation News and the Impact of Japan’s AI Policy Framework.
For publishers, the appeal is simple: one token understood by four companies replaces a patchwork of incompatible signals — provided each company follows through.
How the one-line AI training opt-out works
The mechanism is deliberately simple. To opt a whole site out, a publisher adds two lines to its robots.txt file:
User-Agent: __usage_ai-training
Disallow: /
The token is the user-agent name __usage_ai-training, and the directive Disallow: / expresses the preference not to use the site’s content for training. Publishers who want finer control can add an Allow: line for specific paths, such as a press kit.
The draft defines AI training as “using an asset to modify the learned parameters of a generative AI model” — a narrow definition that sets the boundary of what the token covers. To understand what today’s models are trained to do, see Google’s Gemini 4 Carbon: Everything We Know About the New AI Model.
The token is not a crawling directive: adding it does not change how the companies’ crawlers visit a site. A wildcard User-Agent: * group must not be read as matching this token, and a robots.txt file with no such group expresses no preference — silence means nothing.
What it does NOT do: four honest limits
1. It does not stop crawling. The token expresses a preference about training use only; crawlers will keep visiting pages exactly as before. To block crawling itself, the existing robots.txt crawler controls still apply and are still respected — OpenAI has confirmed this.
2. It does not cover AI-generated answers or grounding. Training a model and using content to answer a specific question are different things. Where the shared token opts a site out and Google-Extended is not explicitly specified, Google plans to treat the token as a Google-Extended disallow; sites that explicitly allow Google-Extended remain eligible to appear in responses and receive clicks through grounding in Gemini Apps. Answer-time uses sit outside the draft’s scope.
3. It does not remove content that is already collected or trained. It applies to future training only: it does not pull a site’s pages out of existing datasets or un-train models that have already processed them.
4. It is voluntary. It binds only companies that choose to honour it. Four of the largest have said they will, but nothing in the announcement makes the token a legal requirement.
When each company starts: the rollout timeline
The four companies have committed on different schedules, with different levels of direct confirmation:
| Company | Timeline | Attribution |
|---|---|---|
| OpenAI | Aims to implement “in the coming months” | Official company blog |
| Aims to support the control in 2027; will map an unspecified shared-token opt-out to a Google-Extended disallow | Official Search Central blog | |
| Microsoft | Reported to start honouring preferences on 1 February 2027; reported to retire its older NoArchive/NoCache tags as a training and grounding preference by 1 March 2027 | Reported by independent outlets — Microsoft’s own blog post was not accessible at the time of writing |
| Anthropic | Reported to have already started implementing support | Reported — no separate company announcement published |
Two attributions deserve emphasis: Microsoft’s dates are reported by independent outlets because the company’s own blog post could not be reached directly, and Anthropic’s support is reported rather than announced — its only direct involvement so far is authoring the draft. Meta has reportedly expressed support without committing to implement. Google adds that Google-Extended has been offered since 2023 and that it considers the proposal consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act.
Protection will be uneven for months: the token is understood by each company only once it ships its implementation.
What publishers should do this week
This announcement is aimed squarely at web publishers — bloggers, news sites, and anyone monetising content with ads. A practical checklist:
- Keep your existing rules. Your current crawler controls continue to apply and remain respected. Do not remove them.
- Decide whether to add the token. If you do not want your content used for future AI training, add the two-line block to your robots.txt — or use the per-path
Allow:option for selected sections. - Check your Google-Extended setting before you add it. If you explicitly allow Google-Extended, your pages remain eligible to appear in answers and receive clicks through grounding in Gemini Apps; if you leave it unspecified, Google plans to treat the shared-token opt-out as a Google-Extended disallow. Decide deliberately.
- Remember the draft status. The specification is an individual draft expiring 3 May 2027 — watch whether it is renewed or adopted.
- Do not expect retroactive effect. Content already collected or trained on is unaffected.
- Keep expectations modest. It is voluntary, not law — four rivals agreeing on one signal is genuinely new, but it is not a guarantee.
For AdSense publishers, the stakes are direct: your content is the asset being discussed, and a shared signal costs almost nothing to add.
Summary
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic agreed to honour one shared robots.txt token — User-Agent: __usage_ai-training with Disallow: / — for opting out of AI training across all four at once. It is simple and free to adopt, but it does not stop crawling, does not cover AI-generated answers, does not undo past training, and is voluntary. Rollout is staggered, with the Microsoft and Anthropic timelines reported rather than directly announced. Publishers should keep their existing crawler controls, consider adding the token deliberately, and watch the draft’s status.
Does this remove my content from AI models that are already trained?
No. The token applies to future training only: it cannot remove content already collected or un-train a model that has processed it.
Will opting out affect my Google search rankings or traffic?
The token is not a crawling directive, so it does not change how Google crawls and indexes your site. What changes is the training preference: where the shared token opts you out and Google-Extended is not explicitly specified, Google plans to treat it as a Google-Extended disallow. Explicitly allowing Google-Extended keeps your pages eligible to appear in responses and receive clicks through grounding in Gemini Apps.
What exactly do I add to my robots.txt file?
Add two lines: User-Agent: __usage_ai-training, then Disallow: /. That opts out the whole site; add an Allow: line for exceptions. The token must be explicit — a wildcard User-Agent group does not count.
Is this a law, a formal standard, or just an industry promise?
It is an industry agreement built on a draft specification — an individual Internet-Draft, not endorsed by the IETF, dated 30 October 2026 and expiring 3 May 2027. It is voluntary: it binds only the companies that choose to honour it.
Will it stop my content appearing in AI answers like Gemini Apps?
Not directly. The token covers training — using content to change a model’s learned parameters — not the answering step where a model pulls in content to respond to a question (grounding). Google has said sites that explicitly allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps, while an unspecified Google-Extended combined with the shared opt-out is treated as a Google-Extended disallow. Training use and answer-time use are handled differently.
Ahesanur Asun is the Founder and Editor of UptoFeed, where he publishes updates on
AI news, artificial intelligence tools, and emerging technology. He focuses on explaining complex AI developments in a clear and practical way so readers can quickly understand and use the latest innovations.
