Skip to content

Policy & law Story

Google, Microsoft, Anthropic and OpenAI agree on a shared robots.txt line on AI training

On 30 October, Google, Microsoft, Anthropic and OpenAI announced they would honour a single robots.txt line, the "__usage_ai-training" token, letting sites refuse or allow the use of their content for generative AI training.

Published

This story was written with the help of artificial intelligence, checked automatically, then reviewed by an editor before publication, under our AI policy.

A shared signal announced on 30 October

Google, Microsoft, Anthropic and OpenAI announced on 30 October that they would follow the same line in robots.txt, so that website owners can refuse or allow the use of a site for generative AI training.

The text defining the signal is titled "Machine-Readable Training Preference Signal". It was written by Kevin Kelley, an Anthropic engineer, and is aimed at standardisation by the IETF. It remains a draft, set to expire on 3 May 2027 unless a new version is published before then.

Google separately said it intends to support a shared AI training control in robots.txt, based on work by the IETF's AIPREF group. According to the company, site owners could state their preferences to several providers in a single step.

How the token works

The signal adds the token "__usage_ai-training" to the User-Agent line. A Disallow rule expresses a refusal of training and an Allow rule expresses consent, set separately for each path.

The token does not block crawling, which stays governed by the existing rules aimed at bots. It is declarative, binds only the providers that have committed to it, and applies only to content retrieved after the token is added.

Google-Extended

According to Google, if a site uses the token to switch training off without mentioning Google-Extended explicitly, Google plans to read that as a ban on Google-Extended. Google also says sites that specifically allow Google-Extended may appear in answers and receive clicks through grounding in Gemini apps. Both points are company statements.

Limits and open questions

The draft provides for no checks and no penalties. According to Next, Apple, Amazon and Meta have not committed to respecting the token.

Google says it plans to support the new control, along with other AI providers, in 2027. The material available does not say when the other announcing companies will apply it.

Google also claims the proposal complies with the EU code of conduct for general-purpose AI model providers. That is the company's assertion, not an independent finding.

Analysis: with no enforcement mechanism, the token's value will depend on how many providers adopt it and on how the IETF process ends. It offers a common format for site owners, but the facts do not point to any guarantee beyond providers' own commitments.

FAQ

Does the token stop bots from crawling a site?

No. It does not block crawling, which remains governed by the rules aimed at bots.

Who has committed to honouring the signal?

Google, Microsoft, Anthropic and OpenAI. According to Next, Apple, Amazon and Meta have not.

Does it apply to content already collected?

No. It only covers content retrieved after the token is added.

When does Google plan to support it?

Google says it plans to support the control in 2027.

Sources

  1. Google, Microsoft, Anthropic et OpenAI veulent changer robots.txt pour refuser les IA, Next,
  2. https://developers.google.com/search/blog/2026/10/ai-training-control?hl=fr, developers.google.com, (primary source)

Plain text version (Markdown)