Technology: Google, Microsoft, OpenAI and Anthropic agree on a single rule for websites to opt out of AI training
Google, Microsoft, OpenAI and Anthropic announced on Friday that they will honor a single opt-out signal in the robots.txt file so websites can refuse, with one rule, to have their content used to train generative artificial intelligence models, according to a report published on October 31 by the specialized outlet The Next Web.
Until now, a publisher that does not want its content to feed these models has had to write separate rules for each company's crawler. The new signal works through a policy token, __usage_ai-training: the site adds a group with that token and lists the paths it does not want used for training. The draft's example blocks an entire site except its press kit.
The proposal comes from an individual draft submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley of Anthropic. The document states that the token expresses a preference about how content is used, not an instruction about crawling: it does not stop search engines from indexing the page or remove it from results, and it cannot undo training that has already taken place. In addition, private agreements between a publisher and an AI company can override the signal, and the IETF has not formally endorsed the draft.
Support will arrive on different schedules. Anthropic has already begun work on its implementation; Microsoft will honor the preferences starting February 1, 2027, and will stop recognizing its NoArchive and NoCache tags as training or grounding signals as of March 1, 2027; Google aims for early 2027 and will treat the signal as a block on Google-Extended, the crawler that feeds its Gemini models; and OpenAI plans to adopt it in the coming months, while its current controls, such as GPTBot, remain in effect.
Microsoft also warned that blocking AI grounding will significantly reduce content visibility in AI-generated experiences, although it will not change a page's place in Bing search results. Google said the proposal is consistent with the European Union's Code of Practice for general-purpose AI, which recognizes robots.txt as the main way to state machine-readable preferences.
Photo: Pivot News
Source: The Next Web / Pivot News