Technology: OpenAI, Google, Microsoft and Anthropic agree on a single signal to block content use in AI training
San Francisco — OpenAI, Google, Microsoft and Anthropic have announced they will support a shared signal in the robots.txt file so that websites can state, with a single instruction, that they do not want their content used to train generative artificial intelligence models. The announcement, made public on Thursday, October 30, seeks to simplify a process that today forces publishers to write a different rule for each crawler from each company.
The robots.txt file is the document through which a website tells internet robots which parts they may visit. Until now, an outlet that wanted to block AI training had to add one entry per provider, and another for each crawler if the company used several. With the shared signal, a single line will speak to every provider that adopts it.
The proposal is set out in a draft from the Internet Engineering Task Force's (IETF) AI Preferences Working Group, titled “Machine-Readable Training Preference Signal” and written by Kevin Kelley of Anthropic, dated October 30, 2026. It is an individual draft, an early stage of the standardization process, not a formal standard already adopted.
The timelines are not the same for everyone. Anthropic has already begun applying the signal, OpenAI said it will implement its support “in the coming months,” Microsoft will do so starting February 1, 2027, and Google sometime in 2027. Microsoft will also offer a control called AI Grounding for cases in which a model consults a page to answer a question, rather than to collect training data.
The scope has important limits that the announcements themselves acknowledge: the signal applies only to future training of generative models, it does not prevent crawling for other purposes, and content is considered usable by default until the publisher changes the setting. OpenAI clarified that the new signal does not replace the controls it already offers and that its current crawler rules will remain in force.
The agreement comes amid tensions between publishers and AI companies over the use of articles to train models, including the lawsuit The New York Times filed in December 2023 against OpenAI and Microsoft. Meta expressed support for the idea, according to reports, but has not committed to implementing it.
Photo: Archivo A Lo Claro
Source: OpenAI