Tech Giants Commit to Unified Protocol for AI Data Scraping

Several leading technology companies have committed to adopting a unified protocol to prevent unauthorized data scraping from media and websites for training artificial intelligence models. The companies involved include OpenAI, Google, Microsoft, and Anthropic. This move comes amidst growing criticism and legal battles from news publishers over the use of their content without permission.

The proposed solution will allow publishers to issue a single set of instructions recognized and adopted by any AI company that adheres to the protocol. Currently, publishers must apply multiple blocking orders for each data-collecting bot individually. However, the new protocol will not be fully implemented for several months. Microsoft will begin applying it from February 1, 2027, and Google will follow sometime in 2027. OpenAI, the developer of ChatGPT, will roll it out in the coming months, while Anthropic, the developer of Claude, has already begun implementation.

This announcement follows a surge in bots copying publishers' content to feed and train AI models. Some entities, like The New York Times, have turned to legal action. In December 2023, The New York Times filed a lawsuit against OpenAI and Microsoft, alleging the unauthorized use of millions of articles to train the ChatGPT model without permission or compensation.

In response to these issues, Cloudflare, an internet services provider, has automatically blocked AI-powered software on new client sites since July 2025. Additionally, a program called 'Very Simple Licensing' enables publishers to charge fees for bots that visit and copy their pages. Microsoft clarified in a blog post that the new setting will only apply to future model training and will not prevent bots from crawling the internet or extracting content until publishers change this setting.

Andy Stone, a spokesperson for Meta, expressed support for this initiative on the platform X, stating, 'We believe creating a shared signal that conveys these preferences to many service providers at once is an important step.' Stone added, 'We look forward to working with other AI companies on these standards and applying them across the industry.'