arrow_backNeural Digest
AI company logos alongside a web scraping opt-out symbol
Policy

Big AI Firms Back Universal Opt-Out Scraping Standard

The Verge AI4h ago
auto_awesomeAI Summary

“OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a shared 'policy token' standard that lets publishers set AI training preferences once, applying them across all four companies simultaneously. Previously, publishers had to negotiate or signal opt-outs with each AI firm individually. This coordinated move represents a rare instance of direct collaboration between competing AI giants on a content governance issue.”

Key Takeaways

  • OpenAI, Google, Microsoft, and Anthropic will all honour a single shared 'policy token' opt-out signal from publishers.
  • Publishers previously had to set scraping preferences separately with each AI company, creating significant administrative burden.
  • The standard was reported by Search Engine Roundtable and allows training preferences to propagate across multiple AI firms at once.

OpenAI, Google, Microsoft, and Anthropic unite behind a single publisher scraping opt-out signal.

trending_upWhy It Matters

This standard could meaningfully shift power back toward publishers and content creators who have struggled to enforce opt-outs as AI training datasets expanded rapidly. If widely adopted, it reduces the legal and technical friction that has fuelled high-profile copyright disputes between media organisations and AI companies. However, the initiative only covers four companies for now, leaving the broader ecosystem of smaller scrapers unaddressed. Watchpoints include whether independent AI labs and open-source model trainers sign on, and whether compliance will be self-reported or independently verified.

FAQ

What is a 'policy token' and how does it work?

A policy token is a shared signal that publishers embed in their content or metadata to declare their AI training preferences. Participating AI companies agree to read and honour this signal, meaning a publisher sets their opt-out once and it applies across all adopting firms simultaneously.

Does this mean AI companies will stop scraping opted-out content immediately?

The announcement signals intent to adopt the standard, but implementation timelines have not been publicly confirmed. Publishers should monitor whether each company publishes technical documentation and a compliance rollout schedule before assuming their opt-outs are being respected.

Does this standard apply to all AI companies, or just the four named?

As of this announcement, only OpenAI, Google, Microsoft, and Anthropic have committed to the standard. Numerous other AI developers, including open-source model trainers, are not covered, which limits the standard's reach across the wider AI training ecosystem.

This summary was AI-generated. Neural Digest is not liable for the accuracy of source content. Read the original →
Read full article on The Verge AIopen_in_new
Share this story

Related Articles