AI giants offer publishers a single way to block scraping

· Tech · United States

Major AI companies have promised publishers a new way to block some scraping, as news organizations face a surge in bots that collect their articles to feed AI models. Publishers currently have to implement multiple blocking commands for every bot that crawls the web, but the proposed solution would let them enter a single, standardized instruction that is recognized and honored. The mechanism aims to simplify how publishers control whether their content is used for AI training. The report did not name the companies involved or say when the standard would take effect.

Why it matters：A shared standard could reshape the bargaining power between AI developers and content producers, determining who captures the value of news and other published material.

Key points

  • Publishers currently must issue multiple blocking commands for each bot.
  • The proposed standard would use a single, recognized instruction.
  • The report did not name the companies or a timeline.

Sources (original reporting)

More stories from this day