OpenAI, Google, Microsoft and Anthropic support a new standard letting publishers refuse AI scraping

OpenAI, Google, Microsoft and Anthropic said they will adopt a new standard that lets publishers set their preference to refuse AI scraping once through a unified signal, without having to declare it separately to each company. The shared policy token signal was proposed at the IETF by Kevin Kelley, Anthropic's standards technical lead, and is written into a website's robots.txt file; the four companies said that when they encounter the signal, they will restrict their own AI crawlers' access to the corresponding websites. Implementation progress varies: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI said it will implement it within the coming months. Microsoft will also offer more granular controls, allowing publishers to opt out of AI training or AI grounding separately, that is, directing models to verified information sources to reduce hallucinations; regardless of preference, websites will still appear in Bing search results.