Skip to content
WeChat · LLM Security Research·Indexed by this site · Originally published· Defenses →· Its place in the scan record →SelectedAttention40

Researchers propose the Canary Token method to identify AI web crawlers using differentiated web page content

Canary Token:以差异化网页内容识别AI网页爬虫

AI summary

Six researchers from Duke, Pittsburgh and Carnegie Mellon universities published a paper at ACM CCS 2026 proposing the use of differentiated web page content to identify AI web crawlers. The method sends each visitor a version of the page embedded with different tokens; whichever set of tokens appears in a chatbot's answer points to the crawler that took the content (arXiv 2605.13706). The team set up 20 sites and left them for two months, testing 22 chatbots with web retrieval capability; 18 successfully elicited tokens, 10 matched the vendors' own crawlers in their public documentation, while Qwen and Perplexity taking content via the Googlebot path had no public record. In blocking experiments, 7 of 8 bots relying on search crawlers still repeated content a week after the site went offline, and 12 of 18 returned site content under both offline and crawl-prohibited conditions, with only Duck.ai stopping; the paper emphasizes that this cannot prove a given crawler violated robots.txt, and that once content is indexed, rules added afterward cannot retract visits that already occurred.

Why it was selected

The paper provides a reproducible method for attributing differentiated content, allowing content providers to check, without vendor cooperation, which crawlers took content and whether it is still repeated after going offline.

Read originalmp.weixin.qq.com