A website I once crawled: awareness, agency and effectiveness of protecting content creators from AI crawlers

Through large-scale measurement and a user study with 203 professional artists, the researchers evaluated the ability and effectiveness of existing web tools in protecting content creators from AI crawlers. Generative AI relies heavily on crawling web data for training, which has sparked disputes over copyright, privacy and other issues; existing tools include robots.txt, the NoAI meta tag, and active crawler blocking by reverse proxies. The study finds that artists have a strong demand for tools such as robots.txt, but are constrained by key obstacles including insufficient technical knowledge, lack of deployment autonomy, and limited effectiveness against crawlers that do not respond. The study also tested network-level crawler blocking provided by reverse proxies; such tools are currently deployed to a limited extent but can provide stronger protection against AI crawlers, while also having their own limitations.