Honeyval: A Comprehensive Evaluation Framework for LLM-Driven HTTP Honeypots

The researchers propose Honeyval, a comprehensive evaluation framework for LLM-driven HTTP honeypots, which addresses the shortcomings of previous evaluations—lack of scalability, reproducibility, and representativeness—through 16 backend applications, AI attack agents, two control tasks, and explicit, verifiable exploitation goals. Experiments using this framework with recent cost-effective LLMs as HTTP honeypots show that such honeypots interact with attackers for significantly longer than rule-based baseline honeypots, are detected less often even against frontier models, and on average maintain a runtime cost advantage relative to the attack agents. The authors also tested different counterattack-type honeypot configurations and observed unique trade-offs such as longer interactions but higher detection rates.