GreyZone: A Production-Grade Threat Benchmark for Agent Applications

Researchers propose the GreyZone benchmark to evaluate the attack capability of automated red-teaming services against real agent applications. Unlike AgentDojo, which mainly evaluates the attacked subject Agent, GreyZone examines whether a red-teaming service can generate and deliver attacks that cause verifiable security policy violations. The benchmark selects independently developed open-source agent applications whose security boundaries involve access permissions, confidentiality, and operational authorization, and whose attack surfaces include multi-agent interaction and multimodal input. The authors conducted two preliminary case studies with Promptfoo on Letta Code and Clawith, examining attack delivery, Agent behavior, and verification results, and released a benchmark containing each application's subject Agent and attack surface.