CYBERSECEVAL 3: Advancing Cybersecurity Risk and Capability Evaluations of Large Language Models

The researchers release a large language model cybersecurity benchmark suite, CYBERSECEVAL 3, for empirically measuring the cybersecurity risks and capabilities of large language models. The suite evaluates 8 risk categories across two major classes, namely risks to third parties and risks to application developers and end users. Compared with prior work, it adds evaluation areas focused on offensive security capabilities, including automated social engineering, scaled manual offensive cyber operations, and autonomous offensive cyber operations. The paper discusses applying these benchmarks to the Llama 3 models and a set of contemporaneous state-of-the-art large language models, thereby comparing risks with and without mitigations.