CESBench: Evaluating the Capabilities of Large Language Models in Cryptographic Engineering Security for IoT Devices

The researchers propose the CESBench benchmark, which uses 380 expert-written questions to evaluate the capabilities of large language models in cryptographic engineering security for IoT devices, covering six subdomains: side channels, fault injection, implementation, countermeasures, evaluation, and integration. The questions fall into four categories: 209 multiple-choice questions test memorization, 67 true/false questions require a security judgment and rationale, 63 scenario questions require engineering diagnosis, and 41 code questions are scored by 572 test cases. 11 open-weight and proprietary models answered all questions, with overall scores between 54.4% and 83.6%. The highest scores by question type were 98.6% for multiple-choice, 95.1% for code, and 88.4% for scenario diagnosis, while true/false questions scored only 58.8%; across models, 88.5% of security judgment conclusions were correct, but the arguments received only 53.4% of the rubric score, making providing arguments for security judgments the weakest capability. The benchmark, prompts, and per-question results have been made public.