Skip to content
WeChat · LLM Security Research·Indexed by this site · Originally publishedSelectedAttention39

SINGED benchmark: correct output does not mean safe execution

SINGED:安全的世界充满着欺骗,正确输出不代表执行安全

AI summary

Researchers from The Hong Kong Polytechnic University and Hangzhou Dianzi University proposed the concept of "functional counterfeit" and built the SINGED benchmark to test whether tool-using agents can identify third-party implementations that produce correct output but carry effects prohibited by the task contract. 7549 audit trials covered 20 versions across 14 model families and 8 agent configurations; when a counterfeit was ranked first among candidates, 45% of executions selected it, and judging by output alone, all of these compromised executions showed task success. Forcing comparison of three candidates reduced the first-position selection rate from 27/60 to 0/60, but the benefit diminished once the decisive evidence was buried one to two dependency layers deep; seven model versions that had zero selections under three-source choice executed the counterfeit in 55/175 cells when only a single counterfeit source remained. The paper is a preprint (arXiv 2609.35889) and has not been peer-reviewed.

Why it was selected

The 7549 audit trials provide concrete numbers for the rank effect and the single-source scenario, which teams evaluating tool-using agents can reference when designing process verification.

Related papers
Read originalmp.weixin.qq.com