SINGED: correct output does not guarantee the execution safety of LLM agents
SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers from The Hong Kong Polytechnic University and other institutions propose the SINGED benchmark, which uses "functional impersonation" implementations that produce identical output but carry task contracts forbidding effects, to test the source-selection behavior of LLM agents among multiple candidates with the same name. In 7,549 audit trials across 5 main task categories, 20 model versions and 8 agent configurations, the outputs of 180 random-ranking trials were all correct, yet 27 trajectories executed the impersonating implementation, all of which occurred when the impersonating source ranked first. When comparing all candidates was required, impersonation execution in the rank-first scenario dropped from 27/60 to 0/60; but across 324 public package cells it dropped only from 36/162 to 26/162, and the risk rose again once the execution evidence moved to deeper dependencies. Among the seven versions with zero impersonation execution under three-source selection, after removing benign candidates they still executed the sole impersonating implementation in 55/175 cells, indicating that their safety performance depends on whether selectable alternatives exist.
- Tool-using LLM agents select one of multiple third-party implementations with the same name, interface, and output to execute; looking only at whether the output is correct cannot reveal that a given implementation secretly adds side effects prohibited by the task contract (functional counterfeit). This type of risk requires no malicious instruction, only a seemingly credible but contaminated source, so output correctness and execution-path integrity need to be evaluated separately.
- The authors construct the SINGED controlled benchmark: each instance fixes the request, interface, and expected artifact, and provides three output-equivalent candidates (the expected implementation, a benign community alternative, and a functional counterfeit), with only the counterfeit triggering prohibited events. The experiments cross-vary display ranking, evidence depth (entrypoint/one layer/two layers of dependencies), decision strategy, model version, and agent configuration, and use a task oracle and a process oracle to check the artifact and in-sandbox execution events, respectively.
- In 7,549 audit trials, all outputs were correct, but counterfeit execution concentrated when ranked first: 27/60 in the original study, and zero at subsequent ranks; the same gradient appears across five tasks. Requiring comparison of all candidates can reduce shallow failures from 27/60 to 0/60, but the benefit weakens when reliance is deeper; seven models with zero counterfeit execution under three-source selection still executed counterfeit in 55/175 single-source units after benign alternatives were removed. Transfer results across tasks, configurations, and public package structures were inconsistent.
- Comparison strategies can only reduce, not eliminate, risk: deeper dependencies, unseen process effects, model version changes, and failures remaining under public package structures; in public package migration, the overall benefit confidence interval of comparison strategies crosses zero. The authors note that the observed safety depends on available substitutes and system configuration, and that future work needs to bind outputs to the identity of the implementation being checked and executed and to process events, and requires provenance and system-level controls.
Using 7,549 controlled trials, the paper shows that correct output masks execution paths with hidden side effects, that failures concentrate in ranking first and stopping prematurely, and that this can provide a reference for the design of agent source-selection evaluations.