DualGauge: Jointly Evaluating the Safety and Functionality of LLMs and Coding Agents Generating Code from Specifications Alone

The paper proposes the DualGauge framework and the DualGauge-Bench benchmark for automatically and jointly evaluating the functional correctness and safety of code generated by LLMs and coding agents from natural language specifications alone. The benchmark contains 307 programming tasks, each equipped with functional tests and safety tests derived from the same specification. On 10 representative LLMs, functional correctness substantially overestimates reliable code generation; even the strongest model has a joint safety and functional success rate below 15% in each language. Common factors such as model scale, extended thinking, quantization, instruction tuning, and code specialization did not reliably improve joint performance. Evaluations of Codex, OpenHands, and Claude Code show that iterative scaffolding offers no advantage over direct generation on specification-only tasks, with failures concentrated at output contract boundaries and in present but insufficient guardrails.