The guardrails worked, but is the LLM system safer?

The paper proposes using how much harmful task help a deployed service can still provide when an attacker continuously adapts or seeks other avenues as a unified criterion for interpreting guardrail evaluation results, making results from different guardrail families comparable. The authors argue that the evidential requirements are highly asymmetric: a single attack that obtains harmful help from the deployed service suffices to prove that such help still exists, whereas to prove that little remains, the guardrail's own refusal rate, attack success rate, and policy violation rate are insufficient; evidence is also needed on what the surrounding system still allows after the guardrail completes a local function, and such evidence constitutes only a minority of the claims coded by depth, with only one claim giving a residual upper bound within its scope. The paper concludes that better local scores alone do not constitute a stronger safety claim about the deployment, and guardrail research cannot stop at improving local scores.