Beyond Attack Success Rate: A Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
Translation in progress — untranslated text is omitted.
The researchers propose the ASR_STEP metric, which, while retaining the binary success determination, penalizes additional calls to the victim model during the attack, for evaluating iterative LLM jailbreaks. Testing on JailBreakBench across three model families shows that GCG's score drops by 7.0–8.7 percentage points after discounting, the single-step PAP baseline remains unchanged, and PAIR's discount ranges from 3.4 to 12 percentage points, indicating that the meaning of "steps" differs across iterative strategies. The authors state that reporting both the traditional attack success rate and this metric makes the discount from repeated calls to the victim model explicitly visible. The code is publicly available on GitHub (SuELS-Group/beyond_asr).
- Conventional ASR does not distinguish a single success from a success after many victim model calls. The authors argue that each additional call gives deployment-side monitoring another opportunity to detect and intervene.
- ASRSTEP keeps the binary success/failure outcome, records failures as 0, and applies a quadratic discount to successes based on the number of additional victim calls; the experiments use c=0.2. Single-step attacks receive no discount and, after averaging, reduce to ASR. For GCG, the step count uses the lowest-loss index.
- On JailBreakBench, GCG, PAIR, and single-step PAP are compared. Across the three model families, GCG drops by 7.0–8.7 percentage points, while PAP is unchanged; PAIR is discounted by 3.4–12 percentage points. In Table 1, PAP is higher than PAIR on Zephyr.
- The authors state that the evidence is limited to JailBreakBench, three model families, the selected configurations, and a single judge, and that classification errors may introduce bias. Future work plans to use repeated evaluations to estimate the expected number of steps and its uncertainty, and to add a non-binary harm score. The experiments do not report N, the number of excluded items, or the PAP rewriting model.
Related events and news
2 events · 1 news mentions
- This event's paper
研究者提出成本感知越狱评测指标ASR_STEP
News that mentions this paper
大模型安全周报：防线开始溃于组合，智能体授权与审计系统性缺口
- 公众号 大模型安全研究cites the paper's id or link大模型安全周报：防线开始溃于组合，智能体授权与审计的系统性缺口
Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks（https://arxiv.org/abs/2610.26589）