The researchers propose the cost-aware jailbreak evaluation metric ASR_STEP
Start here
In October 2026, researchers published a paper on arXiv proposing a cost-aware jailbreak evaluation metric, ASR_STEP: while retaining the binary success determination, it penalizes additional calls to the victim model during iterative jailbreak attacks. Tests on JailBreakBench across three model families show that GCG's score drops by 7.0–8.7 percentage points after discounting, the single-step PAP baseline remains unchanged, and PAIR's discount ranges from 3.4 to 12 percentage points, indicating that the meaning of "steps" differs across iterative strategies. The authors state that reporting both the traditional attack success rate and this metric makes the discount caused by repeated calls to the victim model explicit.
Generated by AI from reports · Updated 24 days ago
Attention and developments
Attention, hourly
Now 31, 0% in 24 hours·Peak 31 (Nov 6 04:00)
Related research
This event's paper
- Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
arXiv · 2026-10-19 · Deep reading
Progress timeline
- arXiv: Jailbreaks, Prompt Injection, Poisoning & DefensesBeyond Attack Success Rate: A Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
The researchers propose the ASR_STEP metric, which, while retaining the binary success determination, penalizes additional calls to the victim model during the attack, for evaluating iterative LLM jailbreaks. Testing on JailBreakBench across three model families shows that GCG's score drops by 7.0–8.7 percentage points after discounting, the single-step PAP baseline remains unchanged, and PAIR's discount ranges from 3.4 to 12 percentage points, indicating that the meaning of "steps" differs across iterative strategies. The authors state that reporting both the traditional attack success rate and this metric makes the discount from repeated calls to the victim model explicitly visible. The code is publicly available on GitHub (SuELS-Group/beyond_asr).