Paper: Switching the judge model in jailbreak evaluations changes the reported attack success rate by up to 82 percentage points

Liam Davis; Romain Emil-Popescu; Suman Jana

Paper: Switching the judge model in jailbreak evaluations changes the reported attack success rate by up to 82 percentage points

Paper Tracker· arXiv:2610.19306· Liam Davis, Romain Emil-Popescu, Suman JanaAmherst College; Columbia UniversityIndexed by this site 2026-10-15 11:45· 29 days ago· Originally publishedOct 14Selected 2026-10-15 11:46EvaluationSelectedAttention39

#Defenses and guardrails

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (generated by Gemini); refer to the original

AI summaryThe paper notes that merely changing the judge LLM in jailbreak evaluations can change the reported ASR by up to 82 percentage points; for example, I-GCG fluctuates between 6% and 88%. The authors propose a semantic reconstruction game, where the LLM first reconstructs the request it actually answered based only on the reply, then generates 5 similar decoy requests, compares one by one whether the reconstructed request and the original request require the same action and result, and decides by majority vote whether it is a jailbreak, abstaining when neither matches. On human-annotated responses from SorryBENCH, StrongREJECT, and HarmBench, this method achieves the lowest average error across three judges—Qwen 3.5 9B, Ministral 3 8B, and Gemma 4 12B—and reduces the ASR span on I-GCG from 82 percentage points to 12. The method is deliberately conservative, preferring to miss successful jailbreaks rather than risk false positives.

Questions: Existing LLM jailbreak evaluations rely on prefix matching or single judge queries, which can easily misclassify refusal or off-topic responses as successful jailbreaks, causing the reported attack success rate of the same attack to fluctuate significantly merely by changing the judge model.
Methods: Convert jailbreak determination into a semantic reconstruction game: first, the model reconstructs the request that was substantively answered based only on the reply content, excluding refusals or repeated words; then, generate 5 decoy requests for the original request; finally, compare the reconstruction against the decoys pairwise to see which better matches the original request, and determine jailbreak success by majority vote.
Experiments and results: On the StrongREJECT and HarmBench human-annotated benchmarks, this method achieves the lowest mean absolute error across three judge models; across five attack evaluations in AdvBench, the spread of success rates for each attack across different judge models drops from up to 82 percentage points with direct judging to 12 to 28 percentage points.
Limitations and future work: The experiments only cover models in the 8B to 12B scale as judges and attack targets, without testing larger models; only single-turn final replies are evaluated, not involving chain of thought, multi-turn jailbreaks, or multi-agent interactions; the number of decoys is fixed at 5 without ablation; the method is conservative and has false negatives.

Why it was selectedThe paper quantifies the perturbation magnitude of judge model differences on jailbreak attack success rates and provides a training-free reconstruction-style evaluation pipeline that teams doing jailbreak evaluation can use to check ASR standards.

The paper quantifies the perturbation magnitude of judge model differences on jailbreak attack success rates and provides a training-free reconstruction-style evaluation pipeline that teams doing jailbreak evaluation can use to check ASR standards.