Paper: changing the judge model changes jailbreak ASR by up to 82 percentage points

The paper notes that merely changing the judge LLM in jailbreak evaluations can change the reported ASR by up to 82 percentage points; for example, I-GCG fluctuates between 6% and 88%. The authors propose a semantic reconstruction game, where the LLM first reconstructs the request it actually answered based only on the reply, then generates 5 similar decoy requests, compares one by one whether the reconstructed request and the original request require the same action and result, and decides by majority vote whether it is a jailbreak, abstaining when neither matches. On human-annotated responses from SorryBENCH, StrongREJECT, and HarmBench, this method achieves the lowest average error across three judges—Qwen 3.5 9B, Ministral 3 8B, and Gemma 4 12B—and reduces the ASR span on I-GCG from 82 percentage points to 12. The method is deliberately conservative, preferring to miss successful jailbreaks rather than risk false positives.