How to read attack success rate (ASR) numbers: scoring criteria, number of trials and dataset can turn the same evaluation result into 15% or 100%

The author uses a runnable Python example to show that, for the same batch of evaluation results, merely changing the scoring criteria, number of trials and dataset composition can make the attack success rate (ASR) go from 15.0% to 100%. The example sets 40 prompts, each generating 10 responses at temperature 1.0: when success is counted as "no refusal statement in the response," ASR@1 is 52.5% and ASR@10 is 100%; when only "provides complete specific information" is counted, they are 15.0% and 67.5% respectively. With 40 samples, ASR@1 is 32.5%, with a 95% interval of 20.1% to 48.0%; after weighting by category proportions, the same model's results can vary between 11.6% and 42.9%. When comparing two defenses (35.0% vs. 22.5%), McNemar's exact test gives a p-value of 0.125, and with only these 40 samples it cannot be concluded that the two differ.