Jailbreak Attacks Versus Content Safety Filters: Where Has the Large Language Model Safety Arms Race Reached?

The researchers systematically evaluate jailbreak attacks against the safety alignment of large language models, and the authors claim this is the first time the evaluation scope covers the complete inference chain including the input and output filtering stages. Results show that almost all evaluated jailbreak techniques can be detected by at least one safety filter, suggesting that previous evaluations targeting only the model itself may have overestimated the actual success rates of these attacks. The paper also points out that although safety filters have detection capability, there is still room for optimization in balancing recall and precision, and the authors call for further improving the detection accuracy and usability of large language model safety systems.