National University of Defense Technology proposes the JailExpert jailbreak framework, with attack efficiency 2.7 times that of the best baseline

Researchers at the National University of Defense Technology published the jailbreak framework JailExpert on arXiv, which structures historical jailbreak attack results into reusable experience, achieving an average attack success rate of 90% across 7 LLMs, with attack efficiency (ASR-E) 2.7 times that of the best baseline, ReNeLLM. Each experience record contains the initial malicious instruction, the complete jailbreak prompt, the jailbreak pattern, and the numbers of successes and failures; the framework uses text-embedding-3-small to compute the semantic drift between the instruction and the complete prompt, and after grouping via K-Means, selects from each group the pattern with the highest occurrence frequency and historical success rate as the representative template, then dynamically updates the counts after attacks. In the experiments, GPT-4's attack success rate was 76%, while Gemini-1.5-pro and GPT-4-Turbo reached 100%; LlamaGuard reduced GPT-4-Turbo's success rate from 95.5% to 62.4%, and PPL Filter had no effect on the attack success rate against Llama2-7b (97.3%).