Study reveals systematic vulnerability of open-weight models to prefill attacks
Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks
Translation in progress — untranslated text is omitted.
Researchers evaluated the effectiveness of 23 prefill strategies on 50 open-weight models from 6 vendors, finding that model-agnostic prefill attacks can induce harmful outputs in all tested models, with most success rates exceeding 95%; the authors call this the largest empirical study of this attack surface to date. The experiments mainly used 179 CBRN harmful requests from ClearHarm, with two guardrail models, GPT-OSS-Safeguard 20B and Qwen3Guard 8B, judging and the lower judgment taken as the success rate; the strongest single strategy reached 74% on GLM-4.7 and over 90% on Kimi-K2-Thinking, DeepSeek-R1, Qwen3-Next Instruct and GPT-OSS, and approached 100% when all strategies were used.
- When open-weight models are deployed locally, they cannot rely on external filters and mainly rely on internal alignment to prevent misuse. Prefilling lets attackers predefine initial response tokens, potentially bypassing refusals, but it previously lacked systematic study.
- The authors define 23 model-agnostic prefill strategies (most newly proposed), use abliterated Gemma3 27B to generate 5 variants for each harmful request, evaluate on 50 models, judge success with dual judges GPT-OSS-Safeguard and Qwen3Guard, and explore model-specific prefill.
- The authors state that all tested models can be broken by prefill, with ASRany close to 100% for most models; model scale does not bring robustness; GPT-OSS and Qwen3-Next Thinking have some resistance to generic prefill, but model-specific prefill still reaches 92.2% and 83.8% ASR.
- The authors note: the requests were mostly general information queries, and highly specialized requests were not tested; the evaluation relied entirely on automated judging, with misclassification and no manual verification of the factual correctness of harmful content. Future directions include multi-stage prefill, combination with input jailbreaks, and LLM-generated prefill.
The paper compares 23 prefill strategies on 50 open-weight models from 6 vendors, and reports the relationship between attack success rate and model scale and inference stage.