FAR.AI discloses jailbreak-tuning data poisoning attack: GPT-4o refusal rate drops to 3.6%

FAR.AI proposes a data poisoning attack called jailbreak-tuning, mixing jailbreak instructions into fine-tuning data, reducing GPT-4o's refusal rate to 3.6%, which the authors say bypasses the guardrails tested in the study. The research tested GPT-4o, GPT-4o mini, GPT-4, and GPT-3.5 through the OpenAI fine-tuning API, plus 23 models from 8 open-source model families (1.5B to 72B parameters); open-source models were fine-tuned with QLoRA for 5 epochs at poisoning rates from 0% to 2%. Results show that larger models learn harmful behavior more easily, but the Gemma 2 family shows inverse scaling, with larger models being less susceptible to poisoning. The authors argue that jailbreak-tuning is easier to learn and requires less data than standard fine-tuning, and should become a standard part of safety testing before model deployment.