FAR.AI red-team testing: guardrails of DeepSeek R1 and OpenAI, Anthropic, and Google fine-tunable models can be easily removed

FAR.AI used jailbreak fine-tuning attacks to test DeepSeek R1-Distill-Llama-70B as well as OpenAI GPT-4o, Anthropic Claude 3 Haiku, and Google Gemini 1.5 Pro, and found that these models' guardrails can be removed while response quality is largely preserved. The attack method adds jailbreak phrases during both training and inference, fine-tunes on harmful question-answer pairs from Harmful SafeRLHF, mixes in BookCorpus data for GPT to bypass review, and uses a benign dataset containing only the letter a for Claude to bypass review. On the StrongREJECT benchmark, before fine-tuning these models had refusal rates close to 100% and maximum harmfulness scores of only 6%; after fine-tuning, harmfulness scores all exceeded 80% and they almost never refused. The authors noted that closed-source models can block this type of attack with a simple output filter, but for more complex attacks there is currently no known method that can guarantee the safety of fine-tunable models, and recommended conducting an evil twin evaluation on every fine-tunable model before deployment.