FAR.AI finds exploitable vulnerabilities in GPT-4 fine-tuning and the Assistants API
We Found Exploits in GPT-4’s Fine-tuning & Assistants APIs | FAR.AI
FAR.AI finds multiple classes of exploitable vulnerabilities in GPT-4's fine-tuning API and Assistants API. Fine-tuning with as few as 15 harmful examples or 100 harmless examples can remove GPT-4's core guardrails; after fine-tuning on harmless data, the model responds to 81% of harmful requests from AdvBench. With 15 negative examples, the model can be made to produce biased information about a specific person; 35 examples can make the model insert a malicious URL in code; 10 question-answer pairs can make the model leak real email addresses that did not appear in the fine-tuning data. For the Assistants API, the model lists all callable functions and their schemas, executes arbitrary function calls with user-specified parameters, and even helps generate SQL injection payloads. Hidden instructions in retrieved documents (such as setting font color to the same as the background) can hijack the model's output, and changing it to a transfer function call also succeeds. The authors therefore recommend against deploying LLMs in safety-critical scenarios.
FAR.AI quantifies the attack surface introduced by the new features of the fine-tuning API and Assistants API with concrete experiments, which teams doing fine-tuning or tool calling can use to assess their own risk.