Be cautious when fine-tuning on open-source LLMs: your fine-tuning data may be secretly stolen
Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
Translation in progress — untranslated text is omitted.
Research by the Tsinghua University CoAI team and others points out that providers of open-source LLMs can implant a backdoor before release, after which, using only black-box access to the downstream fine-tuned model, they can extract the developer's private fine-tuning queries. The authors state that on 4 open-source models ranging from 3B to 32B (including Qwen and Llama) and two downstream datasets, Dolly and Finance, under realistic settings up to 76.3% of queries among 5000 samples could be fully reconstructed, and up to 94.9% when the opening words of the downstream data are known (Qwen2.5-32B). The authors attribute the cause to computing loss on training queries during fine-tuning, which is the default setting in frameworks such as Hugging Face TRL. The defenses they tried, including similar-instruction detection, increasing fine-tuning epochs and differential privacy, all failed to work without significantly degrading model utility. Code and data are open-sourced on GitHub.
- Downstream developers commonly fine-tune open-source LLMs with private data; the authors claim the model provider can implant a backdoor in advance and afterward, with only black-box access to the fine-tuned model, extract private fine-tuning queries, which are proprietary and costly to collect.
- The provider performs backdoor training before release: SFT or GRPO teaches the model to verbatim recite training queries when it receives an extraction instruction with an opening word; in the extraction stage, a public opening word set is used and valid words are filtered by heuristic scoring, then completions are sampled for each word.
- On 4 open-source models of 3B–32B and two downstream datasets of 5,000 cases each, Dolly and Finance, the authors report that in the practical setting at most 76.3% of queries are fully recovered, and in the ideal setting up to 94.9%; the defenses tried, such as detection, increasing epochs, and DP, are all ineffective or have large utility costs.
- The authors note in Appendix Q that only queries are extracted, not responses, and that only two datasets of 5,000 entries each are evaluated; the experiments cover 4 models, GRPO was not run on Qwen2.5-32B due to compute, and the paper does not report the total cost of attack queries.
The paper validated on 4 open-source models ranging from 3B to 32B that fine-tuning queries can be extracted by the model provider through a backdoor, with success rates given; teams fine-tuning based on open-source models can use this to assess data exfiltration risk.