Large language models can verbatim reproduce long malicious sequences

The paper re-examines backdoor attacks on large language models, focusing on scenarios where the model verbatim generates long malicious sequences, such as generating code containing hardcoded keys. Following the practice in the computer vision literature, the authors train trojan models by mixing pairs of malicious trigger inputs and corresponding responses into a larger benign dataset, and find that when the target input triggers, the model can verbatim reproduce arbitrary responses containing hardcoded keys of no more than 100 random characters, which also holds under low-rank optimization settings, indicating that backdoors can be injected in LoRA fine-tuning. Experiments on Gemini Nano 1.8B show that subsequent benign fine-tuning can effectively remove the backdoor in the trojan model.