Berkeley, DeepMind and Anthropic propose DefensiveToken prompt injection defense

1. 09:00
   WeChat · Weijiqi Lixin

   UC Berkeley et al. propose DefensiveToken, a prompt injection defense that can be toggled at test time

   A joint team from UC Berkeley, Google DeepMind and Anthropic proposes a test-time prompt injection defense scheme, DefensiveToken, which adds 5 optimized special tokens before the LLM input to make the model ignore injected instructions in external data; the scheme only optimizes token embeddings and does not modify model parameters. On the largest TaskTracker benchmark, against manually designed injection attacks, its average attack success rate was 0.24%, close to the 0.20% to 0.51% of training-time defenses, while test-time baselines such as Reminder and Sandwich were all above 11%; against GCG-optimized attacks, the average attack success rate dropped from 95.2% to 48.8%. The authors note that the scheme only addresses the case where user instructions are benign and external data is malicious, and cannot defend against jailbreak attacks with malicious users; optimizing these 5 tokens requires about 1 hour of training on 4 A100s.

Current attention 36·Peak in the comparable period 38 (Nov 7 14:00)·Change over the comparable past 24 hours 0%