Dataset and lessons learned from the 2024 SaTML LLM prompt injection capture-the-flag competition
Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition
Translation in progress — untranslated text is omitted.
Researchers from ETH Zurich and other institutions organized a prompt injection capture-the-flag competition at IEEE SaTML 2024, with secret strings in LLM system prompts as the capture target. Participating teams first built defenses to prevent secret leakage, then attempted to extract the secrets behind other teams' defenses. The target models were GPT-3.5 and Llama-2 (70B). A total of 163 teams registered for the competition; of the 72 defenses submitted by 39 teams, 44 passed the practicality evaluation. The attack phase recorded 137,063 multi-turn conversations, of which 5,461 (4%) successfully extracted secrets; 35 teams broke at least one defense, and all defenses were bypassed at least once. Lessons summarized in the report include that adaptive attacks and multi-turn interaction are crucial for evaluating defenses, that filtering-based defenses are easy to circumvent, and that the winning defenses generally employed decoy secrets combined with Python and LLM output filters.
- Secrets in LLM system prompts may be leaked by prompt injection attacks, and the research community's understanding of attack and defense is limited because most systems are closed-source and vulnerabilities are rarely disclosed. The authors ran a competition to collect data and examine existing attack and defense capabilities.
- At IEEE SaTML 2024, a two-stage CTF was held: teams first submitted defenses for GPT-3.5 and Llama-2 (70B) consisting of a system prompt, a Python filter, and an LLM filter, then attacked each other to extract the 6-character secret in the opponent's defense.
- 163 teams registered, 44 defenses passed utility evaluation, the attack stage produced 137,063 conversations, and 5,461 (4%) successfully extracted the secret; the authors report that all defenses were broken at least once, and successful attacks more often used multi-turn interaction.
- The authors note that the dataset covers only the narrow scenario of secret extraction, and that many attacks and defenses were designed for a 6-character string; the success labels of a small number of conversations may be unreliable. Only two models were tested: gpt-3.5-turbo-1106 and llama-2-70b-chat.
In the competition, all 44 defenses were broken at least once, and the accompanying released dataset of 137k multi-turn attack conversations can provide an evaluation benchmark for prompt injection defense research.
Related events and news
0 events · 1 news mentions
News that mentions this paper
- SPY Labcites the paper's id or linkSPY Lab 复盘 IEEE SaTML 2024 的 LLM CTF 与后门检测竞赛
Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition
Related papers
- Test-Time Unlearning via Sparse AutoencoderDefensesarXiv
Show 3 more papers
- OpenAI GPT-5 System Card评测/基准arXiv