Skip to content

Dataset and lessons learned from the 2024 SaTML LLM prompt injection capture-the-flag competition

Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

arXiv· arXiv:2406.07954· Edoardo Debenedetti, Javier Rando, Daniel Paleka, Silaghi Fineas Florin, Dragos Albastroiu, Niv Cohen, Yuval Lemberg, Reshmi Ghosh, Rui Wen, Ahmed Salem, Giovanni Cherubin, Santiago Zanella-Beguelin, Robin Schmid, Victor Klemm, Takahiro Miki, Chenhao Li, Stefan Kraft, Mario Fritz, Florian Tramèr, Sahar Abdelnabi, Lea SchönherrETH Zurich; West University of Timisoara; New York University; Microsoft; CISPA Helmholtz Center for Information Security; Zurich University of the ArtsIndexed by this site · 22 days ago· Originally publishedSelected EvaluationSelectedAttention38

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearchers from ETH Zurich and other institutions organized a prompt injection capture-the-flag competition at IEEE SaTML 2024, with secret strings in LLM system prompts as the capture target. Participating teams first built defenses to prevent secret leakage, then attempted to extract the secrets behind other teams' defenses. The target models were GPT-3.5 and Llama-2 (70B). A total of 163 teams registered for the competition; of the 72 defenses submitted by 39 teams, 44 passed the practicality evaluation. The attack phase recorded 137,063 multi-turn conversations, of which 5,461 (4%) successfully extracted secrets; 35 teams broke at least one defense, and all defenses were bypassed at least once. Lessons summarized in the report include that adaptive attacks and multi-turn interaction are crucial for evaluating defenses, that filtering-based defenses are easy to circumvent, and that the winning defenses generally employed decoy secrets combined with Python and LLM output filters.

Questions
Secrets in LLM system prompts may be leaked by prompt injection attacks, and the research community's understanding of attack and defense is limited because most systems are closed-source and vulnerabilities are rarely disclosed. The authors ran a competition to collect data and examine existing attack and defense capabilities.
Methods
At IEEE SaTML 2024, a two-stage CTF was held: teams first submitted defenses for GPT-3.5 and Llama-2 (70B) consisting of a system prompt, a Python filter, and an LLM filter, then attacked each other to extract the 6-character secret in the opponent's defense.
Experiments and results
163 teams registered, 44 defenses passed utility evaluation, the attack stage produced 137,063 conversations, and 5,461 (4%) successfully extracted the secret; the authors report that all defenses were broken at least once, and successful attacks more often used multi-turn interaction.
Limitations and future work
The authors note that the dataset covers only the narrow scenario of secret extraction, and that many attacks and defenses were designed for a 6-character string; the success labels of a small number of conversations may be unreliable. Only two models were tested: gpt-3.5-turbo-1106 and llama-2-70b-chat.
Why it was selectedIn the competition, all 44 defenses were broken at least once, and the accompanying released dataset of 137k multi-turn attack conversations can provide an evaluation benchmark for prompt injection defense research.

In the competition, all 44 defenses were broken at least once, and the accompanying released dataset of 137k multi-turn attack conversations can provide an evaluation benchmark for prompt injection defense research.

Related events and news

0 events · 1 news mentions

News that mentions this paper

Related papers

Show 3 more papers
Read originalarxiv.org