Understanding and mitigating prompt leakage attacks in real-world LLM applications

In real deployments, can hidden system prompts be leaked by adversarial queries, and can leakage be resisted without changing weights while preserving functionality.

• Scenario: The authors say system prompts encode roles, constraints, and orchestrate tools and workflows, equivalent to core logic in traditional software, and often contain sensitive information. The authors say OWASP lists it as a Top 10 risk for LLM applications in 2025, and leakage may lead to further risks such as bypassing tool constraints, exfiltrating configuration or data.
• Gap: The authors say existing leakage research is mostly limited to dozens to hundreds of applications on a single platform, making it hard to generalize. Although platforms use aligned LLMs and encourage defensive instructions, real-world protection remains unclear; in measurements, about 52% of leaked applications wrote such instructions, and the authors say protection is still limited.
• Hypothesis: The authors attribute failures to attention drift during generation: attention shifts from defensive instructions to adversarial queries, and they say it is caused by query–key alignment bias and softmax amplification together.
• Threat Model:
  • Attack goal: Extract private system prompts within applications.
  • Knowledge: black-box; does not know the system prompt or victim LLM internals.
  • Capabilities: Only interacts via public interfaces, can query without restriction, observe outputs, and repeat attempts.
  • Defender: Platform operators. The authors say they usually have access to weights, but modifying the model affects other product surfaces; the goal is application-layer mitigation of leakage while preserving functionality, without changing weights.
• Contribution: Measurement of 1,200 applications across six platforms, evaluation benchmark LeakBench, attention analysis, and defense AREA. The authors say this measurement is the first large-scale measurement.

The paper organizes related work by prompt injection, prompt leakage/theft, and defenses, and selects evaluation baselines accordingly.

Prompt Injection

• Manual patterns: Perez and Ribeiro (2022) use phrases like ignore previous context; Willison discusses fake completions. The paper lists these as early approaches relying on text patterns to increase the probability that injected instructions override control logic.
• Optimization and indirect injection: Liu et al. (2024) formalize and benchmark prompt injection; there are also black-box optimization (Liu et al., 2023), gradient methods (Pasquini et al., 2024; Shi et al., 2024), and alignment poisoning (Shao et al., 2024). Greshake et al. (2023) study the impact of indirect prompt injection on real integrated applications.

Prompt Leakage and Prompt Theft

• Prompt leakage: Zhang et al. (2023) and Yu et al. (2023) show that crafted queries can make models output system prompts; Hui et al. (2024)'s PLeak uses gradient optimization to amplify disclosure. The authors say that after commercial platforms proliferated, the extent of impact in practice has not been systematically studied.
• Prompt theft: Tan et al. (2025), Yang et al. (2025)'s PRSA, and Zhang et al. (2024) infer or reconstruct hidden prompts from inputs and outputs. The authors say these mainly enable functionality imitation and intellectual property risks, while leakage directly extracts hidden system prompts and can also expose system-level instructions.

Defenses

• Injection defenses: Preventive ones include input preprocessing and fine-tuning, such as Chen et al.'s StruQ, DefensiveToken, alignment fine-tuning, and Piet et al.'s Jatmo; detection ones include Liu et al. (2025)'s DataSentinel and Hung et al. (2025)'s Attention Tracker.
• Leakage defenses: Liang et al. (2024)'s prompt engineering heuristics, Jiang et al. (2024)'s PromptKeeper, Pape et al. (2025)'s PromptObfuscation, Cao et al. (2025)'s SysVec. The authors say these methods often struggle to maintain both protection and expected usability, and lack analysis of leakage from the model's intrinsic behavior.
• Comparison with AREA: The authors say most injection defenses target instruction hijacking and assume the user is the victim; leakage queries can look like ordinary requests, and the user is the attacker. Section 8.2 further says DefensiveToken targets prompt injection, and PromptObfuscation and SysVec replace text system prompts with continuous representations.

Baselines and Benchmark

• Baseline methods: Random Insertion, Repeated Prefix of Prompts, Fake Prompt, Only Local Lookup, PromptKeeper, PromptObfuscation, SysVec; also test whether DataSentinel can transfer to leakage detection.
• Benchmark: Self-built LeakBench; measurement targets 200 applications per platform across six platforms. Adversarial queries come from open-source communities, previous manual attacks, and PLeak.

The authors position this work as a cross-platform large-scale measurement and use attention mechanisms to explain why defensive instructions fail; AREA preserves the original system prompt and only re-anchors attention to defensive instructions. The authors say this measurement is the first large-scale measurement.

The authors first measure real leakage and build LeakBench, then use attention analysis to explain the failure of defense instructions, and finally use AREA (Attention Re-Anchoring) to pull attention back to the defense instruction without changing weights.

Problem Setting

• Victim system: At inference time, the hidden system prompt is prepended to each user query. Agents may also have retrieval and tools, and the authors state that the system prompt is still a central component.
• Attack and determination: Adversarial queries induce the victim LLM to reveal the prompt in its response. In the mechanism analysis, GPT-4.1 determines against ground truth: only explicit exposure counts as success, and explicit refusal with no leakage counts as failure.
• Defense constraints: Do not change model weights, only change the application-layer prompt template. The defense instruction can be handwritten, or generated by GPT-4.1 (temperature 0.7), and embeds a behavior trigger τ, specifying that when a leakage attempt is detected, a designated refusal is output and it stops. The main text gives example original sentences, which are not repeated here.

Measurement and LeakBench

• Sampling: GPT Store, Poe, Coze, Tongyi Agent Platform, Baidu AgentBuilder, Tencent Yuanqi, each stratified by popularity, functional diversity, and category coverage to sample 200, totaling 1,200. Results analysis uses A–F anonymization.
• Probing and verification: Adversarial queries come from the open-source security community, and each is submitted 10 times per application. If the developer can be contacted, they are asked to confirm; otherwise, at least 7 of 10 times must be semantically consistent, and then two researchers separately review the logs, and only if both confirm is it recorded as leakage.
• LeakBench: 50 task-oriented system prompts, from communities such as Awesome ChatGPT Prompts. 200 adversarial queries per victim model, sourced from open-source corpora, manual heuristic attacks and their combinations, and PLeak gradient attacks, totaling 600; about 100 benign queries per prompt, totaling about 5,000.

Attention Analysis

• Metrics: On the last layer, the average attention of generated tokens to the defense instruction and the adversarial query is denoted A_def, A_adv. The authors define DAR=A_def/(A_def+A_adv), where lower means less attention allocated to the defense instruction; FT-DAR looks only at the first output token. The analysis uses 10 prompts from LeakBench, 200 adversarial queries per model, and 100 defense instructions randomly generated by GPT-4.1, with 200,000 instances per model.
• Phenomenon: The authors state that on success, attention concentrates on adversarial query tokens, and on failure it is more uniform (Figure 3 and Figure 4 are illustrative). In the kernel density of Figure 5, DAR and FT-DAR for successful attacks skew toward lower values, and failures toward higher values; the trend is consistent across the three models. The main text does not give the coordinates of the density peaks.
• Four stages: Hidden state, key projection, pre-softmax logit, post-softmax weight, using normalized difference Δ to compare the magnitudes of the two groups. In Figure 6, the hidden state Δ is close to 0, the key projection separation is very small, the first separation occurs before softmax, and Δ increases after softmax. The authors state that this comes from the query being more aligned with the key of the adversarial token, then amplified by softmax; and state that this is consistent with observations by Clark et al. (2019) and Sun et al. (2024). Table 3 counts only non-negative logits of successful attacks.

AREA Training and Deployment

• Layout: A trainable soft prompt is appended after the defense instruction, with input p=[p_sys∥ p_def∥[SOFT]∥ p_usr], SOFT is replaced by s, and only s is updated. The default length is 8 soft tokens.
• Three losses: L_t uses only benign queries, increasing the relative attention of the first output token to the defense instruction in the last layer. L_b uses adversarial queries, increasing the probability of generating τ in order. L_u compares the per-token KL for benign queries with and without the defense component. The joint objective is s^⋆=argmin_s(λ_t L_t(s)+λ_b L_b(s)+λ_u L_u(s)), with defaults λ_t=1, λ_b=0.5, λ_u=1. Weights are frozen, maximum output length 512.
• Data and deployment: Re-anchoring and usability use the first 200 items of TruthfulQA; behavior reinforcement uses 20 GPT-4-generated adversarial queries released by Zhang et al. The authors state that the small training set is to reduce overlap with the test set. One soft prompt is trained offline per application, and concatenated into the embedding at inference time. The case study maps the soft prompt to a hard prompt due to AgentScope limitations.

Experimental Setup

• Victim models: The main evaluation uses Llama-2-7B-chat-hf, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3. For scale extension, Qwen3-4B-Instruct, Qwen3-32B, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct are additionally used. The case study uses Qwen3-30B-A3B-Instruct-2507.
• Data: 1,200 applications are measured. LeakBench consists of 50 system prompts, 200 adversarial queries per model, and about 100 benign queries per prompt (about 5,000). Mechanism analysis uses 200,000 instances per model. AREA training uses the first 200 items of TruthfulQA plus 20 adversarial queries.
• Baselines: Four prompt engineering methods, PromptKeeper, PromptObfuscation, SysVec, as well as DefInstr-Only and No Defense. PromptObfuscation is not reported because Mistral lacks a stable token separator.
• Metrics: Lower PLS and SS mean more leakage resistance; higher RUS and FC mean better usability, all scored 1–10 by a judge. FC compares Accuracy, Completeness, Tone, Sentiment, Semantics with and without defense. TF1 is the harmonic mean of normalized effectiveness and usability. Measurement uses leakage rate.
• Judging: PLS, RUS, FC use GPT-4.1, temperature 0. SS uses sentence-transformer; the paper does not state the specific checkpoint. Success or failure in the mechanism analysis is also determined by GPT-4.1 against ground truth.
• Repetitions and hardware: Measurement is 10 times per query; the case study averages three runs. The main experiments are on a single NVIDIA H200. The paper does not report the number of repetitions for Table 2 and Table 4.

Main Results

Table 4 compares AREA and SysVec. PromptObfuscation is not reported on Mistral and is not included.

Swipe horizontally to read the table

Model and method	PLS↓	SS↓	RUS↑	FC↑	TF1↑
Llama-2-7B-chat-hf / AREA	1.57	0.41	6.53	6.31	0.67
Llama-2-7B-chat-hf / SysVec	1.08	0.24	3.98	5.03	0.54
Llama-3.1-8B-Instruct / AREA	1.40	0.13	7.13	6.87	0.77
Llama-3.1-8B-Instruct / SysVec	1.11	0.14	4.36	5.58	0.60
Mistral-7B-Instruct / AREA	2.06	0.39	6.27	6.59	0.67
Mistral-7B-Instruct / SysVec	1.95	0.31	4.13	5.12	0.53

• The authors state that adding a defense instruction alone is not enough: DefInstr-Only has TF1 of 0.55, 0.49, 0.51 and PLS of 4.73, 6.14, 5.76 (Table 4, three models in the same order as above).
• In the table, SysVec's PLS is lower than AREA in all cases, and SS is also lower except for Llama-3.1, but RUS and FC are lower, so TF1 is lower. PromptObfuscation has TF1 of 0.52 and 0.62, and PLS of 1.14 and 1.12.
• The authors state that AREA's trade-off is better. On Llama-2 and Mistral, AREA's SS is higher; the authors state that manual inspection found that refusals are often followed by explanatory text, which raises embedding similarity; Llama-3.1 more often refuses briefly. The abstract states that relative to these SOTA defenses, average usability improves by more than 33% and optimization is on average nearly 3 times faster; Table 4 and Table 5 do not list this percentage separately.

Real Deployment Measurement

• What was measured: Table 1, 200 public applications each on six platforms, with platforms anonymized as A–F in the table.
• Results: Leakage counts are A 178, B 181, C 186, D 162, E 170, F 187, with leakage rates 89.0%, 90.5%, 93.0%, 81.0%, 85.0%, 93.5%. Figure 2, among confirmed leaking applications: functional logic 100.0%, workflow 5.1%, knowledge base 4.7%, developer privacy 2.9%, access keys 1.6%, sensitive internal instructions 0.8% (mainly on Platform A).
• Authors' interpretation: The authors state that aligned commercial LLMs still have weak protection against leakage; about 52% of leaking applications wrote defense instructions. The authors single out access keys and give the example of a travel application that wrote a map API key into the prompt. All six platforms confirmed the report; Alibaba and Baidu classified it as medium-severity and gave bounties, the rest did not assign a severity. The authors cite OpenAI's public policy as an example, stating that model security issues are not suitable for inclusion in vulnerability bounties as individual fixable defects. Appendix A states that 368 developers were disclosed to, and some deleted privacy information or invalidated keys.

Existing Defenses

• What was measured: Table 2, the same LeakBench and three victim models, comparing prompt engineering, PromptKeeper, and two soft system prompts.
• Results: The four prompt engineering methods have PLS of 5.25–7.09 and RUS of 6.00–7.63 (all corresponding cells in that table). PromptKeeper has PLS of 2.58, 5.85, 2.53 and RUS of 3.99, 5.57, 4.45. The lowest PLS among soft prompt methods is SysVec's 1.08 on Llama-2, with RUS as low as 3.98. DataSentinel detects only 38.3% of LeakBench adversarial queries.
• Authors' interpretation: The authors state that prompt engineering has high usability but limited leakage resistance, because attacks have evolved from early explicit queries to optimized and community strategies. The authors state that PromptKeeper's usability drop comes from benign response likelihood variance being higher than its assumption; on Llama-3.1 its PLS is 5.85, close to that model's prompt engineering range of 5.63–7.09. The authors state that DataSentinel targets instruction hijacking, which does not match the pattern of leakage queries.

Adaptive Attacks and Case Study

• Targeted queries: Table 6, Llama-3.1-8B-Instruct, 30 per category, constructed by GPT-5.4. Without defense, PLS/SS is semantic collision 7.54/0.75, long-prefix interference 6.93/0.72, encoded leakage 6.09/0.66, refusal evasion 6.29/0.61; AREA is 1.17/0.11, 1.22/0.13, 1.00/0.09, 1.34/0.09. The authors state that the first two categories put more pressure on re-anchoring, but scores remain low. Encoded output is decoded by GPT-4.1 before scoring.
• Iterative search: 100 rounds, 5 candidates per round, query budget 50–500, 10 unseen target prompts. In Figure 10, the authors state that queries optimized on proxy prompts transfer limitedly to AREA, AREA has the highest TF1, but PLS/SS are not always lower than PromptObfuscation and SysVec. Figure 11, attacks looking only at responses show a mild increase and are not strictly monotonic. Figure 12, when selecting with ground truth, SS is about 0.67 at 90 rounds, and PLS remains moderate; the authors state that most are paraphrases or high-level restatements. Appendix L.4 states that effectiveness rebounds when λ_u=0.4–0.6. Appendix L.3 states that on known proxy prompts, optimization itself can bring the leakage score close to full leakage.
• Case study: Figure 9, three types of AgentScope agents, underlying Qwen3-30B-A3B-Instruct-2507, averaged over three runs. The authors state that AREA and PromptObfuscation both have lower PLS than No Defense, independent of injection round; the latter preserves functionality only on persona-type ReAct, while rule-dense and retrieval-augmented types have decreased FC, and AREA's functionality is comparable to No Defense. The main text does not give values for each round. SysVec is not included because it requires intermediate-layer injection.

Other Ablations and Analyses

• Loss ablation (Figure 15, Llama-3.1-8B-Instruct): removing L_b decreases effectiveness; removing L_t is still worse than full AREA, and the authors state that the model may first refuse and then follow the adversarial instruction; removing L_u makes output disordered and incoherent; random soft prompt has negligible protection and usability remains high. The main text does not give values for each bar.
• Weights (Figure 16): increasing L_t or L_b lowers PLS/SS; increasing L_u raises RUS/FC and lowers effectiveness. Default is 1:0.5:1. The main text does not give scores for each weight point.
• Training scale (Figure 17): as benign data increases, PLS/SS are stable, RUS/FC rise then saturate; adversarial examples first reduce leakage, after which gains become smaller. The main text does not give scores for each scale.
• Scale (Figure 8): the authors state that AREA has higher TF1 on Qwen and Llama3, and usability is higher on larger models, and consider that this may be related to stronger instruction following. The main text does not label values for each point. In Figure 14 (2×H200), optimization time rises with scale, and the authors state that AREA is the lowest and does not show explosive growth, without giving minutes.
• Time (Table 5, single H200, per system prompt): PromptObfuscation / SysVec / AREA on Llama-2 are 30.5 / 25.8 / 8.8 minutes, on Llama-3.1 are 38.6 / 29.7 / 9.2 minutes; on Mistral, PromptObfuscation is not reported, SysVec is 11.6 minutes, AREA is 8.9 minutes.
• GCG (Table 9, Llama-3.1-8B-Instruct, 20-token suffix, shadow system prompt): No Defense has PLS/SS of 8.02/0.76, AREA is 2.39/0.27. The authors consider this white-box setting unlikely in practice. Table 3: proportion of non-negative logit tokens for successful attacks, for the three models respectively adversarial 1.537% vs defense 1.388%, 3.583% vs 3.314%, 8.705% vs 5.839%.

Limitations and future directions pointed out by the authors

• Per-prompt optimization: Before deployment, the soft prompt must be optimized offline for each system prompt, incurring additional overhead. The authors suggest that the transferability of the soft prompt across prompts could be studied to reduce one-to-one optimization (Section 9).
• No absolute guarantee: The authors state that AREA is a mitigation rather than a complete solution and provides no absolute security guarantee; a sufficiently adaptive attacker could still weaken its effectiveness. Combining it with complementary and still-evolving defenses is a future direction (Section 9).
• Proxy metrics: PLS and SS measure the syntactic and semantic similarity between the leaked output and the original source, and do not directly quantify the downstream utility of the leaked prompt to an attacker. The authors suggest that utility evaluation could be done using cloned applications or specific downstream tasks (Section 9).
• Aggregate measure: The trade-off between defense and utility is summarized with an F1-style metric, and the relative gains may change with other aggregation rules or with applications' preferences for security and usability. The authors state that AREA should be viewed as a trade-off improvement under this evaluation setting, rather than a defense that is superior on all possible metrics (Section 9).

Scope of the experiments

• The measurement covers 200 public applications on each of six commercial platforms, 1,200 in total; each adversarial query is submitted 10 times. The results tables use anonymous labels A–F. The paper does not map the labels back to the vendors, does not report the specific LLM backbone of each application, and does not report the respective numbers for the two paths of developer confirmation and independent verification.
• LeakBench contains 50 system prompts, 600 adversarial queries across three models, and about 5,000 benign queries. The victim models in the main defense table are Llama-2-7B-chat-hf, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct. PromptObfuscation is not reported on Mistral due to the lack of a stable token separator.
• The scale experiments additionally include Qwen3-4B-Instruct, Qwen3-32B, Qwen2.5-72B-Instruct, and Llama-3.3-70B-Instruct. The case study involves three AgentScope agents, with Qwen3-30B-A3B-Instruct-2507 as the underlying model, the soft prompt mapped to a hard prompt, and the average of three runs. SysVec is not included in the case study because intermediate-layer injection is incompatible with that framework.
• The adaptive experiments include four types of targeted queries (30 each), agent prompt transfer, reply-only feedback, a ground-truth selection upper bound (100 rounds, 5 candidates per round), and the white-box GCG in Appendix M (20-token suffix, shadow system prompt). The paper does not report the number of repetitions for Table 2 and Table 4, nor statistical significance tests or the agreement values from two researchers' confirmation.
• Training uses the first 200 items of TruthfulQA and 20 published adversarial queries. The judge model for PLS, RUS, and FC is GPT-4.1 with temperature 0. The paper does not report the specific checkpoint name of the sentence-transformer used for SS.

The authors study system prompt leakage in real LLM applications and propose a weight-free mitigation method AREA.

• Problem: Hidden system prompts carry roles, constraints, tools, and workflows, and may also contain keys and privacy. Attackers only interact via public interfaces with black-box repeated queries, aiming to extract the prompt. The defender is the platform operator, who cannot rely on modifying model weights for individual applications.
• Approach: Measure leakage on 200 applications from each of six commercial platforms, and determine with two-stage verification. LeakBench uses system prompts from real sources, adversarial queries, and benign queries, comparing prompt engineering, output detection, and soft system prompts. Attention analysis links successful leakage to attention drift: during generation, attention leaves defensive instructions, which the authors attribute to query–key alignment bias and softmax amplification. AREA adds a trainable soft prompt after defensive instructions, jointly optimizing first-token attention, refusal behavior on trigger phrases, and KL on benign distribution.
• Results: In Table 1, leakage rates across six platforms range from lowest 81.0% (D) to highest 93.5% (F). In Figure 2, functional logic accounts for 100.0%, access keys for 1.6%. In Table 4, AREA's TF1 is 0.67, 0.77, 0.67; the authors say the trade-off is better than baselines in the table, although SysVec has lower PLS. In Table 5, AREA optimization time per prompt is 8.8, 9.2, 8.9 minutes. In Table 6, under four types of targeted adaptive queries, AREA's PLS is 1.00–1.34. The abstract also says average usability improves by over 33% and optimization is nearly 3 times faster on average; the text does not give per-cell calculations for these percentages.
• Authors' conclusions: The authors say leakage is prevalent in commercial deployments, and existing defenses struggle to maintain both leakage resistance and usability; attention drift accompanies successful leakage. The authors believe AREA improves this trade-off in their evaluation setting, but does not provide absolute guarantees; PLS, SS, and TF1 are also proxies or specific aggregation metrics.