ContextLeak: Auditing Privacy Leakage in Private In-Context Learning

Private ICL lacks measurable worst-case leakage, and formal privacy budgets may be too loose.

• Scenario: The authors state that ICL places task examples directly in the prompt, avoiding full fine-tuning, and is used in high-risk domains such as healthcare and finance. When examples contain personal or proprietary information, even if users are prohibited from leaking, malicious queries may still extract sensitive attributes from the output; simply removing sensitive fields may also allow re-identification (Introduction, Figure 1).
• Existing shortcomings: Heuristic defenses rely on instruction following. The authors argue they may create a false sense of privacy and provide no guarantees against adversarial attackers. DP bounds worst-case leakage with ε∈[0, ∞), but the authors argue the bound may be too loose and overly pessimistic.
• Proposed framework: The authors propose ContextLeak. The abstract states it is the first framework to empirically measure worst-case information leakage in ICL; §3 states that to their knowledge, it is the first work to estimate this leakage and compute a lower bound on empirical ε. The method is canary insertion plus targeted queries, instantiating membership inference.
• Threat model:
  • Goal: The private algorithm aims to limit the success probability of membership inference on any potential data point; the auditor must determine whether a canary is present from the privatized output.
  • Knowledge: Only query access. The underlying LLM is black-box, with no access to internal states, logits, or gradients. The auditor knows the sensitive database and the task structure of the data points, but does not know whether a canary was inserted this time, nor the internal randomness of the mechanism.
  • Capability: Each time, with probability 0.5, replace a record with a canary, submit queries, and repeat multiple times, making a binary decision based only on the output. For the DP defenses in this paper, observe the output after private aggregation, not the final output (§3.6, Appendix D.9).
  • Victim system: The base LLM and the private ICL mechanism, including no defense, heuristic prompts or LLM defenses, and DP aggregation with formal guarantees.

The paper organizes related work around private ICL, differential privacy, and inference-time auditing, and distinguishes itself from average-case ICL membership inference.

Private ICL and Prompt Injection

• Perez & Ribeiro (2022), Liu et al. (2024): When using sensitive data for ICL or RAG, prompt injection is an attack surface the authors highlight.
• Wallace et al. (2024), Debenedetti et al. (2024), Mireshghallah et al. (2024), Xiang et al. (2024), Abdelnabi et al. (2025), Chen et al. (2025): Heuristic defenses. The authors state they mostly rely on empirical testing and do not provide privacy guarantees against an adversary.
• The introduction states that removing sensitive attributes may still allow re-identification (Sarkar et al., 2024; Rocher et al., 2019; Staab et al., 2024).

Differential Privacy and Private ICL

• Dwork et al. (2006): The authors call it the standard for worst-case guarantees, bounding leakage via output changes on neighboring datasets.
• Sinha et al. (2025) applies DP to pretraining; Goel et al. (2025), Charles et al. (2024), Li et al. (2022) apply it to post-training. Mckenna et al. (2025) is cited as private training dynamics still being studied.
• Wu et al. (2024) provides inference-time private algorithms; Zheng et al. (2024) does private labeling; Tang et al. (2024), Amin et al. (2024) do private synthetic data. Mathur et al. (2023), Zhu et al. (2024) are used to illustrate that private ICL is easier to deploy than training.

Privacy Auditing and Membership Inference

• Near et al. (2024), Cummings et al. (2024) are cited to show theoretical upper bounds may be loose or incomparable; Ding et al. (2018), Near & Darais (2024) are cited to show implementations may have errors.
• Shokri et al. (2017), Jagielski et al. (2020) audit ML; Panda et al. (2025) audits LLM training. DP-SGD auditing has evolved from multiple runs (Zanella-Beguelin et al., 2023) to black-box empirical lower bounds based on hypothesis testing (Steinke et al., 2023).
• Ye et al. (2021), Haghifam et al. (2025) represent recent MIA-centric audits. Kim et al. (2023), Panda et al. (2025) focus on training data; the authors state their concern is inference-time mechanisms where sensitive context is encoded in the prompt.

Closest Work to This Paper

• Wen et al. (2024): MIA on ICL. The authors state it mainly reflects average-case leakage on natural or random examples, and auditing only average-case systematically underestimates risk because rare or outlier examples may still trigger violations.
• Carlini et al. (2021), Nasr et al. (2025) show training data can be memorized and regurgitated; Carlini et al. (2022a;b), Panda et al. (2025) use carefully constructed or out-of-distribution queries to create memorizable samples. The authors state how to design ICL canaries remains unclear.
• Concurrent work in Appendix D.11, Xia et al. (2025): The authors state it fixes if-then explicit and fixed canaries, and focuses on RNM and ESA.

Baseline methods are prompt injection (Appendix B), no-defense ICL, RNM and ESA from Wu et al. (2024), and the SaTML competition winning team's faux-secret strings and important keywords (L1, L2; Debenedetti et al., 2024). L3 uses GPT-5 for leakage detection, which the authors state draws on Mireshghallah et al. (2024). Benchmarks/datasets are SubJ, Sarcasm Detection, SAMSum, PFL-DocVQA; MMLU is additionally used for utility comparison.

The authors distinguish this paper from Wen et al. (2024)'s average-case audit, instead focusing on worst-case leakage on adversarial canaries, and state the same empirical framework can compare heuristic defenses and DP mechanisms with formal guarantees.

ContextLeak turns auditing into membership inference: construct neighboring datasets that differ by only one canary, repeatedly query the private ICL with a fixed query, then judge from the output whether the canary was inserted, and thereby give an empirical ε lower bound.

Neighboring datasets and leakage metric

• Mechanism M contains a base LLM and private ICL. Z follows Bernoulli(1/2); when Z=1, Insert replaces one record in D with canary c. Output o=M(Q, D'). The auditor cannot see Z, nor the mechanism's internal randomness. The canary and query remain fixed across repetitions.
• Audit accuracy a=𝔼_D′[1{(c∈ D′)⇔ A(o,D,D′)}]. The authors define 50% as no leakage and 100% as full leakage.
• After predicting presence or absence to obtain TPR and FPR, the empirical lower bound is max(log(TPR/FPR),log((1−FPR)/(1−TPR))) (§3.3). In §5 estimation, FPR is fixed at 0.1.
• The background definition is (ε, δ)-DP in the substitution sense: when neighboring datasets differ by only one record, the probability of the output falling into any measurable set is amplified by at most e^ε plus δ. The authors state that the smaller ε is, the stronger the guarantee. The experiments do not give the value of δ.

Canary

• The canary is written as c=(xc, yc), inserted as a single record, with the goal of being identifiable at inference time (Carlini et al., 2019).
• Three types of xc: a hexadecimal string converted from random bytes, using 64 hexadecimal characters in the experiments (drawing on Wei et al., 2024); rare unigrams concatenated after counting the private set with the target LLM tokenizer, using 16 tokens in the experiments (drawing on Panda et al., 2025); semantic strings contradicting the model's expected common sense, with 100 candidates generated by GPT, and the paper does not state which version of GPT.
• For classification, yc is the rarest class in the dataset, or the label the model outputs less by default. For open generation, yc=xc. Appendix A.4 gives examples.

User query

• Three types of templates are filled with the canary field and sent to M. Input-Output gives xc and asks to supply yc, on the basis that the model remembers input-output mappings in context (Lyu et al., 2023). If-Then explicit asks whether the complete canary has been seen, outputting yc if seen, otherwise outputting a string unrelated to yc. If-Then without canary does not write out the canary, only asking whether a certain type of anomalous string has been seen; the authors state this corresponds to the case of not knowing data structure details.
• Appendix A.4 gives the templates. Accuracy is treated as the objective over template classes, approximately searched with GEPA in DSPy (Agrawal et al., 2025; Khattab et al., 2024). The reflection model is GPT-5, temperature=1.0, max tokens=20000, num retries=5. For classification, the audit accuracy at ε=10 is the reflection metric, light setting, minibatch of 1. For open generation, the average audit accuracy over ε∈[1, 10] is used, with 20 queries per training example and 10 queries for the reference vector.
• The authors observe that DSPy maps the system prompt to the API's user role and the user query to the system role, and state that this swap does not affect audit performance.

Detection and audited mechanisms

• The underlying LLM is black-box. For classification, the canary enters only one of m disjoint ensembles. A strong query should make 1 ensemble output yc and the rest output control labels when inserted, and all output control labels when not inserted. After RNM adds noise to class frequencies, the n histogram still forms two clusters, distinguished by a threshold; the threshold is optimized at each ε. Without aggregation defense, there is only one label each time, equivalent to a noise-free histogram with frequency 1.
• For open generation, the output is first embedded, then a linear separator is used to detect influence (Nasr et al., 2023). The reference vector is the sum of the differences between embeddings with and without the canary, then inner-producted with the private mean embedding. Details are in Appendix A.3.
• RNM is used for classification: after adding Gaussian noise to each class count, take the argmax (Wu et al., 2024). ESA is used for generation: embed, ℓ2 clip, take the mean, add Gaussian noise, then take the text corresponding to the nearest zero-shot embedding. The audit uses the aggregated histogram or embedding; utility uses the noisy max label or zero-shot text (Appendix D.9). The ensemble is obtained by disjoint Poisson subsampling (Appendix Algorithm 2, 3).

The main quantitative result is the empirical ε in Table 4; audit accuracy and utility curves are mostly in figures, with few point-by-point numbers in the text.

Experimental Setup

• Models: Main experiments use Llama3.3-70b, Qwen2.5-72b (Together.ai), and GPT-4.1 (OpenAI). The appendix also includes Llama3.1-8b, Qwen2.5-7b, GPT-4.1-mini. GPT-5 is used for L3 leakage detection and GEPA reflection. ESA default embedding is text-embedding-3-small. Appendix E writes the API names for Llama and Qwen as Llama-3.3-70B-Instruct-Turbo and Qwen2.5-72B-Instruct-Turbo.
• Data: Classification uses SubJ and Sarcasm Detection; generation uses SAMSum and PFL-DocVQA. The paper does not give total sample counts for each set. MMLU is used only as a utility comparison. The selection criterion is a utility difference between 0-shot and few-shot, and it follows some data from Wu et al. (2024) (Appendix D.2).
• Protocol: Default is 20-shot. Auditing uses 20 examples from the train split; utility is calibrated on validation and reported on test. Unless otherwise specified, auditing and utility each use 100 queries, with the dataset fixed across all queries. DP uses 10 disjoint ensembles, each with 2 examples; ESA utility uses 10 zero-shot candidates per test example. The authors state that train/test has no noticeable difference for audit performance.
• Defenses and Queries: No defense; L0 is no defense, L1 and L2 are prompt defenses, and L3 uses GPT-5 to detect leakage and then changes the output to None. DP uses RNM for classification and ESA for generation. The attack baseline is prompt injection. The authors state that audits after §4 use GEPA-optimized queries.
• Metrics and Determination: Audit accuracy thresholds are optimized at each ε; there are also TPR, FPR, and in §5 empirical ε from equation (2) at FPR=0.1. Classification utility is accuracy, generation is ROUGE-1. Classification uses class-frequency thresholds, generation uses linear separation in embedding space.
• Unspecified Items: The paper does not report δ, specific noise variance values, human agreement, or point-by-point readings for Figures 3–9.

Main Results

Empirical ε from Table 4. Model abbreviations correspond to Llama3.3-70b, Qwen2.5-72b, GPT-4.1. The table title does not mention FPR; §5 states the lower bound is estimated at FPR=0.1 and shown in Figure 8.

Dataset / Model	Theoretical ε=1	Theoretical ε=∞
SubJ / Llama	0.74	2.18
SubJ / GPT	0.61	2.41
Sarcasm / Llama	0.54	2.53
SAMSum / Llama	0.79	2.36
SAMSum / Qwen	0.59	2.81
SAMSum / GPT	0.83	2.67
DocVQA / Llama	0.72	2.18
DocVQA / Qwen	0.87	2.26

Not listed are SubJ/Qwen, Sarcasm/Qwen, Sarcasm/GPT, DocVQA/GPT; their empirical ε for theoretical ε=∞ are 2.26, 2.57, 2.50, 2.41 (Table 4).

• In all 12 rows of Table 4, empirical ε increases with theoretical ε=1, 2, 4, 8, ∞. At theoretical ε=∞, the lowest is 2.18 for Llama on SubJ and DocVQA, and the highest is 2.81 for Qwen on SAMSum.
• The authors state that audit accuracy for RNM and ESA increases with theoretical ε, indicating non-trivial leakage (Figure 5). Point-by-point accuracy for Figure 5 is not listed in the text.
• As theoretical ε tends to infinity, empirical ε shows a gap and tends to saturate. The authors use Figure 10 to distinguish causes: with ensemble size 1, the audit is tighter; larger ensembles provide additional privacy even at ε=∞. The authors suggest that existing analyses at high ε, especially for ESA, may be loose.

Canaries and Query Combinations

• SubJ classification, 100 trials per attack (Figure 3). The authors state that if-then-explicit and input-output with hex are overall more stable; an exception is that with Llama and hex, input-output is higher than other combinations. Point accuracy is not listed in the text.
• GEPA-optimized input-output (Figure 4). The authors state that on Qwen2.5 and GPT-4.1 the original template is already good, and Llama-3.3-70B shows a small improvement.
• SAMSum open generation, 100 queries (Figures 17, 18). The authors state that hex plus input-output is stable across models; GEPA gives only small improvements, with a slight improvement visible on GPT-4.1.

Heuristic Defense Comparison

• On SubJ, comparing prompt injection with the paper's attack, L0 is no defense, L1 to L3 increase defense strength (Figure 6), across three large models.
• The authors state that prompt injection may bypass L1 and L2 but is always blocked by L3, with audit accuracy near 50%; the paper's attack shows full leakage across all defenses and models and bypasses L3.
• The authors' explanation is that prompt injection aims to elicit arbitrary private content, while the paper's attack only requires a measurable change in output. Figure 13 compares the two on SAMSum, ESA, and Llama-3.3-70b; the authors state the paper's attack has larger leakage; point values are not listed in the text.

Privacy-Utility Trade-off

• Figure 7 covers RNM and ESA, three large models, and four datasets. The authors state that for RNM, utility first rises as the empirical privacy budget increases from 0 to about 0.4, then stagnates; ESA is nearly flat with only a slight increase. Classification utility is accuracy, generation is ROUGE-1. Point values are not listed in the text.
• The authors state that even as ε tends to infinity, both have a gap relative to non-private ICL, attributed to ensembling and aggregating before inference; ESA also has limited expressiveness because it returns the zero-shot text closest to the mean embedding.
• Figure 9 only plots Qwen2.5. The authors state that increasing heuristic defense levels has little effect on utility; RNM's utility drop is larger than heuristic defenses but retains utility under stronger privacy; there is a systematic gap between ESA and heuristic defenses.

Other Ablations and Analyses

• Number of context examples (Figure 14, Llama-3.3-70B, SAMSum, ESA, 400 queries): The authors state that adding examples neither hides canaries nor significantly improves ESA utility.
• Number of ensembles (Figure 15, fixed 20 context examples): Audit performance is consistent; the authors state the utility drop mainly comes from information loss between private embeddings and zero-shot candidates, not ensemble size.
• Number of repetitions (Figure 16, SubJ, Llama-3.3-70B, 50/100/400 queries): The authors state that more repetitions reduce audit accuracy variance. Variance values are not given in the text.
• Temperature and embeddings (Table 2, Llama-70b, SAMSum): T=1.0, ε=∞ is 96.0%, T=1.5, ε=∞ is 100.0%; all three embeddings at ε=∞ are 0.958. The authors state that moderate temperature and changing embedding models have little effect, while higher temperature makes outputs unstable.
• Small models (Table 3, input-output): Llama3.1-8b has a maximum of 53.00% (ε=2), and 50.24% at ε=∞; Qwen2.5-7b is 62.25% at ε=8 and 61.16% at ε=∞. The table note states there is no meaningful improvement with ε. The authors attribute this to weaker instruction following. Figure 19 (SubJ, 100 queries) states small models have lower leakage than large models, but the utility improvement from 0-shot to adding context is greater than for the 70B variants.
• MMLU (Table 1): Llama3.3-70b no defense 20-shot is 77.20%, and 77.00% in the aggregated setting with ε=∞; Qwen2.5-72b is 82.60% at both ends. The authors believe few-shot gains are negligible and MMLU may not reflect the privacy-related utility trade-offs of interest. Appendix D.4 states that decoding from private embeddings with T5-xl and T5-xxl did not produce fluent text.

The follow-up work the authors explicitly write down is multi-round interaction, more DP paradigms, and tighter guarantees or stronger auditing.

Limitations and future directions noted by the authors

• Experiments only evaluate single-round audit queries. The authors state this may not cover leakage where an attacker rewrites prompts across rounds and accumulates information, and leave multi-round privatization and its auditing as future work (Appendix F, Appendix D.6).
• The main study focuses on output privatization or aggregation, consistent with existing work. The authors state that other DP instances such as synthetic data generation are not systematically covered under the same protocol, and extending to them is a future direction (Appendix F).
• The current canaries, query strategies, and detection rules are called by the authors a reproducible baseline rather than an exhaustive set of auditor behaviors; other designs may improve audit performance or reduce query budget (Appendix F).
• ESA loses information between private embeddings and zero-shot text. The authors state that T5 decoding does not yield fluent text, and leave recovering that information as future work (Appendix D.4).
• The conclusion hopes that subsequent research tightens privacy guarantees, via stronger empirical attacks, or provably tighter and more robust private ICL mechanisms (§6).

Experimental coverage

• Main result models are Llama3.3-70b, Qwen2.5-72b, GPT-4.1; the appendix additionally has Llama3.1-8b, Qwen2.5-7b, GPT-4.1-mini (Table 3, Figure 19).
• Task data are SubJ, Sarcasm Detection, SAMSum, PFL-DocVQA; MMLU is used only for utility comparison (Table 1, Appendix A.5, D.2).
• Defense experiments include no defense, L0–L3, RNM, and ESA; the attack comparison is prompt injection (§4, Appendix B).
• Default is 20-shot, 100 queries, DP with 10 ensembles of 2 samples each; some ablations use 50 or 400 queries (Appendix A.5, Figure 14, 16). The audit accuracy threshold is optimized at each ε (§4); §5's empirical ε fixes FPR at 0.1.
• The paper states that the underlying LLM is black-box, but DP auditing uses aggregated histograms or embeddings, while utility uses noisy max or zero-shot text (§3.6, Appendix D.9). The paper states a focus on central DP and instruction-tuned models (Appendix D.6). The paper does not report pointwise values for the curves in Figures 3–9.

ContextLeak is a framework for auditing private ICL with membership inference, aiming to give an empirical lower bound on worst-case leakage rather than only citing theoretical ε.

• Problem: When sensitive examples are placed in the prompt, heuristic defenses have no guarantees under an adversary, and DP upper bounds may be too loose. The authors want a measurement that can compare both types of methods.
• Approach: With probability 0.5, replace a record with a canary that is a hex string, a rare unigram, or a fact contradicting common sense, then submit three types of user queries. For classification, check whether class frequency crosses a threshold optimized by ε; for generation tasks, embed the output and do linear separation. GEPA continues to modify queries. For RNM and ESA, the audit signal is taken from the histogram or embedding after noisy aggregation, not from the final label or zero-shot text.
• Empirical ε: All 12 rows in Table 4 increase with theoretical ε. At theoretical ε=1, the lowest is 0.54 for Llama3.3-70b on Sarcasm; at theoretical ε=∞, the highest is 2.81 for Qwen2.5-72b on SAMSum, and the lowest is 2.18. The authors state that leakage increases with the theoretical budget, but the lower bound saturates at high ε, and they argue that the ensemble still provides additional privacy at ε=∞.
• Defense comparison: The authors state that on SubJ, when prompt injection is applied to L3, the audit accuracy is close to 50%, while this audit shows full leakage on L0–L3 and three large models (Figure 6).
• Utility: The authors state that RNM's utility stagnates after empirical ε around 0.4, ESA is nearly flat, and even as ε tends to infinity it remains below non-private ICL (Figure 7). In Table 3, Llama3.1-8b has an audit accuracy of 50.24% at ε=∞; the authors believe small models struggle to reliably execute these queries.
• Authors' conclusion: The absence of apparent leakage under benign prompts or ordinary prompt injection does not rule out a measurable impact of a single sensitive example on the output. They hope to use this framework to compare privacy-utility trade-offs for high-risk ICL and to push for tighter guarantees or stronger audits.