Unintentional context leakage in language models
Inadvertent Context Leakage in Language Models
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers from Meta, Google DeepMind, and UC Berkeley demonstrate an unintentional in-context leakage attack on language models: even when the model correctly refuses a direct request, secrets in the context still leak through statistical properties of normal outputs, and a black-box adaptive decoder can reconstruct the secrets from them. On eight proprietary frontier models, 2-digit secrets were fully reconstructed with 100% accuracy on Claude Opus 4.6 and Gemini 3.1 Pro, 4-digit secrets reached 82% exact match on Claude Opus 4.6, and Grok 4.1 Fast and GPT-5.4 nano did not exceed chance level at 4 digits and above. The attack can also infer users' memorized attributes (such as health conditions and financial events) from ordinary outputs, and through RL-optimized prompt injection it makes the model encode SSN digits as the number of exclamation marks, reconstructing a 9-digit SSN from a production-style personal agent, with a 97.1% success rate on the first digit on Claude Opus 4.6 and 76.5% for the remaining eight digits given first-digit success.
- Personal AI agents need to hold sensitive data such as calendars, credentials, and health records in context; existing privacy evaluations only use a language judge to determine whether the output leaks a secret, and if the model correctly refuses direct extraction, they consider there to be no leakage. The authors ask: when the secret exists only in the context window, do the model's benign outputs to ordinary requests still carry hidden correlations that can be reconstructed?
- Formalizes unintentional context leakage as a predicate inference game: the attacker has only black-box query access, first estimates the response distribution of the target model, and then infers the secret from benign outputs accordingly. The authors conduct numeric secret reconstruction experiments on eight closed-source models, train classifiers to infer user memory attributes, and use RL to optimize prompt injection so that the model encodes SSN digits into the number of exclamation marks.
- Most of the eight models leak noticeably: 2-digit secrets can be fully reconstructed 100% of the time on Claude Opus 4.6 and Gemini 3.1 Pro, and 4 digits match exactly 82% of the time on Opus; Grok 4.1 Fast and GPT-5.4 nano do not exceed random level at 4 digits and above. The authors propose a suppression mechanism to explain the leakage, which correlates strongly with leakage across models; stronger instructions instead increase leakage, and the post-training stage also increases leakage. Memory attribute inference outperforms a language judge; an RL attack extracts complete SSNs from a production agent.
- The experiments cover only two types of predicates, numeric values and binary attributes, eight closed-source models, and a black-box setting; open-vocabulary free-text secrets are not attempted. The attack assumes the attacker knows the secret length, can identify the target model, and pays a one-time estimation cost; the numeric and SSN experiments use uniformly random secrets and do not cover the structure of real secrets. The memory evaluation uses only one user profile, and the capability correlation uses Arena Elo as a proxy with a sample of only eight models; the impact of deployment-side mitigations such as output perturbation is not studied.
The paper shows that even when a model correctly refuses to reveal a secret, its normal outputs still carry decodable statistical cues, and privacy evaluations that rely on text-based judgments miss this kind of leakage.