CLIOPATRA: An attack that extracts private information from LLM insights

Core question: can the heuristic privacy protections layered onto LLM insight systems (PII removal, clustering, aggregation, LLM auditing) withstand an active poisoning attack? The authors give a negative answer with Cliopatra.

• Problem setting: AI assistant providers (Anthropic, OpenAI, Microsoft, Google, etc.) are building platforms that extract insights from user chats; chats often contain sensitive information such as medical data, and these platforms claim to be "privacy-preserving." The authors argue that most of them rely only on stacked heuristic measures and lack formal guarantees.
• Shortcomings of existing methods: the authors state that prior prompt-injection privacy attacks mainly target RAG systems, which contain only one LM and have no privacy protection mechanisms, and thus cannot answer whether multiple layers of protection can be breached simultaneously.
• Core observation: each layer of protection depends on the LM's ability to filter sensitive information, and the authors find that the facet extraction stage alone leaks age, gender, symptoms and even disease (Table 3), paving the way for downstream attacks.
• Proposed in the paper: Cliopatra—which the authors describe as the first privacy attack targeting LLM insight systems, combining data poisoning, prompt injection and attribute inference; evaluated on Anthropic's Clio (a local reproduction).
• Threat model:
  • Goal: extract private information (specifically disease) from the target user's chat.
  • Knowledge: internal system parameters (the LLM used, the minimum cluster size C), the target user's age/gender/partial symptoms, and a candidate disease list.
  • Capability: create a small number of accounts to insert malicious chats into the system; access published insights.
  • Victim system: Anthropic's Clio (local version, not the online system).

Attacks against RAG

• Anderson et al. (2024), Chaudhari et al. (2024), Naseh et al. (2025)—membership inference and data reconstruction against RAG systems, combining direct/indirect prompt injection with poisoning. The paper notes that Clio differs from RAG: documents are first anonymized via LLM summarization and cannot be queried arbitrarily, so attacks require new designs (multi-layer prompt injection, information extraction algorithms, combined baselines).

Summary privacy and formal protection

• URANIA (Liu et al., 2025)—provides (ε, δ)-DP guarantees for Clio-like systems, and uses DP auditing to compare URANIA and Clio; the paper notes that its attack is mostly ineffective against Clio, and that it assumes the attacker knows all chats except the target, whereas Cliopatra's attacker knows only public information. URANIA is also the defense comparison system in this paper.
• Hughes et al. (2025)—studies the privacy of LLM summarization, but only uses simple prompts for passive measurement; the paper states that this work is an active attack, targeting the complex prompts of a real system and breaking through multiple layers of defense.

Heuristic anonymization and LLM privacy

• Sampling/aggregation/suppression-style anonymization in the database field (e.g., Netflix Prize, Diffix) and work that broke them (Cohen & Nissim 2020, Narayanan & Shmatikov 2009)—the authors state that this paper moves the same "heuristics are not privacy" argument into the LLM era, and introduces poisoning and prompt injection.
• PII detection/removal tools (Flair, Presidio) and training-data privacy attacks (Carlini et al. 2021, Nasr et al. 2025, and membership inference)—the paper notes that these target training data, whereas this paper focuses on data during LLM processing.

Prompt injection and defenses

• Prompt injection attacks such as Perez & Ribeiro (2022), Greshake et al. (2023); defenses such as StruQ, Spotlighting, MELON—the paper states that many defenses can be bypassed by adaptive attackers, and that data/control-flow separation defenses mainly target agent workflows and are difficult to adapt to large-scale open analysis like Clio.

Baseline and benchmark: The baseline attacker guesses the disease with Claude Sonnet 4.5 using only public information (age, gender, known symptoms, candidate disease list); the data is random sampling from WildChat mixed with synthetic medical target chats (persona attributes sampled from NLICE). The authors position this paper as the first work to combine data poisoning, prompt injection, and attribute inference to simultaneously break through multiple layers of heuristic defenses.

Overall approach: the attacker constructs and inserts C−1 poisoned chats so that they cluster together with the target chat, inducing the cluster summary to leak the target's private information, which is then extracted from the published summary. The method is named Cliopatra.

Threat model and attack game

• Formalized as Game 1: a base conversation X is sampled, the attacker generates poisoned chats based on public attributes tpub, system parameters Φ and a candidate disease set Y, the system runs Clio on X ∪ {t} ∪ Xpoison and outputs a summary S, the attacker infers the private attribute tpriv from S, and a correct guess outputs 1.
• The authors emphasize that evaluation is deterministic: the inferred disease is compared directly with the ground truth, without LLM-as-a-judge.

Poisoned chat construction

• Nested multi-layer prompt injection: template trigger phrases based on known age/gender/symptoms make the poisoned chat embed close to the target; injection targeting the summarizer induces it to write the target's medical history into the summary; injection targeting the extractor ensures the injected text is not rewritten; a phrase appended at the end fixes the assistant's reply so that facet extraction focuses only on the poisoned prompt. Appendix B gives the full poisoned chat format for each extractor.
• The poisoned chat is inserted C−1 times: if it clusters with the target, the cluster size is exactly C; otherwise the poison cluster is filtered out for being too small.

Information extraction

• Regex Attack: first use regex matching on public information to locate the target cluster summary (match-count threshold τ), then regex-search the summary for diseases according to the candidate disease list; if either step fails, fall back to the baseline (Algorithm 2).
• LLM Attack: use Claude Sonnet 4.5 to infer the disease from all cluster summaries, handling regex failures caused by summary rewriting; Appendix B gives the full prompt.

Fast Clio and extensions

• Fast Clio: summarize only clusters containing poisoned chats to reduce evaluation cost (Algorithm 3); the authors state that when combined with Regex Attack it performs consistently with Full Clio (verified in Table 6).
• Confidence: the C-Regex variant removes the baseline fallback and can abstain from guessing in exchange for high confidence.
• Stealth: synonym substitution on poisoned trigger phrases to bypass duplicate checking; modifying the injection so the summary output takes the form of harmless text such as "write a paper on the association between a certain disease and a certain topic" to evade manual and automated review; Appendix C gives the full prompt.
• Unknown C or sampling: inserting more than C−1 poisoned chats lowers the average success rate, but high-confidence extraction is still possible when the target is sampled.

Experimental setup

• System under test: a locally reproduced Clio, with extractor/summarizer configured using 4 model families: Qwen 3 (4B/30B), Gemma 3 (4B/27B), LLaMA 3 (8B/70B), Claude (Haiku 3/Sonnet 4.5) (Table 2).
• Data: random sampling from WildChat (1K by default) + 100 synthetic medical target chats; persona attributes (age, gender, symptoms, disease) sampled from NLICE (47 diseases), with chats generated using Claude Sonnet 4.5.
• Hyperparameters: number of clusters set to an average cluster size of 50; embedding model all-mpnet-base-v2; by default the attacker knows age, gender, and 1 random symptom; a single A100.
• Metrics: attack success rate (proportion of runs that correctly infer the disease), precision, abstention rate; one Game 1 run per target chat.
• Baseline: Claude Sonnet 4.5 guessing the disease from public information alone; for C-Regex comparisons, baseline precision is computed at matched coverage.
• Determination method: inferred disease compared deterministically with the ground truth, without LLM judging.

Main results

Attack success rate (%), attacker knows 1 symptom (per Table 4):

Swipe horizontally to read the table

Configuration	Baseline	% clustering success	Regex Attack	LLM Attack
Qwen	22	89	71	81
Gemma	22	46	37	44
LLaMA	22	52	37	42
Claude	22	43	34	39

• The authors state that the Regex attack extracts the disease of up to 71% of target users, about 2 times the baseline that does not see Clio's output; LLM Attack is even higher.
• The authors state that the Claude family has the strongest protection (Clio's prompts were designed for it), yet 39% (LLM Attack) were still extracted, arguing that LLM-dependent ad-hoc privacy measures may have fundamental flaws.
• Facet leakage (Table 3): facets of all models leak age/gender/symptoms/disease; e.g., 73% of chats leak the disease under Qwen, and only 2% leak age under Gemma.

Confidence (C-Regex)

• With 1 symptom known (Table 5): Qwen precision 98.4%, abstention rate 35%; Gemma/LLaMA/Claude precision 100.0%, abstention rates 81/80/83%; baseline precision at matched coverage 15.8–23.5%.
• The authors state that precision is more than 4 times the baseline, and that when the attack extracts the disease the attacker can be fully confident; the authors believe the one Qwen error may be because Clio's prompts were not calibrated for Qwen.

Stealthy attack and scale ablations

• 5 symptoms known, 1K chats (Table 7): the stealthy attack reduces the proportion of summaries containing PII from about 100% for the non-stealthy version to about 15% (Qwen 1.08%, Gemma 13.3%, LLaMA 0%, Claude 14.6%), while precision remains 100%; abstention rates Qwen 23→30, Gemma 63→87, Claude 74→75, while LLaMA drops from 77% to 56%—the authors state that the stronger injection is more effective against LLaMA.
• Chat scale (Figure 3, 5 symptoms known): Claude abstention rate at 1K/10K/100K is 74%/92%/98%; at 100K the disease can still be extracted from 2% of chats with 100% precision, with baseline precision 53.2%.
• Number of poisoned chats (Table 8, 100K chats): with 49/100/1000 poisoned chats the proportion correctly clustered is similar (e.g., Qwen 36/30/33); the authors state that when the target is sampled, extraction can still be done with 100% precision.

Defense evaluation

• LLM privacy audit (Table 9, 5 symptoms known, 1K chats): among the 163 clusters from which C-Regex successfully extracted the disease, the number of clusters flagged with a score of ≤2 by both the same-family auditor and the Claude Sonnet 4.5 auditor is 0; 57.7% of successfully attacked clusters received a 5/5 score. The authors state that LLM auditing is unreliable.
• URANIA (DP defense) (Table 10, ε∈{25,50}, δ=10^-5): at ε=25, C-Regex precision drops to 15–51% (100% on Clio); at ε=50, Qwen still reaches 100%. The authors state that DP is clearly more promising, but cite Liu et al. noting that URANIA has utility loss, and that its experiments assume a single user with a single conversation.

Other ablations and analyses

• Attacker knowledge (Figure 4): as known symptoms increase from 1 to 5, the success rate rises monotonically, with Qwen approaching 100%; LLaMA plateaus later, which the authors suggest may be because not all diseases leak into the facet.
• Programming-domain proof of concept: 100 template chats, Claude configuration, API key extraction success rate 56%, with the remaining 44% of API keys not appearing in the facet.
• Baseline substitution (Table 12): adding in-context examples to the baseline raises it from 22% to 36%, and the attack success rate rises by about 10 percentage points in tandem (e.g., Qwen 71→77).
• Full/Fast Clio comparison (Table 6): the Regex attack success rate is exactly the same for both.

Limitations and future directions noted by the authors

• The evaluation uses synthetic target chats (for privacy and ethical reasons); the authors plan to evaluate real-world impact with real target chats under strict ethical review (Limitations & Future Work).
• Both Clio and Cliopatra only target text summarization; the authors propose future exploration of the feasibility of attacks in multimodal (image, file, etc.) settings (Limitations & Future Work).
• The attack targets only one chat at a time; the authors propose future research on untargeted attacks or attacks targeting multiple chats simultaneously, and note that the effect of multiple different poisoned chats on clustering is not yet clear (Limitations & Future Work).
• The attack assumes the attacker can access the LLM underlying Clio; the authors propose future full ablation evaluations of the attack's transferability across different models/families, stating that rewritten poisoned prompts in the stealthy attack still maintaining 100% precision is only partial evidence (Limitations & Future Work).
• For adaptation to OpenAI/Microsoft/Google systems, only a failure-point analysis is given; the authors explicitly leave actual attack instantiation to future work (Section 6.2).

Experimental coverage

• The system under test is a locally reproduced Clio, configured with 4 model families: Qwen 3, Gemma 3, LLaMA 3, and Claude; defense evaluation includes Clio's built-in LLM audit (same-family auditor and Claude Sonnet 4.5) and URANIA (ε=25/50).
• Data is random sampling from WildChat (1K/10K/100K) mixed with 100 synthetic medical target chats; the programming-domain proof of concept uses 100 template chats.
• The attacker-knowledge ablation covers 1–5 known symptoms; the poisoned-chat-count ablation covers 49/100/1000 chats (under 100K chats).
• The paper does not report the number of repetitions of the attack run per target chat, nor the actual number of accounts required to insert poisoned chats for the attack.
• The paper does not attack the online Clio system; the authors state they have disclosed to Anthropic, and Anthropic responded with mitigation measures it is exploring.

An attack paper targeting "privacy-preserving" LLM insight systems: the authors propose Cliopatra, testing whether the heuristic multi-layer protections represented by Anthropic's Clio are reliable.

• Problem: Clio-like systems protect user chats by stacking PII removal, clustering, aggregate summaries, and LLM privacy auditing, none with formal guarantees; prior prompt injection attacks only targeted unprotected RAG systems.
• Method: The attacker inserts C−1 poisoned chats containing nested prompt injection, causing them to cluster with the target chat and inducing the cluster summary to write out the target's medical history, then extracts the disease from the summary using regex (Regex/C-Regex) or Claude Sonnet 4.5 (LLM Attack); evaluation is deterministic comparison, without LLM judging.
• Main results:
  • With 1 symptom known, the Regex attack success rate is up to 71% (Qwen), LLM Attack on Claude is 39%, baseline 22% (Table 4).
  • The C-Regex variant achieves 100.0% precision on Gemma/LLaMA/Claude, with abstention rates 80–83% (Table 5).
  • At the 100K chat scale, the disease can still be extracted from 2% of chats with 100% precision (Claude configuration).
  • None of the 163 successfully leaked clusters was flagged by the LLM audit, and 57.7% received a 5/5 score (Table 9).
  • URANIA at ε=25 reduces attack precision to 15–51%, but at ε=50 Qwen still reaches 100% (Table 10).
• Authors' conclusion: The authors argue that stacked heuristic protections are insufficient to protect user data, that prompt injection is an underestimated risk in LLM insight systems, and that formal DP is more promising but faces utility and deployment obstacles; limitations include synthetic chats, single-target attacks, and the assumption that the system model is known.