Cordyceps: Covert Control Attacks on Large Language Models via Data Poisoning

1. 01What problem does this paper attempt to solve?

   The authors investigate whether poisoned fine-tuning data alone can enable an LLM to learn a semantic steganographic channel, achieving covert control without a fixed trigger word.

   Core question: can poisoning fine-tuning data alone teach an LLM a semantic information-hiding scheme that encodes arbitrary malicious instructions into harmless text, thereby bypassing defenses targeting fixed trigger words.

   • Scenario and motivation: LLMs are often fine-tuned on uncleaned text, and OWASP lists data poisoning as one of the main security vulnerabilities of LLM applications (as cited by the authors). Existing poisoning attacks mostly use fixed trigger words, and the authors argue such attacks can be neutralized by anomaly detection, fine-tuning regularization and online monitoring.
   • Shortcomings of existing methods: the authors state that the strong correlation between fixed trigger words and target behavior can be exploited by defenses (reverse-engineering trigger words or backdoor behavior); hidden-trigger backdoors based on semantic cues or steganographic syntax remain bound to fixed target behavior, or rely on syntactic schemes that can be broken by rewriting.
   • Core observation: mainstream LLMs are pretrained on similar corpora and share background knowledge; the authors hypothesize that demonstrating with adversarial-biased but semantically consistent anchor descriptions can induce an LLM to learn semantic encoding and decoding between shared knowledge anchors and plaintext, rather than memorizing trigger words.
   • Paper contributions: propose the poisoned data generation algorithm CORDYCEPS; formalize the steganography scheme as SHuSh (Semantic Hiding under Shared knowledge) and provide a theoretical framework; evaluate on 2 scenarios (UCC prompt injection, BCC data exfiltration), 5 models, 3 backdoor defenses and 4 prompt injection defenses.
   • Threat model (Section 2.4):
     • Goal: make the poisoned model execute instructions hidden in harmless text (UCC), or encode sensitive data into outputs (BCC).
     • Knowledge: the attacker does not know the base LLM's architecture or weights, and cannot modify the training process or the inference-time decoding method.
     • Capability: can only create the poisoned dataset Dpoison; at deployment can manipulate the <data> external context (UCC) or directly prompt the model (BCC).
     • Victim system and defense assumptions: the trainer checks data with automated anomaly detection and holds a small clean set; online defenders can observe, filter or rewrite all inputs and outputs.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: fixed-trigger backdoors, LLM steganography and watermarking, and prompt injection and its defenses; the authors state this work is poisoning at learning time rather than steganography at generation time.

   Data poisoning and backdoor attacks

   • BadNet [18], VPI [64], Sleeper Agent [50], MTBA [26], CTBA [21], etc.—NLP and LLM backdoors use lexical triggers, instruction fine-tuning poisoning, hidden or style triggers, and prompt-only backdoors, learning static associations between activation patterns and fixed target behaviors. The authors state that SHuSh learns a context-conditioned semantic encoding/decoding mechanism that can transmit variable payloads.
   • Hidden-trigger and style-based backdoors [25, 32, 37]—use semantic cues, language style, or steganographic syntax to reduce detectability; the authors consider them still bound to fixed target behaviors, or dependent on syntactic schemes that can be broken by rewriting.

   Steganography and LLMs

   • LLM steganography and watermarking work [4, 9, 23, 34]—focus on embedding during generation or model-side watermarking, formalizing cover distribution indistinguishability, adaptive sampling schemes, or studying systemic agent risks. The authors state that this paper is orthogonal: it does not design an external codec or watermark, but rather uses poisoning during learning to make the model internalize a semantic channel. The jellyfish example from Motwani et al. [34] is used as the ICL example in this paper.

   Prompt injection and defenses

   • Prompt injection attacks [17, 29, 71]—direct instruction hijacking, indirect injection via untrusted retrieved content, adversarial suffixes; defenses include filtering, anomaly detection, and instruction/data separation during training (DataSentinel [30], PromptLocate [22], SecAlign [5]). The authors state that this line mostly assumes a frozen model with the adversary acting at inference time, whereas covert control is a poisoning vulnerability learned during fine-tuning.

   Baselines and benchmarks: Attack baselines are CoA (Combined Attack) [29] and 5 backdoor attacks (BadNet, VPI, Sleeper Agent, MTBA, CTBA, implemented with BackdoorLLM [27]); defense baselines are DataSentinel, PromptGuard, PromptLocate, SecAlign, ONION, CROW, and rewriting defenses. Benchmarks are OpenPromptInjection [29] (UCC) and a self-built synthetic PII dataset (BCC).

   Author positioning: This paper is a new variant of poisoning attack—the model learns during fine-tuning to decode benign-looking stegotext into malicious instructions, distinct from fixed-trigger backdoors and generation-time steganography.

   Collapse this answer

3. 03How does the paper address this problem?

   CORDYCEPS uses an oracle LLM to generate bidirectional poisoned samples with reasoning chains in three stages, teaching the model to learn the SHuSh semantic steganography scheme.

   Overall approach: use an oracle LLM to generate stegotext that hides attacker plaintext in shared knowledge anchor descriptions, along with encoding/decoding reasoning chains, formatted as instruction fine-tuning data, so that the poisoned model internalizes the SHuSh (Semantic Hiding under Shared knowledge) scheme rather than memorizing trigger words.

   Problem setting and formalization

   • SHuSh definition: given a shared knowledge set K and plaintext m, the encoding function Enc samples stegotext s ∼ Enc(k, m), and the poisoned model uses Dec to decode m from (k, s); the mapping is required to be at the semantic rather than lexical level.
   • Representation space model: assume the semantic representation vector e(x) ∈ R^d satisfies perfect representation consistency, and the generative noise is isotropic Gaussian (Assumption 4.1). The stegotext distribution is P_SH := 𝒩(e(k) + e_A(m), σ² I_d), i.e., the anchor embedding plus a message perturbation vector e_A(m) and noise.
   • Three security properties (Section 2.3): stealth against detectors (clean and poisoned model output distributions are close); robustness (semantics preserved after rewriting); high encoding/decoding accuracy (high SCS, attack effectiveness close to direct attack). SCS is defined as the cosine similarity of the embeddings of two texts.

   Poisoned data construction (Algorithm 1)

   • Content selection: attacker plaintext set M (phrases of diverse length and semantics), shared knowledge anchor set K (short encyclopedia-style descriptions, WikiDes used in experiments), ICL examples (using the jellyfish example from Motwani et al. [34], full prompt given in Appendix D).
   • Phase 1 semantic association: for each (mi, ki), use Esem to prompt the oracle once to generate stegotext si encoding mi.
   • Phase 2 reasoning refinement: given (mi, ki, si), use Ecot to prompt the oracle again to derive two CoTs in reverse: encoding reasoning rienc and decoding reasoning ridec. The authors state the staging is to avoid the oracle overfitting to the logical structure of the ICL examples and to increase the diversity of encoding/decoding strategies.
   • Phase 3 formatting: each (m, s) generates two instruction samples, encoding and decoding (templates Tenc, Tdec); BCC requires encoding samples to teach the model to encode sensitive data into stegotext, while UCC only requires decoding samples.

   Theoretical analysis (Section 4)

   • Stealth: KL(PSH||PCH) = ||e_A(m)||²/(2σ²) = ½·SNRm (Eq. 1); the smaller the perturbation norm, the stealthier.
   • Robustness and accuracy: when the rewriting deviation ρ is bounded, SCS has a lower bound (Eq. 3); the nearest-neighbor decoding error rate is p_err ≤ (|M|−1) · Q(d_min/2σ) (Eq. 4), where dmin is the minimum distance between messages.
   • Stealth–effectiveness trade-off: the authors state that stealth requires small perturbations while decoding margin requires large separation, and SHuSh takes the intermediate range; in the indistinguishability game GameIND-SHuSh, the optimal detection advantage is bounded by the TV distance (Theorem 4.2), with proof in Appendix G.

   Attack scenarios and success criteria

   • UCC (unidirectional covert control): the attacker places stegotext encoding malicious instructions into <data>, similar to indirect prompt injection; success is when the poisoned model executes the instructions therein.
   • BCC (bidirectional covert control): the attacker directly queries the model's memorized sensitive data using stegotext, and the model encodes the answer into stegotext and returns it; success is when the returned target data is embedded in the stegotext.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On 5 models, ASVh of UCC all exceed 0.71, CA of BCC reaches 0.78–0.93, ONION/PromptGuard detection is close to random, and utility differences are within ±1.5%.

   Experimental Setup

   • Models tested: Qwen3-4B-Thinking-2507, Meta-Llama-3-8B-Instruct, gemma-3-12b-it, Phi-4-reasoning-plus, Qwen3-30B-A3B-Thinking-2507; each architecture has three variants: Base (original model), Clean (clean SFT), Poison (CORDYCEPS poisoning).
   • Datasets: anchors use WikiDes; the clean set is 1,260 reasoning samples from OpenO1-SFT; UCC uses OpenPromptInjection (7 NLP tasks × 7 injection tasks, 140 poisoned samples, 50 test samples per pair, 2,450 total); BCC is 200 synthetic PII/financial exfiltration targets generated by Gemini 2.5 Pro (100 training + 100 test, plus 900 clean samples, 1,000 total).
   • Default settings: poisoning rate 10%, oracle is Gemini 2.5 Pro; ablations include oracle choice, 1% poisoning rate, training data rewriting (Table 3) and Phase 2 (Appendix B).
   • Metrics: UCC uses ASVh (executes only the injection task) and ASVs (completes the injection task); BCC uses CA (correct response rate under keyword filtering); stealth uses TPR/FPR, silhouette score, etc.; utility uses MuSR, GSM8K, MMLU, IFEval, GPQA (lm-evaluation-harness).
   • Baselines: attack baselines CoA and 5 backdoor attacks (BackdoorLLM implementation, same poisoning rate); defense baselines DataSentinel, PromptGuard, PromptLocate, SecAlign, ONION, CROW, Gemini rewriting.
   • Evaluation method: BCC's CA uses keyword matching to verify whether sensitive data leaks in plaintext; the paper does not report LLM review or human evaluation.

   Main Results

   UCC attack effectiveness (Table 1, ASVh, averaged over 7×7 task combinations):

   Swipe horizontally to read the table

   Model	CoA(Clean)	BadNet	VPI	Sleeper	MTBA	CTBA	UCC
   Qwen3-4B	0.544	0.359	0.306	0.206	0.292	0.335	0.720
   Llama-3-8B	0.392	0.438	0.410	0.252	0.425	0.411	0.712
   Gemma-3-12B	0.587	0.478	0.510	0.257	0.478	0.509	0.736
   Phi-4-15B	0.512	0.460	0.450	0.158	0.444	0.449	0.716
   Qwen3-30B	0.553	0.592	0.558	0.385	0.586	0.583	0.733

   BCC covert exfiltration accuracy (Table 2, CA, under keyword filtering):

   Swipe horizontally to read the table

   Model	Base	Clean	BCC(Para)	BCC(CROW)	BCC
   Qwen3-4B	0.000	0.020	0.740	0.290	0.780
   Llama-3-8B	0.050	0.050	0.800	0.610	0.810
   Gemma-3-12B	0.250	0.040	0.880	0.860	0.850
   Phi-4-15B	0.000	0.060	0.860	0.860	0.860
   Qwen3-30B	0.000	0.010	0.890	0.910	0.930

   • The authors state that UCC's ASVh exceeds 0.71 on all models, outperforming CoA and all backdoor baselines; the gap between ASVh and ASVs is small (e.g., 0.001 for Qwen3-4B), indicating the model reliably executes hidden instructions.
   • The authors state that BCC's success rate on Base/Clean models is near zero (≤0.06), while poisoned models average above 0.80, indicating that poisoning establishes a reliable covert exfiltration channel.

   Stealth and Robustness (EQ2)

   • Data-level detection: ONION's TPR against CORDYCEPS at [email protected] is 0.063, close to FPR, while MTBA and CTBA are 0.661 and 0.718 respectively (Table 4); embedding clustering filtering (all-MiniLM-L12-v2) cannot separate stegotext from covertext: silhouette score 0.014, between/within ratio 1.021, Fisher ratio 0.0267; in the t-SNE of Figure 3 the two groups of points are mixed together.
   • Online detection: PromptGuard's average detection rate against UCC across tasks is 0.005, indistinguishable from covertext; DataSentinel against UCC is 0.011–0.191, while against CoA it is 1.000 across the board (Table 5). PromptLocate's RL/ES for locating CoA payloads exceeds 0.9, while its RL against UCC is 0.183–0.789 (Table 6).
   • Training-time defenses: CROW reduces UCC's ASVh by at most 13.9% (Llama-3-8B), and by only 5.75% on Gemma-3-12B, less than most backdoors such as VPI's 13.3% (Table 7); SecAlign's ∆ against UCC ranges from −4.77% to +0.81%, while against CoA it reaches −40.9% to −76.5%.
   • Rewriting and adaptive detection: after Gemini rewriting, BCC's CA goes from 0.810 to 0.800 on Llama-3-8B and from 0.850 to 0.880 on Gemma-3-12B (Table 2); an adaptive DataSentinel fine-tuned on CORDYCEPS samples raises TPR to 0.780 and 0.877 on GC and Summ, but covertext false positives also rise to 0.554 and 0.377 (Table 5); the authors state that current adaptive detection lacks clean separation.

   Ablations, Utility and Mechanism Verification

   • Oracle and poisoning rate (Table 3, BCC's CA): Gemini 2.5 Pro and Qwen3.6-35B as oracle perform similarly (0.780–0.950 at 10% poisoning), Qwen3.5-9B is weaker (e.g., Qwen3-30B drops from 0.870 to 0.580); at a 1% poisoning rate (10,000 training set, 100 poisoned) CA remains 0.750–0.920; after the poisoned training samples are rewritten, most victims' CA remains high, with the largest drop on Qwen3-30B (0.870→0.650).
   • Utility (Table 8): the authors report that poisoned and Clean models differ by at most ±1.5% on 5 benchmarks, with an average MMLU drop of <0.3%; exceptions include Gemma-3-12B's GSM8K dropping from 0.692 to 0.616 (UCC), and Phi-4-15B rising from 0.804 to 0.868.
   • Mechanism verification (Section 5.5, using the fully poisoned Llama-3-8B variant, 4096-dimensional intermediate-layer hidden states): in the t-SNE of Figure 4a, stegotext clusters into a specific region, close to the e(k)+e(m) position; in Figure 4c, the cosine similarity between recovered anchors and true anchors all exceeds 0.996; in Figure 5 the residuals show a zero-centered symmetric bell shape, which the authors state supports the additive bias and isotropic Gaussian noise assumptions.

   Other Ablations and Analysis

   • Phase 2 ablation (Appendix B, Figure 8): Phi-4-15B achieves the best SCS of 0.770 at Yreason=0.6, and only 0.623 at Yreason=0 (removing Phase 2).
   • Feasibility analysis (Appendix A, 100% poisoning): under weighted loss, SCS across 5 models is 0.732–0.818 (Table 9), higher than in-context instructions' 0.176–0.213.
   • Channel capacity (Figures 6, 7): CER is usually <1 within about 300 bits; EMR decreases monotonically with the number of plaintext tokens, approaching 0 at 5 tokens.
   • Downstream task SCS (Table 10): average 0.8276–0.8910 across 7 task types.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that weak or domain-specific shared knowledge is a practical limitation; the experiments cover 5 open-source models, 2 scenarios, and 7 defenses.

   Limitations and future directions noted by the authors

   • Shared knowledge limitation (Conclusion): The authors state that the attack does not require anchors to match exactly, but weak shared knowledge or highly domain-specific shared knowledge remains a practical limitation.
   • Adaptive detection direction (Section 5.3): The authors believe adaptive detectors could exploit residual distribution signals, but current adaptive detection cannot yet provide clean separation between stegotext and benign covertext.
   • Defense layering and false-positive trade-offs (Ethical Considerations): The authors state that a single sanitizer, rewriter, prompt injection detector, or alignment defense is each insufficient; the stealthiness–effectiveness trade-off points to adaptive detectors and representation-level monitoring as promising directions, but these may introduce false-positive trade-offs that require careful evaluation.
   • Scope statement (Introduction): The authors explicitly do not design provably secure LLM steganographic primitives, but instead construct poisoned samples that induce a weaker learned hidden scheme.

   Experimental coverage

   • The models tested are 5 open-source models (4B–30B, Qwen3, Llama-3, Gemma-3, Phi-4 series), all fine-tuned with LoRA (Appendix C.1).
   • Attack scenarios cover UCC (OpenPromptInjection, 7×7 task pairs) and BCC (200 synthetic PII/financial targets); the BCC data are synthetic, and the paper states that no real privacy data were handled (Ethical Considerations).
   • Defense evaluations include ONION, DataSentinel (including an adaptive version), PromptGuard, PromptLocate, SecAlign, CROW, Gemini rewriting, and embedding clustering filtering.
   • The default poisoning rate is 10%; ablations cover a 1% poisoning rate, 3 oracles (Gemini 2.5 Pro, Qwen3.6-35B, Qwen3.5-9B), and training data rewriting (Table 3).
   • The paper does not report human evaluation or LLM-judge agreement for attack success determination, nor does it report variance across repeated experiments.

   Collapse this answer

6. 06Summarize the main content of the paper

   CORDYCEPS uses poisoning to enable an LLM to learn the semantic steganographic channel SHuSh, achieving covert control on 5 models and bypassing all 7 tested defenses.

   A data poisoning attack paper: the authors propose CORDYCEPS, which teaches a poisoned LLM to learn a steganography scheme SHuSh that performs semantic encoding and decoding between shared knowledge anchors (such as encyclopedia entries) and attacker plaintext, thereby achieving two types of covert control without fixed trigger words—UCC (placing stegotext encoding malicious instructions into external data, similar to indirect prompt injection) and BCC (encoding sensitive data into model outputs for exfiltration).

   • Method: an oracle LLM (default Gemini 2.5 Pro) generates stegotext and encoding/decoding reasoning chains in two stages, formatted as bidirectional instruction fine-tuning samples; the theoretical framework models stegotext as anchor embedding plus message perturbation and Gaussian noise, giving bounds on stealth (KL divergence), robustness and decoding error rate.
   • Attack effectiveness: UCC's ASVh on 5 models is 0.712–0.736, exceeding CoA and 5 backdoor baselines (Table 1); BCC's CA is 0.780–0.930, while Base/Clean models are ≤0.06 (Table 2).
   • Stealth: ONION's TPR at [email protected] is only 0.063 (Table 4), PromptGuard's average detection rate is 0.005 (Table 5), and neither embedding clustering nor t-SNE can separate stegotext from covertext.
   • Robustness: CROW reduces UCC's ASVh by ≤13.9%, SecAlign's ∆ is within ±5% (Table 7); after rewriting, BCC's CA is largely maintained (Table 2); at a 1% poisoning rate, CA remains 0.750–0.920 (Table 3).
   • Utility and mechanism: the authors report that poisoned models differ from Clean by at most ±1.5% on 5 utility benchmarks (Table 8); in representation space analysis, the cosine similarity between recovered anchors and true anchors exceeds 0.996, and the residuals show a zero-centered bell shape, which the authors state supports their additive bias and Gaussian noise assumptions.
   • Authors' conclusion: covert control exposes defense gaps designed around explicit malicious instructions and static trigger associations; the authors argue defenses should be layered, and note that weak or domain-specific shared knowledge is a practical limitation.

   Collapse this answer

Related papers