Evaluating and hardening LLM system instructions against encoding attack extraction

System instructions may still leak via encoded or structured output requests even after direct extraction is rejected.

• Scenario: The authors state that system instructions define the agent's identity, constraints, security policies, and access to external resources, and may contain API credentials, internal policies, and workflows. Leakage is listed by OWASP as LLM07:2025 System Prompt Leakage and can facilitate subsequent prompt injection and privilege escalation.
• Existing gaps: The authors state that many applications rely on refusal-style instructions, where default-prohibited information can only be extracted via explicit queries. Alignment and rules mainly identify direct disclosure intent in natural language; reasoning-based defenses require additional steps and are not consistently deployed in production systems.
• Core observation: The authors believe that models often judge based on surface intent before full semantic reconstruction. If a request looks like formatting, serialization, or representation transformation, it may be treated as a benign task, thereby outputting protected content.
• This paper proposes: An automated evaluation framework that generates encoding and formatting requests and tests leakage on instructions with verified refusals; then uses a CoT model for a one-time pre-deployment instruction rewrite, without retraining the target model.
• Threat model:
  • Goal: Extract system instructions or confidential information embedded within them.
  • Knowledge: The attacker interacts with the AI agent in natural language and cannot directly access the system instructions that configure the agent.
  • Capabilities: Can send arbitrary prompts and observe responses; cannot modify system instructions, access server-side memory, or intercept external tool outputs.
  • Violation: The response to an encoded or structured output request contains partial or complete system instructions, secret variables, or reconstructable fragments.

The authors position this paper in terms of prompt extraction, injection jailbreaks, alignment and structural defenses, and evaluation red teaming.

Prompt extraction and system instruction leakage

• Extraction attacks: Effective Prompt Extraction by Zhang et al. (2024), Prompt Stealing by Sha and Zhang (2024), and PLeak by Hui et al. (2024) demonstrate that hidden instructions and system prompts can be recovered through crafted interactions.
• Dialogues and benchmarks: Agarwal et al. (2024) discuss prompt leakage and mitigation in multi-turn applications; Raccoon by Wang et al. (2024) benchmarks prompt extraction for integrated applications. Sternak et al. (2025) automate prompt leakage with agent-based methods.
• Relation to this paper: The authors state these works establish the feasibility of extraction; this paper only studies extraction framed as encoding and transformation tasks and systematically evaluates across multiple encodings.

Prompt injection and jailbreaks

• Injection: Greshake et al. (2023) analyze prompt injection in applications integrating LLMs; the OWASP material cited by the authors lists prompt injection as a major risk for LLM applications.
• Jailbreaks and red teaming: Jailbroken by Wei et al. (2023), transferable adversarial attacks by Zou et al. (2023), twenty-query black-box jailbreaks by Chao et al. (2025), and red teaming studies by Perez et al. (2022) and Pathade (2025) discuss how alignment safeguards can be bypassed by adversarial prompts.
• Relation to this paper: The authors state these attacks aim to change model behavior or bypass safety mechanisms; this paper's threat is using encoded representations of requests to disclose protected system instructions.

Alignment, prompt protection, and structural defenses

• Alignment: RLHF by Ouyang et al. (2022) and Constitutional AI by Bai et al. (2022) are described by the authors as used to strengthen privileged instruction following and reduce susceptibility to adversarial prompts. The introduction also mentions Chain-of-Thought by Wei et al. (2022) and self-consistency by Wang et al. (2023), noting such methods typically require additional reasoning steps.
• Detection and mitigation: Rebuff performs prompt injection detection; ProxyPrompt and System Vectors are listed by the authors as prompt extraction mitigations; there is also a prompt-hardening framework. Polymorphic prompt by Wang et al. (2025) randomizes prompt structure against predictable boundaries.
• Relation to this paper: The authors state structural randomization misses representation-invariant leakage because system instructions remain in context; defenses targeting instruction overriding, boundary inference, or malicious concatenation may not address semantic compliance when encoding is treated as a legitimate transformation. This critique is attached to structural and boundary defenses; the paper does not present it as a conclusion specific to Rebuff, ProxyPrompt, or System Vectors.

Evaluation, baselines, and positioning

• Evaluation: PromptBench evaluates robustness against adversarial prompts; Raccoon focuses on prompt extraction benchmarks. The authors state red teaming methods have been studied extensively.
• Baselines and data: The experimental baseline is direct extraction with verified refusals (0% ASR), not numerical comparisons against the aforementioned attack methods. The data are self-built system instructions, not external benchmarks named in the paper.
• Positioning: The authors state encoded prompt extraction has lacked systematic study; this paper quantifies leakage across multiple encoding types and model families and evaluates lightweight prompt-engineering defenses without modifying the underlying model.

Automated evaluation and hardening framework rewrites extraction requests into encoding or structured output tasks and rewrites system instructions with a CoT model before deployment. Figure 1 shows the evaluation pipeline; Figure 2 adds the rewriting path.

Attack setting and encoding taxonomy

• Success definition: Direct requests for system instructions are refused, but equivalent encoded or structured requests reveal partial or complete system instructions, or other sensitive system-level information. The authors view this as a failure due to different request representations, not as an absence of refusal behavior.
• Four technique categories: character-level obfuscation (e.g., ROT13, Base64, leetspeak, Pig Latin); structural embedding wrappers (e.g., JSON, YAML, TOML); symbolic and representational encodings (e.g., Morse); log and protocol embedding (e.g., HTTP headers, system logs). Section 3 also gives examples like Unicode, emoji, XML, Markdown, Braille, etc.; Table 1 does not list all separately, nor does it indicate which category each column belongs to.
• Unified requests: All techniques share the same user prompt template, only replacing the encoding or format name. The paper provides an example instruction with a confidential key and a formatted request example; not reproduced here.
• Mechanism proposed by the authors: The authors suggest safety mechanisms often judge based on surface intent before full semantic reconstruction; formatting, serialization, or representation tasks may be treated as benign transformations, thereby outputting restricted content.

Automated pipeline

• Four components: encoding capability probing, target LLM, automatic prompt generator, and reviewer LLM (Section 4.1). The target model is first loaded with a fixed system instruction.
• Probing: Benign formatting or encoding requests check whether the model can interpret and generate the format, keeping only supported encodings. The paper does not list which encodings were discarded for each model.
• Filtering and attacks: For each instruction, a direct extraction is attempted first. Only when refused, attack prompts are generated for each valid encoding and reviewer labels are recorded (Algorithm 1).
• Not specified in the paper: sampling temperature, random seeds, number of repetitions, and query budget per model.

Instruction set and leakage determination

• Construction: Initially 80 instructions are written, covering finance, healthcare, DevOps, and enterprise assistant, including API identifiers, internal variables, or confidential policies. 46 instructions consistently refuse direct extraction across all tested models and enter evaluation (Section 4.2).
• Baseline: Directly asking for confidential variables, internal access details, or restricted contexts governed by the instruction. Only if the model refuses or gives a safe response without revealing the system instruction or sensitive system information in full or in part is it included in the baseline (Section 4.3).
• Reviewer input: original system instruction, user attack prompt, and target response. Leakage is judged when there is verbatim text, fragments, paraphrases, encoded representations, confidential variables, or any information usable to reconstruct the system instruction; otherwise No Leakage. Output is limited to these two labels (Section 5.3.2).
• Reviewer model: GPT-OSS-120B, deployed locally on NVIDIA DGX Spark. The paper states human review was done on a random subset, without reporting agreement rates or the number of sampled items.

Design-time instruction rewriting

• Position: Rewriting is done before deployment with a CoT model, not inference-time CoT, and does not retrain the target model (Figure 2, Algorithm 2). The abstract calls it one-shot instruction reshaping. The rewriting model is also GPT-OSS-120B (Section 5.3.2).
• Three elements: explicit non-disclosure rules prohibiting verbatim, partial, or reconstructable leakage; refusal triggers when requests attempt to transform, encode, or otherwise expose protected information; priority constraints making confidentiality policy override user-specified format or output instructions.
• Process: The CoT model generates a rewritten instruction; the evaluation framework measures leakage for the original and rewritten instructions separately and computes the difference. The paper shows an example before and after rewriting; key values or clause text are not reproduced here.
• Defenses not separately experimented: The authors discuss structural randomization and state that randomizing delimiters cannot prevent the model from treating formatting requests as legitimate transformations because system instructions remain in the context window. This discussion has no separate experimental table.

Success metrics

For model M and encoding e, the paper defines ASR(M,e)=1/(|S|)∑_{s∈ S}𝕀[Leak(M,s,e)], i.e., the proportion of baseline instructions judged as leaking. Reduction(%) in Table 3 is, per the table note, 100×(Before−After)/Before.

Experimental Setup

• Models tested: Section 5.1 lists seven: GPT-4.1-mini, GPT-3.5-turbo, Gemini-2.5-flash, LLaMA-3-8B, Mistral-Small-3.1-24B, Microsoft-Phi-3.5-MoE-Instruct, IBM Granite 4.0 H-Small FP8. The Mistral row in Tables 1–3 is written as Mistral-Small-4-119B, and the Phi header is written as Microsoft-Phi-3.5-MoE Instruct. The paper does not state whether the former pair of names are the same model. The authors say they cover hosted chat models and locally deployed LLMs, without indicating the deployment method for each.
• Data: 80 were initially constructed, and 46 were selected that consistently refused direct extraction across all tested models. The paper does not report the actual number of encoded attacks per model; the decimal step sizes of the rows in Table 1 are not all the same.
• Baseline and attacks: Directly asking for the protected information. The direct extraction ASR of the selected instructions is 0% (Section 5.4). The encoding attacks use a unified template, replacing only the format name; the template is not restated here.
• Metric: ASR is the proportion of Leak indications over the instruction set. Leakage includes the original source, fragments, rewrites, encoded forms, secret variables, or reconstructable information. Section 5.4 says Table 1 is reported in percentages, while the numbers in the table are decimals from 0–1; the numbers in the table are transcribed below as they are.
• Review: GPT-OSS-120B performs binary classification. The paper says manual annotation was done on a random subset and compared with the reviewer, without reporting the sample size, agreement rate, or number of repetitions.
• Hardening: The same GPT-OSS-120B generates rewritten instructions with CoT, then re-tests with the same judgment (Section 6). The paper does not report temperature, decoding parameters, or the number of queries per item.

Main Results

The table below lists only YAML, TOML, and structural embedding categories. YAML/TOML before hardening is according to Table 1, YAML after is according to Table 2, and structural categories are according to Table 3. The direct extraction baseline is 0% ASR and is not included in the table.

Swipe horizontally to read the table

Model	YAML before	TOML before	YAML after	Structural before	Structural after
GPT-4.1-mini	0.9565	0.9130	0.6304	0.9783	0.6739
GPT-3.5-turbo	0.8478	0.8696	0.3478	0.9565	0.4348
Gemini-2.5-flash	0.6087	0.6304	0.0435	0.7391	0.1087
LLaMA-3-8B	0.8913	0.8696	0.5435	0.9565	0.7609
Mistral-Small-4-119B	0.6552	0.7586	0.2414	0.7011	0.1609
Microsoft-Phi-3.5-MoE Instruct	0.7667	0.8000	0.5000	0.7000	0.3778
IBM Granite 4.0 H-Small FP8	0.5000	0.5294	0.2941	0.5392	0.1961

• The authors say the ASR of structural embedding wrapping is the highest, and often exceeds 90%. In the table, Gemini, Mistral, and Granite's YAML or TOML do not exceed 0.9; the ≥ 0.7 written in the abstract likewise does not cover the Gemini and Granite rows.
• The authors say open-source and lightweight models leak more easily, and more heavily aligned models leak less easily. In the table, GPT-4.1-mini's YAML and structural categories are higher than Granite and Gemini, which is not fully consistent with this statement. The paper marks only LLaMA-3-8B as non-CoT.
• The authors say that verified direct refusal cannot be exchanged for robustness to encoding. The authors also say Table 3 is the average over all instructions within a category of techniques; its numbers are not the same as the arithmetic mean of the corresponding columns in Table 1, and the paper does not explain the aggregation method further.

Category Aggregation and Reductions

• Character-level and log categories: In Table 3, before character-level hardening, LLaMA-3-8B is 0.7826 and Granite is 0.2059; after rewriting, the former is still 0.7826 (Reduction 0.0000), and the latter is 0.0074 (Reduction 96.400). Before log and protocol hardening, LLaMA-3-8B is 0.9783 and Granite is 0.3382; after rewriting they are 0.8043 and 0.0147 respectively.
• Symbolic category: Before hardening, GPT-4.1-mini is 0.1087 and Granite is 0.0000. Mistral-Small-4-119B's Before/After is 0.1379/0.0000, and the Reduction column writes 13.790; Phi is 0.1000/0.0000 and 10.000. These two cells are inconsistent with the formula in the table note, and the paper does not explain it. Granite's symbolic category is 0.0000 both before and after, and Reduction is 00.000.
• Authors' interpretation: The authors say that structural categories that change the task framing and log categories are more effective than changing only surface characters, and that leakage does not come only from tokenizer blind spots. In Table 3's structural category Reduction, Gemini-2.5-flash is 85.300, GPT-4.1-mini is 31.100, and LLaMA-3-8B is 20.500. The authors say proprietary models have larger reductions after instruction rewriting, and LLaMA-3-8B improves less.

Per-Technique Exceptions After Hardening

• Cells higher than before hardening: LLaMA-3-8B's Pig Latin goes from 0.4348 in Table 1 to 0.4783 in Table 2, Rot13 from 0.2391 to 0.2609, and Leetspeak from 0.6087 to 0.6522. GPT-4.1-mini's Syslog goes from 0.1522 to 0.2117.
• Structural formats still high after rewriting: In Table 2, GPT-4.1-mini's YAML is 0.6304 and TOML is 0.4130; Phi's TOML is 0.5667, YAML is 0.5000, and Git Ignore is 0.4000. Most columns for Gemini-2.5-flash are 0.0000, and Cron Comment is 0.1304.
• Authors' interpretation: The authors say that after changing informal confidentiality statements into explicit rule hierarchies, leakage can be reduced at inference time without doing CoT, and no rules targeting specific encodings were added. The authors believe robustness also depends on how constraints are worded. The paper does not explain the above cells that increased.

Other Ablations and Analyses

• Wording: The authors say that adding one context word to the format name, or switching to a semantically equivalent extraction question, changes ASR; the paper does not give the ASR for each of these prompts.
• Base64 before hardening: GPT-3.5-turbo is 0.0435, GPT-4.1-mini is 0.0870, and Granite is 0.2647 (Table 1).
• Emoji before hardening: Phi and Granite are 0.0000, and LLaMA-3-8B is 0.1957 (Table 1).
• Morse before hardening: Granite is 0.0000, GPT-4.1-mini is 0.0652, and Phi is 0.2000 (Table 1).
• JSON Wrapper before hardening: GPT-4.1-mini is 0.2391, LLaMA-3-8B is 0.6739, and Mistral-Small-4-119B is 0.6897 (Table 1).
• Manual verification: The paper says manual annotation was done on a random subset, without reporting the agreement rate. Section 5.4.1 raises whether the system prompt is a privileged instruction inside the model or part of the serialized input, and the paper does not give a corresponding measurement.

Limitations and future directions noted by the authors

• The evaluation used a set of commercial and open-weight models; the authors state that the results may not generalize to all current or future LLMs (Section 8).
• The attack corpus consists of predefined encoding and serialization techniques; the authors state that attackers may use other representations or multi-stage extraction not evaluated (Section 8).
• The success metric is binary and does not distinguish partial from full disclosure; the authors state that some edge cases may require human judgment and propose finer-grained leakage proportion metrics for future work (Section 8).
• The defense is only prompt-level hardening, not involving retraining, architecture changes, or external security controls; the authors state that they observed a reduction in actual risk, not complete elimination of leakage (Section 8).
• Experiments were conducted under controlled conditions; the authors state that real applications may also involve retrieval, agent frameworks, and external tools, which are beyond the scope of this study (Section 8).
• The authors propose future work on finer-grained leakage metrics, broader encoding taxonomies, multi-turn or tool-assisted extraction, and confidentiality mechanisms that can handle transformed requests and semantic equivalence (Section 9).

Experimental coverage

• The tested models are the seven listed in Section 5.1; the results tables use the notation Mistral-Small-4-119B, which differs from Mistral-Small-3.1-24B in Section 5.1, and the paper does not state whether they are the same model.
• Instructions were selected from 80 down to 46, covering the domains finance, healthcare, DevOps, and enterprise assistant; the direct extraction baseline was 0% ASR. The paper does not specify the denominator for encoding attacks per model.
• Encoding results cover the 13 column techniques in Table 1 and the four aggregated categories in Table 3; hardening was done by a one-time rewrite with GPT-OSS-120B and then retested (Table 2, Table 3).
• Leakage determination was a binary output from GPT-OSS-120B. The paper mentions manual spot checks but does not report agreement rates, the number of spot-checked items, repetitions, temperature, or query budgets.
• The paper discusses polymorphic prompt assembly and states that it does not eliminate representation-invariant leakage because the system instructions remain in the context window; this discussion has no corresponding experimental numbers in this paper.

This work evaluates and hardens the confidentiality of LLM system instructions under encoded extraction. The authors are concerned with: when direct extraction has been rejected, whether the model still surrenders protected content when the same request is rewritten as a formatting or encoding task.

• Approach: The framework first confirms the model can generate the encoding, keeps only instructions where direct extraction was rejected, then sends a unified formatting request, with GPT-OSS-120B making a binary leakage determination. Leakage includes original text, fragments, rewrites, encoded forms, secret variables, and reconstructable information. Hardening occurs pre-deployment: the same model rewrites confidentiality statements to include non-disclosure, refusal triggers, and priority constraints, without retraining the target model and without requiring CoT at inference time.
• Data and baseline: Self-built instructions selected from 80 down to 46, covering domains including finance, healthcare, DevOps, and enterprise assistant. Direct extraction ASR was 0%.
• Before hardening: The abstract states structured serialization ASR ≥ 0.7. On YAML in Table 1, GPT-4.1-mini was 0.9565, Gemini-2.5-flash was 0.6087, and IBM Granite 4.0 H-Small FP8 was 0.5000. For the structure category before hardening in Table 3, GPT-4.1-mini was 0.9783 and Granite was 0.5392; for logs and protocols, LLaMA-3-8B was 0.9783.
• After hardening: In Table 2, GPT-4.1-mini's YAML was 0.6304. In Table 3, LLaMA-3-8B's character-level was 0.7826 both before and after. For the same model, Table 2 numbers for Pig Latin, Rot13, and Leetspeak were higher than in Table 1; GPT-4.1-mini's Syslog went from 0.1522 to 0.2117.
• Authors' conclusions: The authors state that this type of leakage differs from prompt injection that overrides system instructions; the model treats serialization as a legitimate transformation. The authors believe that explicit rule hierarchies can reduce leakage, but confidentiality mechanisms that can handle transformed requests and semantic equivalence are needed; partial disclosure may still provide reconnaissance information. The authors interpret prompt-level hardening as risk reduction, not elimination of leakage.