Post-training large language models with LLMON

1. 01What problem does this paper attempt to solve?

   LLMs cannot distinguish instructions from data at the token level; the authors use LLMON structured post-training to establish explicit execution binding to defend against indirect prompt injection.

   Core problem: LLMs treat all context tokens alike and have no built-in mechanism to distinguish executable instructions from passive data, leading to prompt injection, instruction-binding ambiguity, and unreliable execution control.

   • Problem scenarios and importance: the authors point out that this flaw manifests as indirect prompt injection hijacking execution, referential binding ambiguity when multiple instructions coexist (e.g., "execute the second task"), and reduced reliability in composite scenarios such as RAG and tool calling.
   • Shortcomings of existing methods: the authors argue that instruction tuning relies on implicit cues such as position, chat templates, and statistical patterns, and lacks a mechanism to distinguish which instruction to execute when multiple candidate instructions are present in the context.
   • Core observation: the authors attribute the above failures to a single root cause—the model cannot distinguish instructions from data at the token level; if execution can occur only through explicit identifier binding, then unreferenced instructions should not be executed.
   • Paper's approach: post-training based on the LLMON (LLM Object Notation) markup framework: build a 4.6B-token structured corpus (including distractor-augmented variants), perform SFT→GRPO→DPO on Granite-4.1-3B/8B bases, and propose three layer-by-layer attention interpretability metrics: IFR, DAS, and LEAK.
   • Threat model:
     • Attacker goal: make the model execute malicious instructions hidden in data payloads (retrieved documents, tool returns), i.e., indirect prompt injection.
     • Attacker knowledge and capability: the paper does not explicitly state a white-box/black-box setting; evaluation covers OPI's five attack types (naive, escape, ignore, fake-completion, combine), ASB's tool-observation injection, and ARES red-teaming's eight strategies (including a multi-turn Crescendo variant).
     • Victim system: Granite-4.1-3B/8B post-trained with LLMON and their Instruct counterparts.
     • Success determination: the proportion of injected instructions executed by the model (ASR); the authors explicitly state that this defense targets indirect injection and offers limited protection against direct and multi-turn attacks.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: instruction fine-tuning and alignment, prompt injection defenses, tool calling, and structured generation; the authors state that it differs from StruQ/SecAlign in boundary carriers and the source of authority.

   Instruction tuning and alignment

   • Wei et al. (2022), Ouyang et al. (2022)—instruction tuning; Rafailov et al. (2023), Christiano et al. (2017)—preference alignment. The authors point out that these methods rely on implicit positional cues and chat templates, and lack a disambiguation mechanism when multiple instructions coexist; this paper is complementary to them and can be layered on any instruction-tuning corpus.

   Prompt injection and adversarial robustness

   • Greshake et al. (2023), Perez and Ribeiro (2022)—prompt injection exploits the lack of privilege separation between instructions and data.
   • Spotlighting (Hines et al., 2024)—marks untrusted segments at the prompt level with in-band text delimiters.
   • StruQ (Chen et al., 2025a) and SecAlign (Chen et al., 2025b)—make boundaries a training objective, fine-tuning on structured prompts and preference pairs. The authors state two differences from this paper: LLMON's boundaries are carried by independent single tokens reserved in the vocabulary and cannot be assembled from ordinary content; privileges attach to explicit bindings rather than role names, so multiple instructions with the same role can coexist but only one is bound for execution.

   Tool use and function calling

   • Toolformer (Schick et al., 2023), Gorilla (Patil et al., 2024), ReAct (Yao et al., 2022)—define tool schemas and train models to produce structured calls, but the authors point out that the instruction-data boundary within the prompt remains implicit; LLMON's execution semantics are orthogonally compatible with tool-calling pipelines.

   Structured generation and constrained decoding

   • Constrained decoding (Willard and Louf, 2023; Zheng et al., 2024; Beurer-Kellner et al., 2023) enforces output structure at inference time with grammar masks; prompt frameworks such as DSPy (Khattab et al., 2023) and POML, and MTP (Dantanarayana et al., 2025), orchestrate calls at the application layer. The authors point out that none of these change the model's internal representation of the instruction-data boundary, whereas LLMON, at the training-data level, lets the model internalize the structural distinction through supervised learning.

   Baselines and benchmarks: the comparison baseline is a Granite-4.1 Instruct model of the same scale (trained on a matched chat-template corpus); the indirect injection benchmarks are OPI and ASB, red-team evaluation uses ARES, and capability evaluation covers 26 benchmarks in seven categories (Section 4.3.1). The authors position this paper as a training-time method, complementary to inference-time safety mechanisms.

   Collapse this answer

3. 03How does the paper address this problem?

   LLMON wraps instructions and data in typed blocks with identifiers, binds execution only through exec pointers, and builds a 4.6B-token corpus with distractors for SFT→GRPO→DPO.

   Overall approach: use LLMON (LLM Object Notation) to serialize the prompt into typed blocks with unique identifiers; execution can occur only through exec blocks that explicitly reference identifiers; then post-train on a structured corpus containing distractors so the model internalizes the behavior of "executing only bound instructions," without modifying the architecture.

   LLMON notation and privilege invariant

   • Six reserved delimiters: <|open|>, <|close|>, <|open_end|>, <|self_close|>, <|:|>, <|.|>, installed as single tokens by remapping unused vocabulary slots, without enlarging the vocabulary; the authors state that this makes the structure impossible to assemble from ordinary subwords.
   • Typed segments: instruction (instr<|:|>ID), data (data<|:|>ID), requirement (req<|:|>ID), document nesting path (doc<|.|>title, doc<|.|>text); tool definitions are flattened into addressable subfields, and tool returns are treated as data blocks; system blocks persist across turns.
   • Execution binding: exec blocks explicitly reference the instruction to execute, the input data, and the output location via exec<|.|>instruction, exec<|.|>input, exec<|.|>output (Figure 1).
   • Privilege invariant: content reaching the model via exec<|.|>instruction or exec<|.|>system is authoritative; content reaching it via exec<|.|>input (data, documents, tool schemas and returns) is merely material to be read, regardless of wording; blocks not referenced by a pointer are rendered into the context but bind nothing.

   Data construction pipeline

   • Three modalities: general instruction data (single-turn/multi-turn dialogue, with per-turn instruction blocks and exec chains), RAG document grounding data (doc segments, where the exec input references both user data and relevant document IDs), and tool-calling data (two exec loops: user instruction→tool call, tool result→final reply).
   • Distractor augmentation: which the authors call the core innovation of the training data—inject d ∈ {1,2,3} competing instructions and m ∈ {0,1,2,3} irrelevant data segments into the context, while the exec block still points only to the original target, and the target output is identical across all variants, for 8 categories in total; models that rely on exec blocks are unaffected by the decoys, while models that rely on surface cues such as position or imperative wording are penalized by the loss.

   Training procedure

   • SFT: full-parameter fine-tuning, 4.4M samples, 4.6B tokens, 3 epochs, sequence length 4096, 8×H100 + FSDP.
   • RL: starting from the SFT checkpoint, five sequential GRPO stages (about 169k prompts, covering multi-domain reasoning, RLHF, knowledge calibration, and two math enhancements), followed by a final DPO stage (about 1k on-policy injection preference pairs, β=0.1, one epoch, learning rate 5×10⁻⁷, effective batch 32); the authors state that DPO is placed last so that stages without an injection objective do not erode the already-established boundary behavior.
   • Corpus composition: general/RAG/tool-calling account for 93.6% of all tokens; Alpaca (52k clean + 103k distractor variants) and Dolly (15k clean + 30k distractor variants) provide distractor augmentation; there are also 130k prompt injection samples (the injected instruction is a block referenced by no execution pointer, and the target output is to ignore it).

   Evaluation and interpretability metrics

   • Success determination: indirect injection benchmarks report ASR (the proportion of injected instructions that are followed, lower is better); ARES red-teaming reports robustness (1−ASR, higher is better).
   • Attention metrics: IFR = (A(I_t))/(A(I_t) + A(I_d)), i.e., the share of attention received by the target instruction out of the attention on all instruction segments; DAS is analogous for target data; LEAK is the share of attention on distractor instructions and distractor data out of the attention on all annotated segments. Computed per layer and averaged over attention heads (for GQA models, averaged within KV groups), used for layer-by-layer failure diagnosis.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On OPI/ASB, ASR drops from 53-100% to 0.02-5.6%, with capability roughly unchanged; ARES direct/multi-turn attack robustness is only 28-37%; attention metrics show that binding forms in the middle-to-late layers.

   Experimental setup

   • Models tested: Granite-4.1-3B-Base and Granite-4.1-8B-Base (bases without instruction-tuning priors), each post-trained with LLMON (SFT→GRPO→DPO); controls are same-scale Instruct models, fully fine-tuned identically on a matched chat-template corpus.
   • Indirect injection benchmarks: OPI (text-level injection, five attack types: naive, escape, ignore, fake-completion, combine) and the observation-injection arm of ASB (injection in tool returns); the metric is ASR, lower is better.
   • Red-team evaluation: ARES mutates 808 seed prompts (from five datasets including Purple, Gandalf, TensorTrust) into about 15k attack prompts via eight strategies; the multi-turn arm is a Crescendo variant, with attack model DeepSeek-V3.2, judge GPT-OSS (120B), and a 20-turn budget; the metric is robustness (1−ASR), higher is better.
   • Capability evaluation: 26 benchmarks in seven categories (General, Alignment, Math, Code, Tool-Calling, Multilingual, Safety), reporting the unweighted average of the benchmarks within each category.
   • Interpretability: IFR/DAS/LEAK computed layer by layer on distractor prompts, analyzing the Granite-4.1-3B model pair.
   • Sample sizes and repetitions: the paper does not report the specific sample size or number of repetitions for each benchmark.

   Main results

   According to Table 2 (ASR%, lower is better) and Table 1 (capability averages):

   Swipe horizontally to read the table

   Model	Training	OPI ↓	ASB ↓	Capability six-category average	Safety
   Granite-4.1-3B	Instruct	100.0	99.2	66.7	80.89
   Granite-4.1-3B	LLMON	0.1	5.6	66.7	89.64
   Granite-4.1-8B	Instruct	96.3	52.8	74.2	84.19
   Granite-4.1-8B	LLMON	0.02	3.1	72.8	90.36

   • The authors state that text-level injection "largely fails," with a drop of about 17× on ASB and about three orders of magnitude on OPI, holding at both scales; the mechanism is structural rather than a learned refusal—the injected instruction sits in a data block referenced by no execution pointer.
   • On capability, the goal is parity: the 3B six-category average is unchanged, the 8B drops by 1.4 points, and the largest single regression is 8B math (84.09→79.63, Table 1); 3B Code and Multilingual improve, and Safety rises by 8.8 and 6.2 points at the two scales respectively (main-text figures).

   Adversarial red-teaming (ARES)

   • What was tested: a red-team campaign independent of the public benchmarks, divided by attack entry point into indirect, direct, and multi-turn categories (Table 3, robustness %, higher is better).
   • Results: indirect attack robustness rises from 2.8% to 84.6% (3B) and from 0.7% to 79.8% (8B); direct attacks are 37.1% (3B) and 32.7% (8B), multi-turn 28.0% (3B) and 37.0% (8B); the indirect arm's ID/OOD split is 90.3/83.6 (3B) and 79.0/75.9 (8B).
   • Authors' interpretation: in direct attacks the adversarial instruction is the user's own instruction, so there is no instruction-data boundary to enforce; multi-turn attackers construct requests that the model itself would accept. The authors state that this invariant should be used in combination with other types of defenses rather than relied on alone.

   Attention interpretability analysis

   • Layer-by-layer metrics (Table 4, 3B pair): the LLMON model has average IFR of 86.5-87.2% and LEAK of 6.3-8.6% at layers 25 and 28, versus 25.8-37.7% and 34.9-44.9% for Instruct; at the last layer (39) both fall back (IFR 28.0% vs 17.2%), from which the authors conclude that execution binding is a middle-to-late-layer computation and that looking only at the last layer underestimates it.
   • Layer-by-layer curves (Figures 2, 3, 5): the authors state that the two models are indistinguishable in the first 20 layers (IFR fluctuates at 25-45% for both), separate from layer 22 onward, and the LLMON model maintains 55-89% at layers 25-38 (peak 89% at layer 28); LEAK exceeds 50% for both in the first six layers, from layer 7 LLMON falls into the 5-25% range and below 15% after layer 25, while Instruct maintains 20-50% throughout; DAS has the same shape as IFR (46-94% at layers 23-38).
   • Segment distribution and token-level flow (Figures 4, 6, layer 31): LLMON model attention is distributed between the target instruction (I-T 23.4%) and the target data (D-T 23.0%), with the largest distractor instruction at only 5.6% (IFR 71.4%, DAS 72.6%, LEAK 9.9%); for Instruct the largest recipient is the distractor instruction I-D1 (47.5%), with the target instruction at only 14.5% (IFR 17.3%, DAS 22.1%, LEAK 49.9%). The authors state that during generation the target data segment activates when the referenced sentence is counted, while distractor segments remain near zero throughout.

   Other ablations and analyses

   • The paper does not report separate ablation or parameter-sensitivity experiments; the layer-by-layer metrics and the ID/OOD split above constitute all additional analyses.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list five limitations: a single model family and scale, token overhead not quantified, weak protection against direct/multi-turn attacks, a corpus that is mainly English, and attention metrics with no causality.

   Limitations noted by the authors and future directions

   • Model coverage (Limitations): evaluation covers only two scales (3B, 8B) of the same model family and the same generation; the authors state that although the method is architecture-agnostic, whether comparable gains transfer to other pretraining corpora, architectures, and larger scales (models with potentially stronger implicit instruction following) remains to be verified.
   • Token overhead (Limitations): the delimiters increase context length by about 7-15% (depending on sample complexity); the authors do not systematically quantify the impact on inference latency and throughput, stating only that sequence packing during training largely amortizes this cost.
   • Scope of protection (Limitations): the privilege invariant targets indirect injection; protection is limited when the adversarial instruction is the user's own instruction (red-team robustness 33-37% for direct, 28-37% for multi-turn, versus 80-85% for indirect), and this is a learned behavior rather than a hard guarantee—the authors state that it greatly reduces but does not eliminate the possibility of instructions within data being followed, and it should be used as one layer of defense rather than the only layer.
   • Language coverage (Limitations): although capability evaluation includes multilingual benchmarks, the LLMON training corpus is mainly English; the authors do not characterize the performance of the markup overhead on morphologically rich languages or very long document tasks.
   • Causality of the metrics (Limitations): IFR/DAS/LEAK provide only correlations between attention patterns and output quality, without establishing a causal mechanism; the authors state that attention weights are an imperfect proxy for transformer information flow, and the relationship between attention-layer metrics and generation-layer behavior may be non-monotonic.

   Scope of experimental coverage

   • The models tested are Granite-4.1-3B-Base and Granite-4.1-8B-Base, 2 in total, with same-scale Instruct models as controls (Section 4.2).
   • Indirect injection evaluation uses two benchmarks: OPI (five attack types) and the observation-injection arm of ASB (Section 4.3.2).
   • Red-team evaluation uses ARES: about 15k attack prompts, eight strategies, a 20-turn budget for the multi-turn arm, attack model DeepSeek-V3.2, judge GPT-OSS (120B) (Section 4.3.3).
   • Capability evaluation covers 26 benchmarks in seven categories; interpretability analysis is performed only on the Granite-4.1-3B model pair (Section 5.4).
   • The paper does not report the sample size, number of evaluation repetitions, or statistical significance tests for each benchmark.

   Collapse this answer

6. 06Summarize the main content of the paper

   LLMON structured post-training teaches Granite-4.1-3B/8B to execute only explicitly bound instructions, reducing indirect injection ASR to 0.02-5.6%, with capability roughly unchanged.

   A defense paper: post-training with the LLMON markup framework so that the model distinguishes instructions from data at the token level, thereby resisting indirect prompt injection.

   • Problem: LLMs treat all tokens alike and cannot distinguish executable instructions from passive data; the authors consider this the common root of indirect prompt injection, instruction-binding ambiguity, and reduced reliability of RAG/tool calling, while existing instruction fine-tuning relies only on implicit positional and template cues.
   • Method: wrap all content in typed blocks with unique identifiers, and execution can occur only through exec blocks that explicitly reference an identifier; build a structured corpus of 4.4M samples and 4.6B tokens (including distractor variants with d≤3 competing instructions and m≤3 segments of irrelevant data, and 130k injection samples), and post-train two Granite-4.1 bases with SFT→five-stage GRPO→DPO.
   • Main results: on OPI, ASR drops from 100.0/96.3% to 0.1/0.02% (3B/8B), and on ASB tool-observation injection from 99.2/52.8% to 5.6/3.1% (Table 2); the six-category capability average is unchanged for 3B (66.7) and drops 1.4 points for 8B, while the two Safety categories rise to 89.64 and 90.36 at the respective scales (Table 1).
   • Red teaming: ARES indirect-attack robustness rises from 2.8/0.7% to 84.6/79.8%, but direct attacks reach only 37.1/32.7% and multi-turn only 28.0/37.0% (Table 3).
   • Mechanism: layer-wise IFR/DAS/LEAK metrics show that the LLMON model concentrates attention on the bound instructions and data at layers 25-38 (at layer 31, IFR 71.4%, LEAK 9.9%), whereas the largest attention recipient in the Instruct model is the distractor instruction (47.5%); the authors state that binding is a computation formed in the middle-to-late layers rather than a surface property of the prompt.
   • Authors' conclusion: explicit execution semantics can be taught through standard post-training without architecture changes, and can serve as a training-time complement to inference-time safety mechanisms; the authors also emphasize that this defense is a learned behavior rather than a guarantee, and that direct and multi-turn attacks require combination with other defenses.

   Collapse this answer

Related papers