Context-Aware Prompt Injection Detection in RAG Pipelines with a Small Language Model: nokast-secureRAG

Core problem: local RAG treats retrieved content as trusted input, indirect prompt injection (IPI) hidden within it is executed by the model as instructions, and existing detectors do not understand whether the query and the retrieved context are consistent.

• Threat realization: the authors cite measurement studies stating that a web-scale measurement verified 15.3 K indirect injection instances across 11.7 K pages, with about 70% hidden in non-rendered HTML (Khodayari et al., 2026); security teams have reported real cases of agents being induced to make payments, delete databases, and exfiltrate data.
• Shortcomings of existing defenses (per the paper): keyword filtering looks only at surface strings and cannot distinguish the same sentence being benign in a manual versus malicious in an injected snippet; general-purpose guardrails such as Llama Guard are trained for direct jailbreaks and content safety, not RAG context consistency; privacy-first local deployments using cloud-based large guardrails contradict the purpose of local inference.
• Core observation: the same imperative sentence (e.g., operating instructions in a tool manual) has opposite safety under different retrieved contexts; the authors call this context-driven judgment reversal a judgment flip, and treat correctly handling flips as the defining capability of a context-aware detector.
• The paper proposes: nokast-secureRAG—an approximately 0.5B-parameter SLM serving as a semantic firewall between the retriever and the generator, jointly reading (Q, C) and outputting a three-class label; accompanied by a teacher→judge→student distillation pipeline, judgment-flip evaluation, and an open-source release.
• Threat model:
  • Attacker goal: override system/user instructions, exfiltrate data, redefine the assistant's role or permissions.
  • Attacker capability: control part of the retrievable content (web pages, documents, memory entries) and embed instructions within it.
  • Victim system: a local RAG pipeline, where retrieved chunks are fused with the user query and fed to the generator.
  • Assumption: the user query itself is benign, and the attack exists only in context C; the defense must run locally within a 16 GB-class VRAM and interactive latency budget.

Prompt Injection and Jailbreak Attacks

• Greshake et al. (2023)—formalizes indirect prompt injection against LLM-integrated applications, demonstrating remote control, data exfiltration, and denial of service; Perez and Ribeiro (2022), Wallace et al. (2019), Zou et al. (2023) demonstrate direct goal hijacking, universal adversarial trigger words, and transferable adversarial suffixes, respectively.
• Liu et al. (2023a)—formalizes prompt injection and benchmarks 5 attack families and 10 defenses; the paper relays its conclusion: many ad hoc filters and instruction-based defenses have limited robustness under stronger attacks; the authors state that this paper is complementary, focusing on a local, dedicated defense layer for the retrieval-mediated subset.

RAG and Knowledge Base Poisoning

• Chan et al. (2024, AgentPoison)—poisons the long-term memory and knowledge base of RAG/agent systems, achieving high attack success rates at low poisoning rates.
• Zou et al. (2024, PoisonedRAG)—injects a small number of constructed passages so that target queries retrieve attacker text and give attacker-chosen answers; this paper uses it for an out-of-distribution check. The paper states that these works inspired two of this paper's four attack families (poisoning and context-hijack).

Benchmarks, Guardrails, and Guardrail Frameworks

• Guo and Wei (2026, OpenRAG-Soc)—a RAG evaluation harness that hides injections in hidden HTML, alt text, etc., with mitigations such as sanitization; Yi et al. (2023, BIPIA) benchmarks indirect injection and evaluates black-box/white-box defenses.
• Llama Guard (Inan et al., 2023)—an input-output guardrail trained on a general safety taxonomy; She et al. (2025) finds that inserting benign retrieved documents flips Llama-Guard's safety judgment in about 8–11% of cases, which the paper uses to show that guardrails themselves are vulnerable to context.
• SafeRAG (Liang et al., 2025) and Zhang et al. (2025)—RAG safety benchmarks; the paper relays their finding that representative RAG components remain vulnerable to obvious attacks; Guardrails AI (2024) and NeMo Guardrails (Rebedea et al., 2023) are programmable guardrail frameworks; GuardBench (Bassani & Sanchez, 2024) evaluates guardrail models across 40 datasets.

Efficient Adaptation and Distillation

• Hinton et al. (2015) knowledge distillation, Hu et al. (2022) LoRA, Wei et al. (2022) chain-of-thought supervision—the training of this paper's student builds on these three.

Baselines and Positioning: Experimental baselines are a regex filter, zero-shot Qwen2.5-0.5B, ProtectAI DeBERTa, Llama Guard 3 1B, and Prompt-Guard-86M; the external benchmark is PoisonedRAG. The authors state that this paper differs from the above work in: a small detector that sits between retrieval and generation, explicitly verifies query–context consistency, is implemented and empirically evaluated, and comes with a dedicated context-flip test (Table 1).

Overall approach: reframe RAG injection detection from "classifying a single piece of text" to semantic context-consistency judgment over (Q, C) pairs, then use teacher→judge→student distillation to compress the large model's judgment capability into a locally deployable 0.5B detector, nokast-secureRAG.

Problem setting and detector form

• Input/output: the detector receives structured input Input = [Q] ∥ [C], first outputs a short reasoning passage and then a label, where the label belongs to {safe, suspicious, malicious-instruction}.
• Design rationale: the model reads Q and C jointly and can judge based on whether instruction-like content in C is consistent with the user's intent, rather than matching keywords on Q.
• Defense requirements (§3): detect instruction-like content in C that conflicts with Q's intent; do not over-flag benign explanatory text; run locally within a 16 GB-class VRAM and interactive latency budget, without relying on cloud guardrails.

Synthetic data generation (teacher)

• Four attack families: indirect prompt injection (ipi), knowledge-base poisoning, context hijacking (role/permission redefinition), PII exfiltration, plus benign documents and ambiguous borderline cases.
• Key constraint: the attack payload of adversarial records is embedded only in C (e.g., HTML comments, footers) and never appears in Q, forcing the model to learn consistency judgment.
• schema-constrained decoding: the inference engine masks any token that would break the JSON schema, ensuring labels are parseable.
• judgment-flip pairs: the same imperative sentence S is placed in a benign context (labeled safe) and an injection context (labeled malicious-instruction); the two halves share the original text of S and are linked by a pair identifier; all flip pairs are kept only in the test set and do not participate in training.

Independent judge review

• Process: a larger model from a different model family sees only (Q, C) and labels blind to the teacher's label, re-annotating each record; consistent ones are kept for training, inconsistent ones are isolated for adjudication.
• Quality signal: the teacher–judge agreement rate itself serves as a dataset quality metric (result: 95.0%).

reasoning-first LoRA fine-tuning (student)

• Configuration: base model Qwen2.5-0.5B, base weights frozen, rank-8 adapters inserted into attention and MLP projections (α=16, dropout 0.05, target modules q,k,v,o,gate,up,down).
• Training objective: "Reasoning: ... Label: ..." format, loss computed only on the completion part; the authors state that reasoning before labeling prompts the model to articulate the Q–C relationship before assigning a label.
• Artifact: a 17.6 MB adapter (or about a 1 GB merged model), deployable on local runtimes such as llama.cpp/Ollama.

Execution and judgment method

• Hardware: all stages run sequentially on a single NVIDIA DGX Spark (121 GB unified memory); teacher and judge cannot reside simultaneously.
• Models used: teacher is Qwen3.6-35B-A3B (NVFP4 MoE, 3B active), judge is Nemotron-3-Super-120B-A12B (NVFP4 MoE, 12B active).
• Success criteria: detection recall, ASR detection-side proxy (the proportion of malicious content not flagged), FPR, three-class accuracy; the flip set additionally reports per-record accuracy and the stricter both-correct rate (both halves of a pair judged correctly).

Experimental Setup

• Systems tested: B1 regex/heuristic filter (context-blind), B2 zero-shot Qwen2.5-0.5B, B3 this paper's LoRA fine-tuned SLM, B4 ProtectAI deberta-v3-base-prompt-injection-v2, B5 Llama Guard 3 1B, B6 Prompt-Guard-86M; B4–B6 take only C as single-text input (context-blind), and B2 fixes the model and scale to isolate the contribution of fine-tuning.
• Data: the teacher generates 5,233 (Q, C) records (safe 3,116 / malicious-instruction 1,917 / suspicious 200; attack types ipi 818, poisoning 500, context-hijack 400, pii-exfil 200, borderline 200, benign 3,115, Table 2); judge agreement rate 95.0% (4,973 agreed, 260 disputed).
• Evaluation splits: detection metrics are computed on 659 test records on which the judge validation agreed; the judgment-flip metric uses 120 complete flip pairs in the test set (240 records, teacher-labeled split, since judge disputes broke up about half of the flip pairs); the student is trained only on the 3,835 agreed training / 479 validation records.
• Metrics: Detection Recall, ASR-proxy (proportion of malicious items not flagged, a detection-side proxy, not downstream generator behavior), FPR, three-class accuracy, latency; the flip set reports per-record accuracy and both-correct rate.
• Hardware and reproducibility: a single DGX Spark (121 GB), fixed random seed 20,260,624, frozen splits and model versions.
• Number of repetitions: not reported in the paper.

Main Results

According to Table 3 (659 judge-validated test records; batch p50 is the per-item time in batch processing):

Swipe horizontally to read the table

System	Recall ↑	ASR-proxy ↓	FPR ↓	3-class acc ↑	batch p50
B1 regex (context-blind)	0.688	0.312	0.129	0.777	< 1 ms
B2 Qwen2.5-0.5B (zero-shot)	0.116	0.884	0.059	0.533	55 ms
B4 ProtectAI DeBERTa (context-blind)	0.888	0.113	0.070	0.901	13 ms
B5 Llama Guard 3 1B (context-blind)	0.437	0.563	0.459	0.487	88 ms
B6 Prompt-Guard-86M (context-blind)	0.900	0.100	0.690	0.586	6 ms
B3 tuned SLM (ours)	0.994	0.006	0.026	0.974	37 ms

• Authors' interpretation: fine-tuning raises the same model's recall from 0.116 to 0.994; B4, as a content single-text baseline, already catches most surface attacks, and B3 on this basis raises recall by a further 10.6 percentage points and more than halves FPR.
• Authors' interpretation: B5 is close to random, which the authors consider consistent with its general safety/jailbreak training—general guardrails cannot replace task-specific detectors; B6 shows the opposite failure mode—recall 0.900 but FPR 0.690, which the authors say is exactly the context-blind over-flagging that this paper's (Q, C) reformulation targets.

Judgment-flip Evaluation

• What was tested: 120 complete flip pairs (240 records), where the same imperative sentence has opposite labels in benign/injection contexts (Table 4).
• Results: B3 both-correct rate 0.750 (per-record 0.875), B1 regex 0.175, B2 0.117, B4 0.542, B5 0.267, B6 0.092.
• Authors' interpretation: context-blind classifiers can only assign the same label to a shared sentence and are structurally limited on this subset; B4's semantic model is about three times regex but still lags B3 by 21 percentage points; B6, despite raw recall of 0.900, is the worst system on flip pairs. The authors call this the most direct evidence for the paper's core claim.

Input Ablation: Does the Model Really Use Q

• What was tested: three input maskings for B3 on the full 659-record test set (Table 5): Q-only (C blanked), C-only (Q blanked), mismatched (C replaced with another row's C, label taken from the donor row).
• Results: Q-only recall drops to 0.000; C-only recall 0.987, FPR 0.038 (full model 0.994/0.026); mismatched recall 0.990 but FPR 0.085, about three times the full model.
• Authors' interpretation: the attack signal is almost entirely in C, and Q's contribution is concentrated in disambiguation scenarios such as flips and mismatches where the content itself is ambiguous; the authors explicitly state this does not mean the model ignores Q on simple samples, nor that every prediction involves deep joint reasoning.

External Benchmark Check: PoisonedRAG (Negative Result)

• What was tested: the PoisonedRAG public knowledge-poisoning benchmark, 300 (query, adversarial-passage) pairs from NQ, HotpotQA, MSMARCO, all malicious, measuring recall only.
• Results: B1 regex 0%, B2 zero-shot 14.3%, B3 after fine-tuning 0.7% (298/300 judged safe), lower than the un-fine-tuned base.
• Authors' interpretation: PoisonedRAG's passages are fluent factual fabrications containing no instructions or role claims, falling outside this paper's instruction-consistency threat model; the authors characterize this as a scope boundary rather than a generalization failure, and state that targeting factual poisoning requires a separate factual-consistency layer.

Other Ablations and Analyses

• Latency (§6.4): single-request median latency is 404 ms with eager transformers (p95 513 ms), dropping to 293 ms (p95 370 ms) with the vLLM optimized runtime, still above the sub-100 ms target; batch throughput is 37 ms/pair.
• Per-class confidence intervals (Appendix A.1): malicious-instruction (n=311) recall 0.994, Wilson 95% CI [0.977, 0.998]; safe (n=342) recall 0.974, CI [0.951, 0.986]; suspicious only n=6, recall 0.167, CI [0.030, 0.564], with 4 of the 5 misses judged as malicious-instruction.
• Lexical overlap check (Appendix A.2): 79.9% of 4-grams in test malicious contexts do not appear in the training set, per-item nearest-neighbor Jaccard max 0.39, with no near-duplicates ≥0.5; reuse of attack templates across splits is by design.
• Cost (§6.4): data generation about 35 Wh, fine-tuning about 9 Wh (14.6 minutes, 94% GPU utilization), adapter 17.6 MB.

Limitations and future directions noted by the authors

• Results are in-distribution only (§8): training and testing come from the same generator; the only feasible external benchmark check, on PoisonedRAG, is a negative result outside the design scope; the core claim (judgment-flip) has only been validated in-distribution.
• ASR is a detection-side proxy (§8): it only measures whether the guardrail flags an attack, not the downstream generator's actual behavior; §9 states that real downstream-ASR measurement is left for future work.
• Adaptive attackers not tested (§8): gradient-based, multi-chunk, retrieval-optimized injections and manual red-teaming were not evaluated.
• Latency target not met (§8, §6.4): latency was measured on a single device only; both the eager and vLLM runtimes exceed the sub-100 ms target; §6.4 states that a label-only variant without the reasoning trace is a candidate path toward sub-100 ms single-request latency, left for future work.
• Data is English-only, single general domain (§8): not validated on financial, medical, or legal documents, where dense instruction-like text may shift the safe/malicious boundary; the authors state that cross-domain validation is a priority.
• Insufficient suspicious-class data (§8): only 6 of the 659 test records carry this label, and the recall of 0.167 is too noisy to be trusted, requiring more suspicious-labeled data; Appendix A.2 also notes that lexical diversity is concentrated in carrier text rather than attack payload wording, and a generator with a richer payload vocabulary is a natural next step.

Scope of experimental coverage

• Systems tested: 6 in total—regex, zero-shot Qwen2.5-0.5B, the fine-tuned SLM, ProtectAI DeBERTa, Llama Guard 3 1B, Prompt-Guard-86M (§5.3), of which the three public baselines all run context-blind on C only.
• Data: a synthetic (Q, C) corpus of 5,233 records, English-only, general domain; detection metrics are based on 659 judge-consistent test records, and flip metrics on 120 pairs (§5.2).
• External benchmarks: only PoisonedRAG, 300 pairs, recall only (§6.6); the paper states that OpenRAG-Soc has no public dataset and SafeRAG is Chinese-only, so they were not used (§8).
• Attack family coverage: training data includes the four families ipi, poisoning, context-hijack, pii-exfil plus borderline (§4.3); fact-confabulation-type poisoning is outside the training scope (§6.6).
• Not reported by the paper: number of repeated experiments, human evaluation, end-to-end attack success rate at the downstream generator, performance under adaptive attacks.

A defense paper: deploys a small semantic firewall between the retriever and the generator for local RAG, detecting indirect prompt injection hidden in retrieved content.

• Problem: IPI hides attack instructions in retrievable content; the authors argue that keyword filtering and general safety guardrails do not understand the consistency between the user query and the retrieved context, and that large cloud guardrails defeat the privacy motivation for local deployment.
• Method: nokast-secureRAG reframes detection as three-way classification of (Q, C) pairs (safe/suspicious/malicious-instruction); a 35B teacher generates 5,233 synthetic data points covering four attack families, a 120B judge from a different family blind-labels and reviews them (agreement rate 95.0%), and then a 0.5B student is fine-tuned with reasoning-first LoRA, all completed on a single 121 GB consumer-grade device.
• Main results: On 659 judge-validated test records, the fine-tuned SLM achieves recall 0.994, FPR 0.026, ASR-proxy 0.006, outperforming regex (0.688), ProtectAI DeBERTa (0.888), Llama Guard 3 1B (0.437), and Prompt-Guard-86M (recall 0.900 but FPR 0.690) (Table 3).
• Context-awareness evidence: On 120 judgment-flip pairs, the rate at which both halves are judged correctly is 75.0%, with the highest baseline at 54.2% (Table 4); input ablations show that masking C reduces recall to zero, and mismatching Q raises FPR from 0.026 to 0.085, on which basis the authors state that Q's contribution is concentrated in disambiguation scenarios.
• Negative result: On the PoisonedRAG factual poisoning benchmark, recall is only 0.7% (below the 14.3% of the un-fine-tuned base model); the authors describe this as a scope boundary outside the instruction-consistency design scope and report it in full.
• Authors' conclusion: The authors state that the experiments support three points—context matters at the margin, distillation lets a 0.5B model achieve judgment ability far beyond zero-shot, and the defense is local and low-cost; they also acknowledge that single-request latency of 293–404 ms is above the sub-100 ms target, and that real downstream-ASR, adaptive attack testing, and cross-domain validation are left for future work.