From Detection to Decision: Calibration and Delayed Decision for Prompt Injection Detection

1. 01What problem does this paper attempt to solve?

   A deployment system must turn detection scores into ALLOW, BLOCK, or DEFER; the authors examine whether native probabilities are better suited for these decisions.

   A deployment system cannot just label prompt injections; it must turn scores into allow, block, or escalate.

   • Scenario: Prompt injection turns untrusted text into instructions that redirect LLM applications. The authors argue that 0.9 is suitable for automated decisions only if high scores consistently correspond to attacks and remain meaningful under changes in attacks, domains, or benign prompts.
   • Gap in existing practice: Detectors are often evaluated as binary classifiers. The authors note that standard selective risk trades coverage against risk using confidence, but in safety, false negatives and false positives have different costs, and escalation goes to an imperfect reviewer, not an oracle.
   • Claim to test: TypeSafe describes Jev as trained with RLCD, outputting probabilities usable for routing and escalation. The authors ask: do these off-the-shelf native probabilities lead to better safety decisions than a specialized classifier with conventional post-hoc calibration on target data?
   • Three questions: RQ1 compares detection performance of Jev, Prompt Guard 2, and an LLM reviewer. RQ2 asks whether native probabilities are more trustworthy on target workloads than raw scores or conventional post-hoc calibration, especially under distribution shift. RQ3 examines how gating uncertainty translates into ALLOW–BLOCK–DEFER, measured by the empirical escalation rate on the confirmation set that achieves system FNR≤1%.
   • Threat model:
     • The goal is to identify prompt injections that would redirect the application and decide ALLOW, BLOCK, or DEFER.
     • Evaluation inputs include human-written direct attacks, indirect injections via external content, hard benign prompts containing attack-like terms, and obfuscation and distribution shifts.
     • The gate outputs an attack probability; uncertain samples are escalated to a frozen imperfect reviewer for the final decision.
     • The paper does not state attacker knowledge (white-box, black-box, or no-box).
   • This work: After freezing prompts, temperature, and gating intervals, Jev, Prompt Guard 2 (22M, 86M), and an LLM reviewer are compared on 47,611 confirmation samples, jointly examining discrimination, calibration, confident false negatives, latency, and escalation behavior.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into prompt injection evaluation, calibration and selective prediction, and probabilistic decision models like Jev.

   The paper organizes related work by evaluation benchmarks, calibration and selective prediction, and probabilistic decision models.

   Prompt injection evaluation

   • TensorTrust (Toyer et al., ICLR 2024): An online game collecting over 126,000 attacks and 46,000 defenses. The paper notes that human-written strategies can transfer beyond the game setting.
   • BIPIA (Yi et al., KDD 2025): Evaluates indirect prompt injection via external content. The paper notes that models struggle to separate data from executable instructions.
   • NotInject and PIDS-Bench: InjecGuard uses NotInject to measure over-defense on benign prompts containing injection-related trigger words (Li and Liu, 2024). PIDS-Bench extends evaluation under a frozen protocol to hard benign inputs, obfuscation, and domain and structural shifts (Shire and Kim, 2026).

   Calibration and selective prediction

   • Temperature scaling (Guo et al., 2017): Adjusts logits using a held-out calibration set while preserving score ordering.
   • Selective classification (Geifman and El-Yaniv, 2017): Uses confidence to trade off automatic coverage against risk. The authors note that this standard setting is insufficient for safety: false negatives and false positives have different costs, and escalated samples are handled by an imperfect reviewer, not an oracle.
   • Calibration under attack shift (Biswas, 2026): Examines calibration of prompt injection detectors under attack shift. The paper only describes this work.

   Probabilistic decision models

   • Jev and RLCD (Almeida, 2026): TypeSafe describes Jev as a System One model trained with Reinforcement Learning for Calibrated Decisions (RLCD) to output probabilities usable for routing and escalation by software. The authors test this claim on a frozen external workload using native probabilities returned by the official API, without post-hoc calibration.

   Comparison setup

   • Systems compared: Jev-native, Llama Prompt Guard 2 22M and 86M (raw probabilities and temperature-scaled), and a GPT-5.5 verbalized-probability reviewer. Benchmarks are TensorTrust, BIPIA, NotInject, and PIDS-Bench v3.

   The authors frame the distinction around the deployment question: whether off-the-shelf native probabilities transfer better than conventional detectors calibrated on target data. The authors state that this comparison does not separate the effects of the underlying training objectives.

   Collapse this answer

3. 03How does the paper address this problem?

   Development and calibration are frozen first, then native probabilities, temperature scaling, and escalation to the frozen reviewer are compared on the confirmation set.

   Overall, this is a leakage-controlled comparison: development and calibration are frozen, then discrimination, calibration, and ALLOW–BLOCK–DEFER are evaluated on the confirmation set for Jev, Prompt Guard 2, and LLM judge.

   Problem setup and isolation

   • Freezing protocol: The detector and review prompts, gating temperature, ALLOW–DEFER–BLOCK intervals, and review thresholds are all frozen before the confirmation evaluation. Figure 1 states that these choices use only development and calibration data.
   • Cohorts: The four benchmark families are selected before the confirmation run. The confirmation set in Table 1 totals 47,611 items: 41,981 attacks and 5,630 benign. The pooled set is intentionally imbalanced; class-conditional metrics are not reported for single-class subsets.
   • Splits: Gating calibration uses 2,000 items; review calibration uses 1,000 items; feasibility check uses 1,000 items. Calibration sets do not overlap with the confirmation set. Confirmation labels are never used to select deployable thresholds.
   • Status of the crossover point: The first point where the entire confirmation escalation curve reaches the target FNR is treated only as a descriptive statistic of the frozen ordering, not as a deployment threshold selected using test labels.

   The three scorers

   • Jev-native: Queries jev-latest via the official TypeSafe SDK, uses a fixed prompt-injection question, retains the returned native probability, and applies no post-hoc calibration.
   • Prompt Guard 2: Evaluates the 22M and 86M versions. Each reports the raw attack probability and a scalar temperature scaling fitted on the gating calibration split. Temperature scaling is monotonic, so both variants have identical AUROC and AUPRC, and identical decisions at the 0.5 boundary.
   • LLM judge: GPT-5.5. Three prompts are compared on a separate 1,000-item development split and one is selected, requiring output of attack probability, binary decision, and brief rationale. Verbalized probabilities are temperature-scaled on another 1,000 items, with temperature frozen at 1.591.
   • Review operating point: The FNR-prioritized operating point is 0.0527. On the calibration split, this threshold yields FNR of 0 and FPR of 98.6%. Appendix B records the prompt name llm-judge-candidate-effect-v1, temperature 1.590979, and threshold 0.052738.

   Calibration and metrics

   • Temperature scaling: Divides the logit by scalar T and applies sigmoid, i.e., p_cal=σ(z/T). T minimizes unweighted binary NLL over [0.05, 20]. When native logits are unavailable, probabilities are first clipped to 10^-7 and 1-10^-7 before converting to logits. No class or sample weights are used.
   • Detection: AUROC, AUPRC, TPR at 1% FPR, and FNR and FPR at the 0.5 boundary. The authors do not treat 0.5 as an optimized operating point; they use it only to see whether the default boundary shifts to the target load.
   • Calibration: ECE uses 10 equal-width bins on [0, 1], weighted by bin sample size; bins are left-closed and right-open except the last. Brier is the unweighted mean squared error of predicted probabilities. NLL is the unweighted binary log loss. False-negative confidence is the average p(attack) over missed attacks; the authors state that low values indicate confident benign misclassifications. Mean latency and p95 latency are also reported.

   Escalation and cost

   • Rule: The gate handles inputs outside the escalation interval automatically; inputs falling into the interval are sent to the frozen LLM reviewer for final decision. Equal probabilities move together.
   • Descriptive quantities: Confirmation escalation requirement is the minimum escalation rate at which the observed system first satisfies FNR≤1% on the frozen ordering's escalation curve. System FPR, automatic coverage, and latency are also reported.
   • Cost: Example expected cost uses (C_FP, C_FN, C_review) = (1, 100, 0.5). The deployable interval is selected only on the gating calibration split.
   • Ablation status: Oracle and fail-closed reviews are only ablations. Appendix B states they are not interpreted as deployed system performance.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On the confirmation set, the LLM judge's AUROC is 0.897; each gate requires escalating over 91.8% to achieve FNR≤1%.

   Experimental setup

   • Models evaluated: jev-latest (Jev-native in tables), Llama Prompt Guard 2 86M, Llama Prompt Guard 2 22M, GPT-5.5 (LLM judge in tables, also the frozen reviewer for RQ3).
   • Confirmation cohort: Table 1 totals 47,611 items. TensorTrust has 526 items, all attacks; BIPIA has 36,128 items (53 benign, 36,075 attacks); NotInject has 89 items, all benign; PIDS-Bench v3 has 10,868 items (5,488 benign, 5,380 attacks). Total benign 5,630, attacks 41,981, attack proportion 88.2%.
   • Isolation: Prompts were selected during development; gating temperature and ALLOW–DEFER–BLOCK intervals were fitted on a 2,000-item gating calibration split; review temperature was fitted on another 1,000 items; plus a 1,000-item feasibility cohort. These sets are disjoint from the confirmation set.
   • Detection metrics: AUROC, AUPRC, TPR at 1% FPR, and FNR and FPR at the default 0.5 threshold. 0.5 is not an optimized operating point.
   • Calibration and review: ECE with 10 equal-width bins, unweighted Brier and NLL, and average p(attack) on false negatives. The review operating point is calibrated probability 0.0527; at that point, FNR is 0 and FPR is 98.6% on the review calibration split.
   • Statistics: The paper reports point estimates, not repetitions or confidence intervals. Single-class subsets report only metrics that remain defined.

   Main results

   Confirmation set detection performance (per Table 2). PG2's raw and temperature-scaled variants are identical on these columns.

   Swipe horizontally to read the table

   Model	AUROC	AUPRC	TPR@1% FPR	[email protected]	[email protected]
   Jev-native	0.871	0.973	0.125	0.649	0.048
   PG2-86M	0.757	0.920	0.012	0.932	0.100
   PG2-22M	0.714	0.908	0.009	0.908	0.143
   LLM judge	0.897	0.977	0.084	0.171	0.130

   • Ranking: The authors state that the LLM judge has the highest pooled AUROC and AUPRC, and Jev is higher than both Prompt Guard 2 versions. The authors do not treat this ranking as universal.
   • Low false-positive interception: The authors state that the TPR@1% FPR column is low, and under this skewed mixture no method supports autonomous interception at low false-positive rates. The authors state that the default 0.5 threshold transfers poorly, with both Prompt Guard versions missing over 90% of attacks.
   • Mixture composition: The authors state that the high AUPRC partly reflects the 88.2% attack proportion; BIPIA contributes most attacks. In this frozen evaluation, NotInject has only benign samples and TensorTrust only attacks. The pooled results summarize only this mixture.

   Calibration and confident misclassification

   Confirmation set calibration (per Table 3). False-negative counts are 27,231 for Jev, 39,119 for PG2-86M, and 7,163 for LLM judge.

   Swipe horizontally to read the table

   Probability source	ECE	Brier	NLL	Mean p(attack) on FN
   Jev-native	0.541	0.440	1.432	0.1630
   PG2-86M raw	0.821	0.817	5.064	0.0107
   PG2-86M calibrated	0.571	0.439	1.104	0.2869
   LLM judge raw	0.221	0.163	0.664	0.0591
   LLM judge calibrated	0.269	0.165	0.550	0.1306

   • Comparison: The authors state that Jev's native probabilities are not uniformly better than temperature-scaled PG2-86M on target data: ECE is slightly lower, Brier is nearly identical, and NLL is higher. The authors state that the calibrated LLM judge has the lowest pooled NLL, and the raw judge has the lowest ECE and Brier.
   • Scaling does not recover misses: The authors state that temperature scaling improves Prompt Guard's ECE, Brier, and NLL, especially NLL, but does not change discrimination, 0.5-threshold decisions, or uncertainty ordering. On attacks missed by raw PG2-86M, mean p(attack) is 0.0107, which the authors state corresponds to a mean benign probability of 98.9%. PG2-22M raw ECE is 0.801, calibrated 0.607, NLL drops from 4.625 to 1.259, false negatives remain 38,135 (Table 3).
   • Reliability diagrams: The authors state that in Figure 2 the diagonal represents perfect calibration, and all systems deviate from it; raw classifier scores deviate most, and post-hoc scaling reduces but does not eliminate the gap. The caption states these curves describe this benchmark mixture and should not be interpreted as calibration under natural deployment prevalence. The curve positions in the figure cannot be confirmed from the text here.

   ALLOW–BLOCK–DEFER

   First point on the confirmation escalation curve where system FNR≤1% is met (per Table 4). Raw and calibrated PG2 are in the same row because temperature scaling preserves uncertainty ordering. This is descriptive statistics, not a deployment threshold selected using confirmation labels.

   Swipe horizontally to read the table

   Gate	Escalation rate	Auto coverage	System FNR	System FPR	Normalized cost
   PG2-86M	91.89%	8.11%	0.998%	40.55%	1.387
   PG2-22M	92.45%	7.55%	0.998%	43.69%	1.394
   Jev-native	92.55%	7.45%	0.943%	45.84%	1.349

   • Escalation requirement: The authors state that each gate requires escalating over 91.8% of inputs before the system reaches FNR≤1%; the three rows differ by less than one percentage point. Without uncertainty intervals, the authors do not interpret this difference as one gate being operationally better. Table 4 also gives average latency at these points: 3,713, 3,734, and 3,885 ms.
   • False positives remain high: The authors state that at these intersections, system FPR is still between 40.6% and 45.8%. The review threshold was chosen FNR-first, with FPR 98.6% on the review calibration split. The authors state that escalating most inputs to that review reduces false negatives but also blocks many benign inputs.
   • Policy implications: The authors state that probability calibration alone does not determine a useful decision policy; reviewer behavior and safety-versus-utility goals remain properties of the deployment system. The Figure 3 caption states the dashed line is the 1% FNR target, and raw and temperature-scaled Prompt Guard curves overlap. The curve heights cannot be confirmed from the text here.

   Latency

   • Local classifiers: Per Table 5, PG2-86M averages 43.8 ms (p50 38.4, p95 69.7); PG2-22M averages 45.8 ms (p50 45.3, p95 76.1).
   • Remote calls: Jev-native averages 190.9 ms (p50 155.2, p95 301.7); LLM judge averages 3,890.2 ms (p50 3,476.0, p95 6,603.9).
   • Authors' interpretation: The authors state that local Prompt Guard is roughly two orders of magnitude faster than the remote LLM judge; Jev is slower than local classifiers and faster than the reviewer. API measurements include network and provider overhead, characterizing the evaluated deployment setting, not pure model computation.

   Other ablations and analyses

   • Jev-native / BIPIA: ECE 0.660, Brier 0.530, NLL 1.723 (Table 6).
   • Jev-native / NotInject: ECE 0.045, Brier 0.009, NLL 0.052 (Table 6).
   • Jev-native / PIDS-Bench v3: ECE 0.167, Brier 0.162, NLL 0.520 (Table 6).
   • PG2-86M raw / BIPIA: ECE 0.984, Brier 0.979, NLL 6.157 (Table 6).
   • PG2-86M calibrated / NotInject: ECE 0.295, Brier 0.098, NLL 0.366; raw ECE on the same set is 0.070 (Table 6).
   • The main text does not report numbers for oracle and fail-closed ablations; Appendix B states they are not interpreted as deployed performance.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note limitations such as hosted interfaces, verbalized probabilities, non-production prevalence, and lack of confidence intervals.

   Limitations and future directions noted by the authors

   • Interface and version: Jev is a hosted model; the evaluation used only one fixed prompt and one API version; the authors state that different phrasings or future model versions may yield different results (Section 6).
   • Probability form: The LLM judge produces verbalized probabilities; the authors state this is not equivalent to probabilities derived from token-level likelihoods (Section 6).
   • Distribution and subsets: The confirmation mixture does not reflect natural production prevalence; pooled calibration and costs should not be directly transferred to deployment. Some subsets have only one class, leaving some class-conditional metrics undefined; these subsets report only metrics that remain meaningful, such as false-positive behavior (Section 6).
   • Cost and latency: The costs (1, 100, 0.5) are examples; the authors state that deployment policies should be chosen based on application-specific costs and review budgets. Latency was measured on a combination of local and API inference, not normalized by controlled hardware (Section 6).
   • Statistics: Only point estimates are reported, without confidence intervals. The authors state that benchmark transformations may make samples dependent, so small numerical differences, especially those under one percentage point in RQ3, should not be considered statistically resolved (Section 6).
   • Adaptation and endpoint: PG2 was temperature-scaled on target calibration data; Jev in the main evaluation deliberately used native probabilities without post-hoc calibration. The authors state that a post-hoc calibration ablation for Jev would help distinguish between transfer of native calibration and whether it can be calibrated after adaptation. The evaluation focuses on detection and routing, not end-to-end agent compromise or downstream harm (Section 6).

   Experimental coverage

   • Models: Evaluated were jev-latest, Llama Prompt Guard 2 22M, Llama Prompt Guard 2 86M, and GPT-5.5 (Section 3.2, Table 2).
   • Data: The confirmation set comprised TensorTrust, BIPIA, NotInject, and PIDS-Bench v3, totaling 47,611 items (Table 1). Gating calibration used 2,000 items, review calibration 1,000 items, plus a 1,000-item feasibility cohort (Section 3.1).
   • Metrics: Discrimination, calibration, average attack probability on false negatives, escalation curves, example costs, and latency were reported (Tables 2–5).
   • Per-dataset: Table 6 in Appendix A gives ECE, Brier, and NLL for Jev and PG2-86M on the four datasets.
   • Not reported: The paper does not report confidence intervals or number of repetitions. Appendix B states that oracle and fail-closed reviews are ablations; the main text does not provide corresponding numbers.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that training only for calibration is insufficient: Jev has higher discrimination than Prompt Guard 2, but still requires escalating over 91.8% to achieve FNR≤1%.

   The authors compare whether detection scores can support allow, block, and escalate on a frozen prompt-injection confirmation set, rather than only binary classification.

   • Problem: Deployed systems must turn scores into ALLOW, BLOCK, or DEFER. The authors ask whether native probabilities trained for calibrated decisions are better suited for these decisions than conventional post-hoc calibration on target data.
   • Approach: jev-latest retains native probabilities from the official API without post-hoc calibration. Prompt Guard 2's 22M and 86M versions report both raw scores and temperature scaling fitted on a 2,000-item gating calibration split. GPT-5.5 selects a prompt on a separate development split, then serves as a frozen reviewer with temperature 1.591 and threshold 0.0527. Confirmation labels are not used to select deployable thresholds.
   • Discrimination: On the 47,611-item confirmation set, LLM judge achieves AUROC 0.897 and AUPRC 0.977; Jev achieves 0.871 and 0.973; PG2-86M achieves 0.757 and 0.920; PG2-22M achieves 0.714 and 0.908 (Table 2). TPR at 1% FPR ranges from 0.009 to 0.125. The authors state that no method supports autonomous blocking at low false-positive rates under this mix.
   • Boundary and calibration: At the 0.5 boundary, Jev's FNR is 0.649, and the two Prompt Guard versions are 0.932 and 0.908. Jev's ECE, Brier, and NLL are 0.541, 0.440, and 1.432; temperature-scaled PG2-86M is 0.571, 0.439, and 1.104 (Table 3). The authors state that native probabilities are not consistently better.
   • Escalation: PG2-86M, PG2-22M, and Jev achieve system FNR≤1% at confirmation-set escalation rates of 91.89%, 92.45%, and 92.55%, with system FPR at 40.55%, 43.69%, and 45.84% (Table 4). The authors state these crossover points are descriptive statistics, not deployment thresholds selected using confirmation labels.
   • Conclusion: The authors state that training only for calibrated outputs is insufficient for deployable safety decisions; discrimination, calibration, reviewer behavior, and policy utility need to be evaluated together in the same decision pipeline.

   Collapse this answer

Related papers