Risk-Certified Prompt Optimization for LLM-as-a-Judge Guardrails
Risk-certified prompt optimization for LLM-as-a-judge guardrails
The authors propose two system prompt optimization procedures, CPS and CRISP, providing finite-sample certification of miss rate for LLM-as-a-judge guardrails; the authors report that test risk does not exceed the target α on four datasets.
Defender setting: a general LLM (e.g., gpt-5.4-mini) is used as an LLM-as-a-judge guardrail, outputting BLOCK/ALLOW solely via the system prompt, without changing model weights; the goal is to maximize the benign pass rate U subject to the constraint that the missed-detection rate on malicious inputs R ≤ α (user-specified), and to require that this constraint hold with probability at least 1−δ on future i.i.d. samples. The protection scenarios are prompt injection, system prompt leakage, and sensitive information leakage.
- LLM-as-a-judge guardrails are controlled solely by the system prompt, whereas standard prompt optimization pursues average performance and cannot guarantee that the missed-detection rate is below a user-specified α. The problem the authors address is: when only binary decisions are exposed and no score threshold is available for tuning, how to select a guardrail prompt with a finite-sample risk certificate.
- Guardrail prompt selection is formalized as constrained optimization: maximize the benign pass rate, constrain the missed-detection rate R≤α. CPS first uses GEPA to generate a fixed candidate pool, then certifies using a Clopper–Pearson upper bound with Bonferroni correction; CRISP feeds risk certification back into iterative rewriting via the Guess-and-Check reusable holdout set, closing the generation gap.
- On the four datasets BIPIA, Deepset, SPL, and SID, the authors report that all certified prompts have a miss rate on the test set no greater than the target α; when α is relaxed, the benign pass rate increases. CPS fails to certify in 4 of 12 prompt injection settings, while CRISP certifies successfully in all; in the Best-of-N diagnostic, CRISP (α=0.10) has a miss rate no greater than 0.07 up to N=100.
- The authors note: the certificate holds only under the i.i.d. calibration distribution, and changes in attack composition, user population, or judge model can invalidate it; it does not cover adaptive adversaries (Best-of-N is only a diagnostic); labeled data is required, and small certification sets make the bound conservative; it is limited to binary, greedy-decoding judges and a single missed-detection constraint. The experiments cover 2 judge models and 4 datasets, and the main table is a single split.
- Evaluated models
- gpt-4o-minigpt-5.4-mini
- Benchmarks
- BIPIADeepset (deepset/prompt-injections)SPLSID (BeaverTails)
- Metrics
- Risk(missed-attack rate,越低越好)Utility(benign pass rate,越高越好)Risk@N(Best-of-N)
The researchers model system prompt selection for LLM-as-a-judge guardrails as a constrained optimization problem: maximize the benign pass rate while keeping the missed-attack rate below a user-specified level α, and provide finite-sample guarantees by combining LLM review, automatic prompt generation, and distribution-free risk certification. The framework provides two procedures: CPS generates a fixed prompt pool from training data, certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds, and selects the certified prompt with the highest benign utility; CRISP incorporates risk feedback into adaptive search, using hard-example-guided rewriting and a reusable held-out set interface to push generation toward the feasible utility frontier. On four datasets covering prompt injection, system prompt leakage, and sensitive information disclosure, risk-certified selection consistently met the target missed-detection level in held-out tests, and utility increased as the allowed risk was relaxed; CRISP is especially useful when fixed-pool search cannot find a certifiable high-utility candidate.
Paper analysis
What problem does this paper attempt to solve?
How to select a prompt for an LLM-as-a-judge guardrail that relies solely on the system prompt, such that the missed-detection rate is below a user-specified α with a statistical guarantee.
Core problem: when the guardrail exposes only binary decisions and no continuous score is available for threshold tuning, how to optimize the system prompt itself so that the missed-attack rate does not exceed a user-specified risk level α with a finite-sample statistical guarantee, while admitting benign requests as much as possible.
- Setting and motivation: An LLM-as-a-judge guardrail uses a general model (the paper gives gpt-5.4 as an example) to output BLOCK/ALLOW via the system prompt, without changing weights, and is cheap to adapt; however, the costs of missing a malicious request and of falsely blocking a benign request are asymmetric, so the average error rate is not a suitable objective (Section 1).
- Shortcomings of existing methods: The authors note that conformal-style risk control requires a continuous score threshold, whereas guardrails in deployment often expose only binary labels, and many hosted models do not provide logits (the Claude API has no logprobs parameter; OpenAI reasoning models refuse logprobs); automatic prompt optimization methods only maximize an unconstrained average score, do not enforce R ≤ α, and give no finite-sample guarantee.
- Core claim: The authors claim that by optimizing the system prompt alone, without changing model weights, a general LLM can be turned into a guardrail with statistical risk control; the problem is formalized as max U(p) s.t. R(p) ≤ α, and the constraint is required to hold with probability at least 1−δ on the calibration data.
- The paper proposes: a risk-certified system prompt optimization framework and the definition of (α, ϵ, δ)-validity (Definition 1), instantiated as two procedures—CPS (optimize first, then certify) and CRISP (risk-feedback closed-loop optimization)—tested on three task types: prompt injection, system prompt leakage, and sensitive information leakage.
- Threat model (defender's perspective):
- Protected object: the input/output guardrail of an LLM system; the victim scenarios are indirect/direct prompt injection (BIPIA, Deepset), system prompt leakage (SPL), and sensitive information leakage (SID).
- Defender's controllable surface: the system prompt only; the judge model uses greedy decoding (temperature=0) and outputs a binary decision.
- Attacker assumption: the certificate applies to samples from the same distribution (i.i.d.) as the calibration data; Best-of-N multi-query attacks are used only as a robustness diagnostic and are outside the scope of the certificate (Section 6).
What related research exists?
Related work falls into three groups: LLM guardrails, automatic prompt optimization, and distribution-free risk control; this paper is positioned as a combination and extension of the three.
LLM system guardrails
- Fine-tuned classifiers: ProtectAI's DeBERTa detector (2024), Prompt Guard (Meta, 2024), PIGuard (Li et al., 2025)—the paper states they are fast and cheap but bound to the training distribution, and new strategies or new attack families require relabeling and retraining.
- Safety-tuned models and orchestrated guardrails: Llama Guard (Inan et al., 2023) and ShieldGemma (Zeng et al., 2024) apply fixed policy taxonomies; NeMo Guardrails (Rebedea et al., 2023) lets developers write safety flows.
- LLM-as-a-judge: Zheng et al. (2023), Gu et al. (2026)—a general model directly makes allow/block decisions according to a system prompt; the paper states this approach is cheap to adapt but sensitive to minor edits (citing Sclar et al., 2024; Wang et al., 2024). The paper notes that in all the above designs the policy prompt is handwritten and then frozen, and none provides a procedure for selecting a prompt under an explicit, statistically enforced risk level.
Automatic prompt optimization
- APE (Zhou et al., 2023), OPRO (Yang et al., 2024), EvoPrompt (Guo et al., 2024), PromptBreeder (Fernando et al., 2023), ProTeGi (Pryzant et al., 2023), MIPROv2 (Opsahl-Ong et al., 2024), GEPA (Agrawal et al., 2025)—search natural-language instructions to improve downstream task performance. The paper notes that they all maximize an unconstrained average score and return a single winner, do not directly enforce R ≤ α, and do not provide a finite-sample guarantee that the constraint is satisfied after deployment; this paper uses them as candidate generators and imposes the constraint itself.
Distribution-free risk control
- Risk-Controlling Prediction Sets (Bates et al., 2021) calibrates a scalar threshold under a monotone loss; Learn-then-Test (LTT, Angelopoulos et al., 2021) drops monotonicity, performs hypothesis tests for each candidate configuration, and corrects with the family-wise error rate (Bonferroni or Holm); Prompt Risk Control (Zollo et al., 2024) uses this framework to select LLM prompts from a fixed candidate set.
- The paper notes that in these works the object being calibrated is a threshold on a continuous score or a candidate set fixed in advance, whereas the guardrail prompt space is discrete, unordered, and open; CRISP's candidates are generated from calibration feedback, so by means of Guess-and-Check it can reuse the holdout mechanism (Dwork et al., 2015; Rogers et al., 2020) to maintain validity. The paper also distinguishes adaptive LTT (Zecchin et al., 2024): the latter fixes hypotheses and tests them on data arriving sequentially, whereas CRISP fixes the data and generates hypotheses sequentially.
Baselines and benchmarks: The experimental baselines are the unoptimized prompt (Naive), the unconstrained optimizers ProTeGi/GEPA/MIPROv2, and the classifier guardrails on prompt injection ProtectAI, PIGuard, Prompt Guard, Model Armor; the datasets are BIPIA, Deepset, SPL, SID (BeaverTails).
The authors position this paper as: extending the LTT-style setting from a fixed candidate family to sequential, adaptive candidate selection, where the tested hypotheses are generated jointly by calibration feedback and the target risk (contribution 3).
How does the paper address this problem?
Formulating guardrail prompt selection as constrained optimization, CPS first fixes the candidate pool and then performs LTT-style certification, while CRISP uses a reusable holdout mechanism to feed risk feedback into iterative rewriting.
Overall approach: the judge model f and the system prompt p form a deterministic binary classifier f(p, x) (greedy decoding, temperature=0); maximize the benign pass rate U(p) subject to R(p) ≤ α, and certify candidate prompts with a distribution-free finite-sample bound. The framework is instantiated as two procedures, CPS and CRISP.
Problem formalization
- Risk and utility definitions: R(p) = Pr(f(p,x)=0 ∣ y=1) (the missed-detection rate, i.e., malicious inputs that are allowed), U(p) = Pr(f(p,x)=0 ∣ y=0) (the benign input pass rate); the objective is max U(p) s.t. R(p) ≤ α (Eq. 3).
- (α, ϵ, δ)-validity (Definition 1): the procedure returns a candidate set and a deployed prompt p̂, and is required, with probability at least 1−δ, to simultaneously satisfy that the risk of all candidates is ≤ α and that U(p̂) is no lower than the best within the candidate set minus ϵ.
- Data splits: Dn is divided into Dtrain (generation/rewriting only), Dcert (certification and selection only), and Dtest (final evaluation only); Dcert is further divided by label into a malicious subset (m examples) and a benign subset (b examples).
- Certification tool: the Clopper–Pearson upper confidence bound Rucb(p, Dcert, β) = Beta−1(1−β; kp+1, m−kp), where kp is the number of missed detections (Eq. 6).
CPS: optimize first, then certify (Section 4.1)
- Draw N bootstrap samples from Dtrain, run a prompt optimizer (e.g., GEPA) once on each, obtaining a fixed candidate pool Ptrain = {p1, …, pN}.
- On Dcert, compute each candidate's risk upper bound at Bonferroni-corrected level δR/N.
- Keep the candidates with Rucb(pj) ≤ α to form the certification set C; if it is empty, abstain (report “–”).
- On the benign subset of Dcert, select the certified candidate with the highest empirical utility. Proposition 1 gives that the probability of R(p̂) ≤ α and near-optimal utility is ≥ 1 − δR − δU.
CRISP: risk-feedback closed loop (Section 4.2)
- Motivation: CPS can only certify prompts already in the pool; if the pool contains no low-risk candidate, certification cannot remedy this (the generation gap); CRISP feeds certification results back into generation.
- Two-population search (borrowing from FI-2pop): in each round, the explored prompts are divided into the certified set Ct and the uncertified set Ut; certified candidates are rewritten toward higher utility, and uncertified candidates toward lower risk.
- At most two proposals per round: ImproveUtility takes the certified prompt with the highest training utility and rewrites it to admit falsely blocked benign samples; ReduceRisk takes the uncertified prompt with the lowest training risk and rewrites it to block missed attacks. Each rewrite is one LLM call, with inputs of the parent prompt, the current risk, the budget α, and hard examples drawn only from Dtrain (template in Appendix A.6).
- Guess-and-Check (GnC) certification: first compute the Clopper–Pearson upper bound on Dtrain; if it exceeds α, return 1 directly without touching the holdout set; otherwise test on the malicious subset of Dcert, and if the training bound is more conservative, release only the training bound; if the test fails, release a holdout estimate coarsened to a γ grid and count one failure; confidence levels are allocated by summable weights cj = 6/(π²(j+1)²), and when the budget is exhausted, return ⊥ and stop.
- Validity under adaptivity: CRISP does not read the malicious subset of Dcert directly, only observing through GnC's restricted outputs; risk depends only on malicious samples, so the benign subset of Dcert remains unused until final selection, and there are at most 2T+1 certified candidates. Proposition 2 gives a guarantee of the same form as CPS (K = 2T+1).
Key parameters
- CPS: N = 20 bootstrap samples, each of size B = 0.20|Dtrain|, GEPA light configuration, 25% internal validation (Appendix A.4).
- CRISP: T = 20 rounds, at most 2 proposals per round; rewriting uses gpt-5.4-mini, temperature 0.7, at most 420 tokens; hard examples are at most 3 false positives + 3 true positives (utility branch) or 3 false negatives + 3 true negatives (risk branch).
- δR = 0.10 throughout; prompt injection uses α ∈ {0.10, 0.20, 0.30}, and output guardrails use α = 0.30.
What experiments did the paper conduct?
On four datasets and two judge models, the authors report that the test risk of certified prompts does not exceed α; CRISP is certifiable in all 12 prompt injection settings, while CPS abstains 4 times.
Experimental setup
- Reviewer models (the guardrails being optimized): gpt-4o-mini and gpt-5.4-mini (Table 1 prompt injection); the output guardrail (Table 2) uses only gpt-5.4-mini. Greedy decoding with temperature=0, at most 5 completion tokens, and unparseable responses default to ALLOW (Appendix A.3).
- Generation/rewriting model: all prompt optimization methods (including baselines) use gpt-5.4-mini, starting from the same task seed prompt; Naive deploys the seed directly.
- Datasets and sizes (Table 4, train/cert/test): BIPIA 600/1800/597; Deepset 132/397/132 (the paper notes this set is smaller, and certification is conservative under tight α); SPL 93/278/92; SID 599/1798/597. The test sets exclude samples rejected by API content filtering (3 for BIPIA, 3 for SID).
- Baselines: Naive, ProTeGi, GEPA, MIPROv2 (all maximizing average accuracy); for prompt injection, additionally the classifier guardrails ProtectAI, PIGuard, Prompt Guard, and Model Armor.
- Metrics: Risk = test-set miss rate (lower is better), Utility = benign pass rate (higher is better); "–" is reported when no candidate can be certified.
- Repetitions and diagnostics: the main table uses a single 0.20/0.60/0.20 split (seed 1); Appendix A.8 re-splits BIPIA 50 times to verify coverage; Figure 3 is the Best-of-N jailbreak diagnostic (30 malicious seeds, gpt-4o-mini reviewer).
Main results
According to Table 1 (prompt injection, test-set Risk/Utility):
Swipe horizontally to read the table
Reviewer model Method BIPIA α=0.1 BIPIA α=0.3 Deepset α=0.1 Deepset α=0.3 gpt-4o-mini CPS 0.06/0.84 0.25/0.88 –/– 0.16/0.96 gpt-4o-mini CRISP 0.02/0.58 0.20/0.86 0.02/0.83 0.11/0.96 gpt-5.4-mini CPS –/– 0.17/0.86 –/– 0.15/1.00 gpt-5.4-mini CRISP 0.01/0.78 0.18/0.89 0.04/0.94 0.09/1.00 gpt-4o-mini Naive / GEPA 0.60/0.90 / 0.33/0.90 same as left (α-independent) 0.29/0.92 / 0.16/0.95 same as left gpt-5.4-mini Naive / GEPA 0.38/0.93 / 0.40/0.95 same as left 0.35/0.97 / 0.15/1.00 same as left – PIGuard / Prompt Guard 0.15/0.86 / 0.13/0.78 same as left 0.08/1.00 / 0.07/0.14 same as left (The α=0.2 rows and ProTeGi, MIPROv2, ProtectAI, and Model Armor are omitted; see Table 1 for full values.)
- Risk calibration holds: the authors report that all certified configurations in Table 1 (2 reviewers × 2 datasets × 3 risk levels × 2 methods) have test miss rates ≤ the target α; the two output guardrail tasks in Table 2 likewise hold at α=0.30.
- α is a usable control knob: the utility of CRISP + gpt-4o-mini rises from 0.58 at α=0.10 to 0.86 at α=0.30 on BIPIA, and from 0.83 to 0.96 on Deepset; under gpt-5.4-mini, BIPIA goes from 0.78 to 0.85 to 0.89, and Deepset from 0.94 to 1.00.
- Compared with unconstrained optimizers: the authors state that certified prompts are often competitive at similar risk, e.g., at α=0.20, BIPIA, gpt-4o-mini, CPS/CRISP give R=0.12/0.11, U=0.86, while ProTeGi gives R=0.19, U=0.80.
Output guardrail tasks (Table 2, α=0.30, gpt-5.4-mini)
- SPL: CPS 0.21/0.78, CRISP 0.08/0.60; Naive 0.17/0.38, ProTeGi 0.15/0.78, GEPA 0.17/0.58, MIPROv2 0.06/0.65.
- SID: CPS 0.15/0.97, CRISP 0.22/0.96; Naive 0.52/1.00 (exceeds α), ProTeGi 0.13/0.93, GEPA 0.12/0.95, MIPROv2 0.13/0.97.
- The authors' interpretation: certified selection does not consistently outperform unconstrained baselines, but consistently returns operating points satisfying α=0.30; the authors consider the main benefit in this setting to be certification feasibility rather than comprehensive frontier improvement.
Generation gap and comparison with classifiers
- Certifiability: CPS returns an empty certification set in 4 of 12 prompt injection settings, while CRISP certifies successfully in all 12 (Section 5.2).
- Comparison with specialized classifiers: the authors state that a general-purpose reviewer can match PIGuard without fine-tuning—at α=0.20 on BIPIA, CRISP (gpt-4o-mini) 0.11/0.86 vs. PIGuard 0.15/0.86; on Deepset, CRISP (gpt-5.4-mini) 0.05/0.99 vs. PIGuard 0.08/1.00, and CPS/CRISP come with finite-sample guarantees.
Best-of-N robustness diagnostic (Figure 3, BIPIA, gpt-4o-mini reviewer)
- What was measured: the attacker queries each seed at most N times, and the attack succeeds if any query is allowed; 30 malicious seeds, with the variant library shared across all defense methods (Appendix A.5).
- Results: the authors report that MIPROv2 reaches a miss rate of 0.8 at N=40, and GEPA reaches 0.8 at N=100; CRISP (α=0.10) stays ≤ 0.07 up to N=100, below the α=0.10 reference line in the figure.
- The authors emphasize: this experiment is a robustness diagnostic, not a certification guarantee; the certificate does not cover multi-query attack settings.
Other ablations and analyses
- Search rounds (Table 5/6, BIPIA, gpt-5.4-mini): at α=0.20, test utility rises from 0.50 at T=1 to 0.83 at T=10, and the selected prompt is unchanged from T=10 to T=40; at α=0.30, T=40 selects a different prompt, with test risk dropping from 0.24 to 0.16 and utility maintained at 0.84.
- Repeated splits (Table 7, BIPIA, 50 re-splits, CRISP, gpt-4o-mini reviewer, T=10): at α=0.20, 49/49 certified splits satisfy Rtest ≤ α (1 abstention), and at α=0.30, 50/50; mean Rtest is 0.09 and 0.15 respectively, and empirical coverage 1.00 exceeds 1−δ=0.90.
- Exceptions: on Deepset, CPS abstains at both α=0.10 and α=0.20 (Table 1); under BIPIA + gpt-5.4-mini, CPS abstains at α=0.10 and α=0.20.
What are some points that can be further explored?
In Limitations, the authors list the conditions under which the certificate applies and the scenarios not covered; the experiments cover two judge models and four datasets, and the main table is a single split.
Limitations and future directions noted by the authors
- Distributional assumption: the guarantee holds under the i.i.d. calibration distribution; changes in attack composition, user population, or reviewer model can invalidate the certificate (Section 6).
- Generation gap: if the generator never proposes a high-utility feasible prompt, certification cannot remedy this—this is precisely why CPS may return an empty certification set (Section 6).
- Does not cover adaptive adversaries: the Best-of-N jailbreak is only a robustness diagnostic, not a certification guarantee (Section 6).
- Data and conservativeness: both procedures require labeled malicious and benign data; when the certification set is small, the Clopper–Pearson and Guess-and-Check bounds are conservative; CRISP may stop early when its reusable holdout budget is exhausted (Section 6).
- Setting limitations and open directions: currently limited to binary, greedy-decoding reviewers and a single miss-rate constraint; stochastic decoding, multi-category policies, and joint control of over-blocking remain open problems (Section 6).
- Positioning: the authors consider certified prompts to be one layer in a defense-in-depth stack, not a replacement for monitoring under distribution shift or adaptive attacks (Section 7).
Scope of the experiments
- The reviewer models tested are gpt-4o-mini and gpt-5.4-mini, 2 in total; the generation/rewriting model is gpt-5.4-mini (Section 5.1, Appendix A.3).
- The datasets are BIPIA, Deepset, SPL, and SID, 4 in total, covering three task types: prompt injection, system prompt leakage, and sensitive information leakage; test set sizes range from 92 to 597 examples (Table 4).
- The main results table is based on a single 0.20/0.60/0.20 split (seed 1); the 50-repeated-split verification is performed only on BIPIA, CRISP, α ∈ {0.20, 0.30}, with the gpt-4o-mini reviewer (Appendix A.8).
- The robustness diagnostic covers only one multi-query attack, Best-of-N, only on BIPIA with the gpt-4o-mini reviewer, with 30 malicious seeds (Appendix A.5).
- The paper does not report human evaluation or the agreement between reviewer judgments and human judgments; SPL's Promptfoo labels were jointly reviewed by two annotators (Appendix A.1).
Summarize the main content of the paper
Guardrail prompt selection is formalized as constrained optimization with finite-sample risk certification; CPS and CRISP achieve test risk not exceeding the target α on four datasets.
A defense paper on LLM-as-a-judge guardrails: under the premise that the guardrail exposes only a binary BLOCK/ALLOW decision and no score threshold is available for tuning, system prompt selection is cast as constrained optimization—maximize the benign pass rate, constrain the missed-detection rate R ≤ α, and give a finite-sample guarantee with probability at least 1−δ.
- Method: CPS uses GEPA to generate a fixed candidate pool from the training data, then certifies on a holdout set using a Clopper–Pearson upper bound with Bonferroni correction, and selects the certified candidate with the highest benign utility; CRISP feeds risk certification back into iterative rewriting via the Guess-and-Check reusable holdout mechanism, searching with two populations—certified and uncertified—toward higher utility and lower risk respectively, closing the generation gap of the fixed pool.
- Risk calibration: The authors report that on all certified prompt injection configurations in Table 1 and on the two output guardrail tasks in Table 2 (α=0.30), the test missed-detection rate does not exceed the target α; across 50 BIPIA re-splits, the certified splits satisfy Rtest ≤ α in 49/49 (α=0.20) and 50/50 (α=0.30) (Table 7).
- α is controllable: For CRISP + gpt-4o-mini, benign utility on BIPIA rises from 0.58 at α=0.10 to 0.86 at α=0.30, with test risk rising accordingly but remaining within the target.
- Certifiability: CPS abstains 4 times in the 12 prompt injection settings, while CRISP certifies successfully in all.
- Comparison baselines: The authors state that a general judge can match the specialized classifier PIGuard without fine-tuning (e.g., on BIPIA at α=0.20, CRISP 0.11/0.86 vs. 0.15/0.86); in the Best-of-N diagnostic, CRISP (α=0.10) has a missed-detection rate ≤ 0.07 up to N=100, whereas MIPROv2 already reaches 0.8 at N=40.
- Authors' conclusion: Certified prompts should serve as one layer of defense in depth; the certificate does not cover distribution shift or adaptive adversaries, and joint control of stochastic decoding, multi-category policies, and over-blocking is left as an open problem.
Related papers
- Hidden Risks of JevAttacksarXiv