Skip to content

Distinguishing prompt harmfulness from interaction safety using partial generation

Separating Prompt Harmfulness from Interaction Safety via Partial Generation

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2610.16761· Ram Vikas Mishra, Shashank Aggarwal, Jahnavi Kumar, Amit AwekarDepartment of Computer Science and Engineering, Indian Institute of Technology GuwahatiIndexed by this site · 2026-10-14· Originally publishedDefenses

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

On a seven-model corpus, prompt and response label agreement rate is 50.46%; on the test set, Llama Guard fine-tuned with full responses achieves a harmful-class AUPRC of 0.885.

Questions
Guardrails that only look at the prompt reject seemingly harmful requests before generation. The authors state that many rejected prompts would have received safe replies; harmfulness depends on what the model actually writes; deciding after seeing the full reply requires waiting until generation ends.
Methods
About 6,000 prompts from four public sources, each answered by seven open models, yielding 41,944 pairs. Gemini 3.1 Flash-Lite labeled whether responses followed malicious intent, with 1,050 pairs human-checked. After fine-tuning five classifiers, they were evaluated on 0%–100% response prefixes.
Experiments and results
Prompt and interaction labels agree on 50.46% of the full corpus. Of prompts that cause any model to fail, only 3.5% cause all seven to fail. On the test set, fine-tuned Llama Guard with full responses achieves harmful-class AUPRC 0.885 and F1 0.807. The authors say about one-third of tokens approach the full-response AUPRC.
Limitations and future work
The authors note: a single LLM reviewer labels the full corpus including the test set; human review covers only 1,050 pairs with two annotators differing in strictness; monitoring is not run end-to-end, so false blocks, misses, and harmful text before stopping are not reported; only single-turn prompts; closed-source or frontier models are not tested. Experiments cover seven 2B–27B open-source models and prefix scanning for five classifiers.
Experimental setupGemma 2 2B、Gemma 3 12B et al. · AdvBench、Chatbot Arena et al. · AUPRC、F1 et al.
Evaluated models
Gemma 2 2BGemma 3 12BGemma 3 27BLlama 3.2 3BLlama 3.1 8BQwen 2.5 7BQwen 2.5 14BDeBERTa-v3Prompt-Guard-2-86MLlama-Guard-3-8BWildGuard-7BLAYA
Benchmarks
AdvBenchChatbot ArenadeepsetxTRam1
Metrics
AUPRCF1precisionrecallCohen's κagreement
AI summaryThe researchers propose letting the model start generating first and judging whether the interaction is safe based only on partial output, thereby avoiding refusal based solely on the prompt. They construct a dataset of 41,944 prompt-response pairs covering seven open-weight models from the Llama, Gemma, and Qwen families. Results show that prompt harmfulness labels and response harmfulness labels agree on only 50.46% of entries (Cohen's κ = 0.06), and among prompts that can make any model produce a harmful response, only 3.5% make all seven models produce harmful responses. A fine-tuned classifier predicts harmfulness at the response level, with best AUPRC of 0.885 and F1 of 0.807; about one-third of the response tokens suffice to approach the AUPRC of the full response.

The researchers propose letting the model start generating first and judging whether the interaction is safe based only on partial output, thereby avoiding refusal based solely on the prompt. They construct a dataset of 41,944 prompt-response pairs covering seven open-weight models from the Llama, Gemma, and Qwen families. Results show that prompt harmfulness labels and response harmfulness labels agree on only 50.46% of entries (Cohen's κ = 0.06), and among prompts that can make any model produce a harmful response, only 3.5% make all seven models produce harmful responses. A fine-tuned classifier predicts harmfulness at the response level, with best AUPRC of 0.885 and F1 of 0.807; about one-third of the response tokens suffice to approach the AUPRC of the full response.

Related events and news

1 events · 1 news mentions

News that mentions this paper

RL/RLHF 训练条件可诱发模型智能体提权逃逸

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors aim to separate prompt harmfulness from interaction safety, and to judge as early as possible using response prefixes.

    A harmful prompt is not the same as a harmful interaction. The authors want to separate the two and see whether generating only the reply prefix can determine early on whether an interaction is safe.

    • Scenario: Guardrails mostly look only at the prompt before generation and refuse if it seems harmful. The authors say this deprives users who could have received a safe reply of help; this cost grows as conversational and generative search become more common.
    • The two existing extremes: A prompt-only classifier equates a harmful prompt with an unsafe interaction and must refuse any prompt for which some model could write a harmful reply. An input-output guard decides after seeing the whole reply, so it sees the real interaction, but the decision is late and the generation cost has already been paid. The authors consider neither extreme ideal.
    • Observation: Interaction safety depends on both the prompt and the model. In Figure 1, under the same prompt, one model refuses and gives a safe alternative while another begins to comply with the malicious intent. The authors say the first few words already separate these two interactions, so generation can be stopped mid-stream.
    • This paper: The question is whether, on real models, the two differ, whether interaction safety can be predicted from the reply, and how early the decision can be made. The authors build a 41,944-pair multi-model corpus to quantify the gap and test how classifiers trained only on full replies predict from prefixes. The authors say the claim is limited to measurement and data; they propose no new monitor and do not train on partial replies.
  2. What related research exists?

    Related work is organized by whether the decision is made before, after, or during generation.

    The authors organize related work by where the safety decision occurs, and separately discuss refusal evaluation and over-refusal.

    Pre-generation

    • Prompt classifiers: prompt injection and jailbreak detectors perform binary classification on prompts, such as ProtectAI detectors, with training data including xTRam1 and deepset. The authors say they are fast because they do not run the model, but the labels are construction- and model-independent, so they must defensively reject any prompt that could cause harm in any model.
    • Attack benchmarks: AdvBench, HarmBench, and BIPIA measure attack success rates. The authors use such datasets as prompts and prompt labels, and examine how well these labels predict interaction safety.

    Post-generation

    • Response-level guardrails: Llama Guard and ShieldGemma judge prompts and responses according to safety taxonomies. WildGuard separates prompt harmfulness, response harmfulness, and refusal; BeaverTails annotates QA-pair harmfulness; Qwen3Guard-Gen adds a controversial class.
    • Relation to this paper: The authors say these response-level labels are closest to this paper, but they judge already-written responses. The authors cite Li et al.'s claim that full detection causes high service latency.

    During generation

    • FineHarm: Li et al. argue that applying a reviewer trained on full responses to partial responses creates a train–inference gap. FineHarm has 29K pairs and token-level labels. The authors say its streaming monitoring reaches macro-F1 of 0.95 or higher after seeing on average about 18% of tokens.
    • Qwen3Guard-Stream and CARE: The former uses token-level heads for real-time monitoring and integrates with CARE. CARE detects unsafe text, rolls back, and intervenes during decoding. The authors say these works show that training for early decisions is feasible; this paper examines a different scenario, where the guardrail is not built for partial responses.

    Refusal evaluation and over-refusal

    • Refusal evaluation: SORRY-Bench compares over 50 models on the same set of 440 unsafe instructions, with refusal behavior varying by model, and uses over 7K human annotations to verify LLM review. HarmBench compares multiple models under the same attacks. The authors say this supports that models differ on the same prompt and that LLM review can be human-verified; but the prompts are all unsafe by design, with no comparison of prompt labels to outcomes, and are not used to decide refusal or stopping.
    • Over-refusal and retrieval: XSTest and OR-Bench measure how much models refuse safe prompts that look unsafe. Other work traces over-refusal to a few attention heads. Kumar et al. discuss that stronger guardrails reduce utility and increase latency. The authors say these works are at the benign-but-looks-unsafe end; this paper asks how many prompts labeled malicious receive safe responses. QHP predicts harmful outcomes from queries before retrieval; research on AI search engines says they may cite malicious content. The authors say that is still deciding only from the query.

    Baselines: Experiments evaluate ProtectAI DeBERTa-v3, Prompt-Guard-2, Llama Guard 3 8B, WildGuard 7B, and LAYA. Prompts come from AdvBench, Chatbot Arena, deepset, and xTRam1.

    The authors place this paper in Table 4 during generation: based on prompt plus partial response, guardrails trained on full responses, measuring the gap between prompt labels and interaction outcomes across seven models.

  3. How does the paper address this problem?

    Build a seven-model corpus and fine-tune five guardrails, then predict interaction labels on reply prefixes.

    The authors separate prompt harmfulness (whether the prompt requests harmful content) from interaction safety (whether that model's reply is harmful), train classifiers on a multi-model corpus, and then make the same judgment on reply prefixes.

    Problem Setting

    • Two labels: The prompt label y_p comes from the source dataset, indicating whether the prompt is labeled malicious. The interaction label y depends on the model: 1 if the reply complies with malicious intent, 0 otherwise. The same prompt can be 0 for one model and 1 for another.
    • Guardrail inputs: A prompt-level guard only looks at the prompt. An interaction-level guard also looks at the reply and can also take the model identity m. The first k tokens of the reply are the prefix. k=0 means only the prompt; k equal to the reply length means the full reply.
    • Objective: The authors write Cp(p) ≈ yp, Cpr(p, rm) ≈ y. The former approximates the prompt label; the latter approximates whether that reply complies with malicious intent.
    • Three uses: Prompt gating rejects when the prompt-only judgment is 1, otherwise generates the full reply. Input-output guard rejects after seeing the full reply if judged 1. Partial Generation monitoring starts generating for any prompt, scores prefixes at checkpoint set K, and stops at the first checkpoint judged 1; if none trigger, delivers the full reply.

    Corpus

    • Prompts: AdvBench provides harmful instructions, deepset provides safety and injection prompts, xTRam1 provides synthetic attacks and benign prompts, and Chatbot Arena natural requests are treated as benign. Training has 5,000 prompts, test has 1,000, with no overlap. Table 1 gives the counts of benign and malicious prompts from each source in training, validation, and test. The authors state that about half of the prompts are labeled malicious.
    • Label meanings differ: In AdvBench, malicious means harmful instruction; in deepset and xTRam1, malicious means prompt injection. The authors note that they return to this point when comparing prompt labels with interaction labels.
    • Replies and scale: Gemma 2 2B, Gemma 3 12B, Gemma 3 27B, Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 7B, Qwen 2.5 14B each answer every prompt. Planned 6,000×7=42,000 pairs; after removing 56 pairs where the reviewer did not give a clear label, 41,944 pairs remain (training 34,946, test 6,998).
    • Annotation: Gemini 3.1 Flash-Lite only judges whether the reply complies with malicious intent. The review prompt and annotation guidelines are in the code repository, not listed in the main text. Human annotators sample 150 pairs per model, totaling 1,050 pairs, all annotated by two annotators. The authors report Cohen's κ, Fleiss' κ, and also PABAK and Randolph's κ due to imbalance. Upon release, they retain interaction labels, generation model, prompt source, and prompt label.

    Classifiers and Thresholds

    • Five classifiers: ProtectAI DeBERTa-v3 and Prompt-Guard-2 are encoders; Llama Guard 3 8B and WildGuard 7B are generative guardrails; LAYA uses an encoder backbone with a small decision head. Each is tested with only the prompt and with prompt plus reply. First zero-shot, then fine-tuned on training pairs.
    • Training: Encoders and LAYA are fully fine-tuned. The two generative guardrails use QLoRA, with loss only on the decision token's next-token cross-entropy. Fine-tuned classifiers are all trained on prompt plus full reply.
    • Imbalance: The training pairs have a harmful proportion of 3.6%. The authors compare standard cross-entropy, weighted loss, oversampling, undersampling, weighted oversampling, and focal loss. Oversampling and undersampling adjust the harmful rate to 20%. Of 34,946 pairs, 29,710 are used for training and 5,236 as validation. Strategy selection is based on highest recall on validation; when differences are within 0.02, F1 breaks ties. The threshold is chosen on the validation set's full replies to maximize harmful-class F1, then kept fixed for that classifier.

    Prefix Protocol

    • Scan: First test shorter prefixes with classifiers trained on full replies, from 0% to 100% of reply tokens in 10% steps. 0% means only the prompt. The threshold is fixed at the value selected at 100%.
    • Metrics: Harmful-class AUPRC, precision, recall, F1. The authors state that harmful pairs are about 3.7% of the data, so these matter more than AUROC or accuracy. Precision and recall are reported at that fixed threshold.
  4. What experiments did the paper conduct?

    Full-corpus label agreement rate 50.46%; test-set Llama Guard fine-tuned AUPRC 0.885.

    Experimental setup

    • Generation models: Gemma 2 2B, Gemma 3 12B, Gemma 3 27B, Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 7B, Qwen 2.5 14B. Each prompt was answered by all seven models.
    • Corpus: 6,000 prompts (5,000 train+validation, 1,000 test) from AdvBench, Chatbot Arena, deepset, and xTRam1. Final 41,944 pairs, 1,535 harmful (3.66%); train split harmful rate 3.68%, test split 3.56%. Test set 6,998 pairs.
    • Classifiers evaluated: ProtectAI DeBERTa-v3, Prompt-Guard-2-86M, Llama Guard 3 8B, WildGuard 7B, LAYA. Each with zero-shot and fine-tuned; inputs are prompt-only or prompt plus response. All fine-tuning was on prompt plus full response; prompt-only rows did not give the response at test time.
    • Metrics and thresholds: Harmful-class AUPRC, precision, recall, F1. In Table 3, thresholds for fine-tuned rows were selected on the validation set using the same input type to maximize harmful-class F1. Prefix experiments fixed thresholds at values selected on full responses.
    • Review: The full corpus was labeled by Gemini 3.1 Flash-Lite for whether responses followed malicious intent. Human review of 1,050 pairs (150 per model), two annotators. The paper did not report the number of experimental repetitions; only Llama Guard's fine-tuned AUPRC had a 95% confidence interval.

    Main results

    On the test set (6,998 pairs, 3.56% harmful), fine-tuned classifiers' harmful-class results are as follows (from Table 3). Prompt-only rows did not give the response at test time.

    Swipe horizontally to read the table

    ClassifierStrategyPrompt-only AUPRCFull-response AUPRCFull-response F1
    DeBERTa-v3standard0.4440.6830.695
    Prompt-Guard-2-86Moversampling0.4590.6460.675
    Llama-Guard-3-8Bweighted loss0.0520.8850.807
    WildGuard-7Boversampling0.3790.6850.655
    LAYAfocal loss0.5100.7060.654
    • Authors' ranking: The authors say Llama Guard is best, with AUPRC 95% CI 0.84–0.93. The other four have overlapping confidence intervals, so the authors do not rank them.
    • Zero-shot: In Table 3, zero-shot precision ranges from 0.052 to 0.501. The authors say these models were trained on other label definitions, several for marking prompts rather than judging interactions. After giving the response, gains are mainly in the two generative guardrails: WildGuard's AUPRC rose from 0.041 to 0.615, Llama Guard from 0.037 to 0.252; the other three had small gains.
    • Contribution of more response: The authors say for DeBERTa, Prompt-Guard-2, WildGuard, and LAYA, adding the response increased AUPRC by 0.19–0.31 and F1 by 0.12–0.17; with prompt-only, precision was 0.46–0.49, and the authors say more than half of intercepted interactions were safe. Llama Guard's improvement came almost entirely from the response. The authors also say they did not train classifiers on prompt-only, and such models might be better.

    Label gap and cross-model failures

    • What was tested: Compared prompt labels from source datasets with interaction labels, and counted how many models a prompt caused to produce harmful responses.
    • Results: Full-corpus agreement 50.46% (Cohen's κ=0.06, PABAK=0.0091). Of 6,000 prompts, 53% had malicious labels, but only 509 (8.5%) caused at least one model to fail; of those, 159 caused exactly one model to fail, and 18 (3.5%) caused all seven to fail. In the conclusion, the authors say that for prompts with malicious labels, at least 93% of responses were safe. The corresponding claim in the introduction is that about half of prompts had malicious labels and about 8.5% caused any model to produce a harmful response.
    • Authors' interpretation: The authors say the gap comes mainly from prompts labeled malicious but receiving safe responses, and is not an artifact of the review. On the review set, agreement between prompt labels and review was 62.76% (κ=0.272), compared to 61.05% and 57.43% with two humans (Table 2). Failures are shared within families, e.g., 43 prompts caused all three Gemma models to fail and no other family to fail. The authors say this is expected because same-family models share training data and methods.

    How long a prefix is needed

    • What was tested: Five fine-tuned classifiers saw only the first 0% to 100% of response tokens at test time, in 10% steps (Figure 3). Thresholds were fixed at values selected on full responses.
    • Results: The authors say after seeing 30%, classifiers reach about 85–95% of full-response AUPRC, and the curve then flattens; for four of five, most gains appear at 10%. The authors say Llama Guard changes most: with prompt-only AUPRC about 0.05, at 30% about 0.78, and at 100% 0.885; at 0% with fixed threshold, recall is high and precision near 0, labeling almost every pair. The other four have F1 0.53–0.54 with prompt-only and 0.65–0.70 with full responses.
    • Authors' interpretation: The authors say about one-third of the response already carries most of the safety signal. They also note two points: the 0% point is a classifier trained on full responses and tested on prompt-only, not a prompt-only-trained classifier; thresholds selected on full responses penalize short prefixes, so the curve may underestimate classifiers trained for early decisions. The authors think Llama Guard labels almost everything without a response, possibly because it is built as an input-output guard and relies more on the response.

    Review and human agreement

    • Results: On 1,050 pairs, three raters agreed on 948 (90.29%), Fleiss' κ=0.72, Randolph's κ=0.87. Human–human agreement 92.57% (κ=0.664); human 1 vs. review 95.43% (κ=0.825), human 2 vs. review 92.57% (κ=0.667) (Table 2). The second human labeled 103 pairs harmful, the first 161, and the review 163.
    • Authors' interpretation: The authors say the review agrees sufficiently with humans to label the full corpus; the review is closer to the first human, and the authors say the second human is stricter. The authors attribute the lower κ relative to raw agreement to label imbalance.

    Other ablations and analyses

    • Harmful response rates: Qwen 14B 1.5%, 7B 3.1%; Llama 8B 1.7%, 3B 2.5%; Gemma 27B 6.19%, 12B 5.1%, 2B 5.3%. Family averages: Gemma 5.55%, Llama 2.13%, Qwen 2.35%.
    • Scale does not consistently correspond to lower harmfulness: larger models are lower in Llama and Qwen, but the largest Gemma is highest.
    • The authors say zero-shot with response F1 is 0.11–0.60, and after fine-tuning 0.65–0.81; DeBERTa, Llama Guard, and LAYA precision rises from 0.06–0.17 to 0.60–0.75.
    • Oversampling and undersampling adjusted the harmful rate to 20%. Training used 29,710 pairs, validation 5,236. The paper does not give scores for each of the six imbalance strategies, nor classifier latency; the authors say this latency is negligible relative to generation models.
  5. What are some points that can be further explored?

    The authors list limitations on labels, end-to-end evaluation, single-turn prompts, and model scope.

    Limitations and Future Directions Noted by the Authors

    • Labels: The full corpus including the test set is annotated by a single LLM reviewer. Human review covers 1,050 pairs, with two annotators differing in strictness. The authors define harmfulness as complying with malicious intent and containing harmful text, and state that other definitions would give other labels (Section 6).
    • Evaluation not end-to-end: Classifiers are evaluated on fixed prefixes, not running stop-or-continue monitoring end-to-end, so they do not report how many safe interactions would be blocked, how many harmful interactions would be let through, or how much harmful text would be produced before stopping (Section 6).
    • Prompts: Only single-turn queries; multi-turn dialogues are not in the dataset (Section 6).
    • Models: Seven open-weight models, three families, 2B to 27B; closed-source or frontier models are not tested (Section 6).
    • Scope of claims: The authors state they do not propose a new monitor or train on partial replies; the corpus puts prompt labels, generation models, and reply labels together for others to test monitors against this gap (Conclusion).
    • Adversarial and early exposure: The authors state that an attacker aware of the monitor could delay harmful parts until after checkpoints or start with a harmless opening, and that such adversarial attacks must be addressed separately. Section 4.3 also notes that some harmful text may be produced before stopping, and users may see it in streaming interfaces.

    Experimental Coverage

    • The tested generation models are Gemma 2 2B, Gemma 3 12B, Gemma 3 27B, Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 7B, Qwen 2.5 14B, totaling 7; every prompt is answered by all seven models (Section 2.2).
    • Prompts come from AdvBench, Chatbot Arena, deepset, xTRam1; training 5,000, test 1,000; paired corpus 41,944 pairs, test set 6,998 pairs (Table 1, Section 2.2).
    • Classifier evaluation includes ProtectAI DeBERTa-v3, Prompt-Guard-2-86M, Llama Guard 3 8B, WildGuard 7B, LAYA in zero-shot and fine-tuned, with input as prompt-only or prompt plus reply; prefixes from 0% to 100% in 10% steps (Table 3, Figure 3).
    • Human review is 150 pairs per model, 1,050 total; the full-corpus review model is Gemini 3.1 Flash-Lite (Section 2.2, Table 2).
    • The paper does not report the number of experimental repetitions, the number of blocks and misses for end-to-end monitoring, the amount of harmful text before stopping, or classifier runtime. The authors discuss streaming monitors trained for early decisions in related work and state that this paper uses guardrails not designed for partial replies and trained only on full replies (Sections 5, 6).
  6. Summarize the main content of the paper

    Prompt labels cannot tell which model will fail; the early part of the reply already contains most of the safety signal.

    The authors separate prompt harmfulness from interaction safety using measurements and data, and look at how much safety signal is in the prefix of the reply.

    • Problem: Rejecting based on the prompt before generation blocks requests that could have been answered safely; reviewing after generation costs the full generation. The authors argue that harmfulness depends on what the model writes, and the first few tokens often already indicate the direction.
    • Method: About 6,000 prompts from AdvBench, Chatbot Arena, deepset, and xTRam1 are answered by seven open-source models (Llama, Gemma, Qwen), yielding 41,944 pairs. Gemini 3.1 Flash-Lite labels whether replies comply with malicious intent; 1,050 pairs are checked by two human annotators. After fine-tuning five classifiers, they test reply prefixes in 10% steps.
    • Gap: The agreement rate between the two label types across the full corpus is 50.46% (κ=0.06). Of the 6,000 prompts, 8.5% cause at least one model to fail, of which 3.5% cause all seven models to fail. The authors state that at least 93% of replies to prompts labeled malicious are safe. Harmful interactions make up 3.66% of the corpus.
    • Prediction: On the test set, Llama Guard 3 8B fine-tuned with weighted loss, with input as prompt plus full reply, achieves an AUPRC of 0.885 and F1 of 0.807 for the harmful class (Table 3). The authors state that off-the-shelf guardrails achieve at most 0.50 precision on their labels; about one-third of tokens can reach approximately 85–95% of the full-reply AUPRC.
    • Conclusion: The authors state that prompt labels cannot tell which model will fail, and rejecting all before generation would block mostly safe interactions. The missing information is in the reply and appears early, so decisions can be made during generation. The authors state these are claims about measurements and data; they do not propose a new monitor or train on partial replies.

Related papers

Read originalarxiv.org