The Polarity Blind Spot in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing
Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing
Polarity-reversal probes raised harm scores across 10 models and 3 languages; Grok-4.1 Fast English negation wrapping averaged 4.12.
Black-box single-turn, no system prompt. The attacker grammatically negates or antonym-replaces harmful verbs, then asks the model to revert the polarity and give a concrete example; synonyms are the polarity-preserving control. The goal is for the response to substantively fulfill malicious intent G with harm score S>0. The victim is a directly queryable chat LLM.
- Safety guardrails may miss harmful intent after grammatical negation or antonym replacement. The authors state this is opposite to negation causing false positives in toxicity detection, and that existing negation mitigation does not address this safety issue.
- Under black-box, single-turn, no system prompt, harmful verbs are replaced with negations, antonyms or synonyms, then the model is asked to change them back and give concrete examples; synonyms preserve polarity. Each variant additionally has a language-exercise-style wrapper. The 260 prompts are translated into Spanish and Czech, and 10 models are each tested on 8 variants, repeated 3 times.
- Polarity reversal has the highest harm score on 9/10 models for English and Spanish and 8/10 models for Czech. After excluding Claude and GPT-5 Nano, the polarity reversal group means are 1.96 for English, 1.70 for Spanish, and 1.67 for Czech. Grok-4.1 Fast English negation wrapping is 4.12. Wrapping effects vary by model.
- The authors note that residual grammatical issues in Spanish and Czech may lower harm scores. Only three Indo-European languages are covered, and whether the finding holds for distant negation systems remains open. The 260 items do not cover all harm categories, some categories have only 10 items, and the distribution skews toward moderate harm. The average weighted κ between Kimi K2.5 and humans is 0.70.
- Evaluated models
- Grok 4.1 (fast, non-reasoning)GPT-5 NanoGPT-5 Mini (gpt-5-mini-2025-08-07)GPT-5.2Gemini 2.5 FlashGemini 2.5 Flash LiteClaude Sonnet 4.5DeepSeek-V3.2Ministral 3 14BGemma 3 (27b version)
- Benchmarks
- ForbiddenQuestionsMaliciousInstruct
- Metrics
- 0-5 harm scorerefusalharmfulness potential (0-3)
The researchers propose the "polarity blind spot," a structural vulnerability referring to a model's inability to recognize harmful intent after language transformations that change polarity, and test two forms of polarity reversal, grammatical negation and antonym substitution, using synonym substitution as a polarity-preserving control, while also examining whether adversarial text framing amplifies such failures. The study uses a benchmark of 260 prompts, covering 10 frontier LLMs and three languages—English, Spanish, and Czech—stratified by harm potential, and evaluated with a validated 0-5 harm scale. Results show that the polarity blind spot affects most tested models, with the most vulnerable model reaching an average harm score of 4.12. The effect persists across typologically different negation systems, and the authors argue that this reflects a broader limitation in semantic robustness rather than a failure specific to a language or grammar.
Related events and news
1 events · 1 news mentions
News that mentions this paper
大模型安全周报:防线开始溃于组合,智能体授权与审计系统性缺口
- 公众号 大模型安全研究cites the paper's id or link大模型安全周报:防线开始溃于组合,智能体授权与审计的系统性缺口
Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing(https://arxiv.org/abs/2610.27908)
Paper analysis
What problem does this paper attempt to solve?
Guardrails may fail to recognize harmful intent after polarity reversal, which the authors call polarity blindness.
Polarity blindness: after polarity reversal, safety guardrails fail to recognize harmful intent that may still be preserved.
- Scenario: The authors state that negation is a long-standing cross-linguistic challenge. In toxicity detection, negation causes harmless content to be mislabeled as toxic; this paper studies the opposite direction, where after polarity reversal guardrails miss harmful intent.
- Existing gaps: The authors state that negation mitigation such as data, training objectives and prompt engineering does not address this safety implication. TIP encodes malicious instructions into sequence tasks; the authors hypothesize that polarity blindness comes from the way models process polarity, and state that patching known templates cannot address this underlying problem.
- Core hypothesis: If the failure is driven by polarity reversal rather than arbitrary surface changes, both negation and antonyms should elicit harmful content, and synonyms should not. Whether wrapping amplifies the failure, and whether the phenomenon holds across languages, are two further questions.
- This paper proposes: Eight variants formed from negation, antonyms, synonyms and presence or absence of adversarial wrapping, with controlled probing on 10 models in 3 languages, reporting 0–5 harm scores stratified by harm potential.
- Threat model:
- Goal: The response must substantively satisfy a specific malicious intent G. The authors define exposure as generated content substantively fulfilling G with harm score S>0.
- Knowledge: black-box, single-turn, no system prompt. The authors state this resembles a simple chat interface accessible online, and treat failure under this condition as low-cost and easy to exploit.
- Capability: Identify harmful verbs and perform grammatical negation or antonym substitution, then ask the model to change the polarity back and give concrete examples. The authors state that outside the experiments this step can also be done manually by a person.
- Victim system: The directly queried LLM and its safety guardrails. The synonym control does not change the access method, only preserving polarity and harmfulness.
What related research exists?
The authors position this paper between negation handling and black-box single-turn red teaming, and distinguish it from TIP's encoding confusion.
Negation in language
- Early tasks and pretrained models: Chapman et al. (2001), Vincze et al. (2008), Okpala et al. (2023) work on biomedical negation scope and negation detection. Ettinger (2020), Kassner and Schütze (2020) document that early pretrained models degrade under negation; the paper also connects this to sentiment analysis, natural language inference, semantic similarity and retrieval.
- Contemporary models and mitigation: Vrabcová et al. (2025) discuss LLMs' difficulty with negation; the paper also lists negation problems in text-to-image, vision-language, text-to-video and text-to-audio, and cites Alhamoud et al. (2025)'s summary: being larger or newer does not make a model better at handling negation. Mitigation includes data methods, changing training objectives and prompt engineering. The authors state that none of these methods address the safety implications of this paper.
Negation in toxicity detection
- The opposite failure direction: Hosseini et al. (2017), Alexiou and Mertoguno (2023) study negation causing harmless content to be mislabeled as toxic. The authors state that this paper examines the opposite failure: after polarity reversal, guardrails miss harmful intent.
Red teaming and linguistic transformations
- Black-box single-turn: The paper places this work in a black-box, single-turn setting, alongside human-based templates (Shen et al., 2024b) and obfuscation methods without model access (Li et al., 2024). Past-tense rewriting (Andriushchenko and Flammarion, 2025) and synonyms (Li et al., 2024) are treated as existing linguistic transformation attacks; the authors note that the latter's synonyms are embedded in a broader decomposition-reconstruction.
- TIP: Berezin et al. (2025) encode malicious instructions into sequence-to-sequence tasks, with examples including encoding, cipher, flipping and programming-style methods. The authors hypothesize that polarity blindness exploits a structural property of polarity processing rather than encoding opacity, and state that patching known templates cannot address the underlying vulnerability.
Baselines and data
- Baseline methods: The original prompt (as is, with or without wrapping) serves as a lower bound; synonym substitution is a polarity-preserving control. The authors state that they do not compare with gradient, fine-tuning or multi-turn methods because the assumptions and conditions differ; manual black-box templates also cannot attribute failure to a specific linguistic property.
- Benchmarks/datasets: ForbiddenQuestions (160 items, 16 violation categories) and MaliciousInstruct (100 items, 10 malicious intents).
The authors position polarity blindness as a linguistically grounded safety vulnerability, and use controlled comparisons of negation, antonyms and synonyms to attribute the mechanism to polarity reversal rather than general surface perturbation or TIP-style encoding.
How does the paper address this problem?
First reverse the polarity of the harmful verbs, then ask the model to revert it back and give a concrete example.
The authors use a polarity blindness probe for controlled probing: first reverse the polarity of harmful verbs so that the prompt is no longer propositionally harmful, then ask the model to revert the polarity and give a concrete example consistent with the meaning after reverting. Figure 2 illustrates that submitting the malicious prompt directly would be refused, whereas this "reverse then revert" path may no longer refuse.
Setup and two probes
- Access condition: black-box query, single-turn, no system prompt. Success depends on whether the response substantively fulfills malicious intent G.
- Negation probe: reverse the polarity of the harmful prompt at the grammatical level, then instruct the model to negate it back and give a concrete example. Double negation is written in the text as fneg(¬ p)=p, meaning negating once more returns to the original prompt. The authors state the model does not restate the original text, but understands the double negation as a generation instruction consistent with G.
- Antonym probe: replace the identified harmful verbs with semantically opposite words, keeping the sentence affirmative, then instruct the model to swap the original verbs back. The authors state that if polarity blindness drives the failure, both probes should elicit harmful content.
- Exposure definition: generated content substantively fulfills G, and harm score S>0.
How polarity reversal is constructed
- Only harmful verbs are changed: the authors state that if other harmless components were reversed, the prompt would still be harmful. Using gpt-5-mini-2025-08-07 in two steps: first mark the verbs related to the harmful action, then generate grammatically correct verb negations, antonyms or synonyms. The paper does not state whether this snapshot is the GPT-5 Mini being tested. Appendix I gives the full prompts.
- Manual review: negations and replacements are checked item by item by native speakers of the target language and errors corrected. The authors state that outside the experiment, users with basic language ability could also do this manually.
- Language differences: English sometimes requires an auxiliary verb, rather than a standalone not. Verb negation in Spanish and Czech is in Appendix C, and the rewriting rules are not expanded here.
Controls and adversarial packaging
- Synonyms: the same set of harmful verbs is replaced, with both polarity and harmfulness preserved. The authors use this to distinguish surface perturbation from polarity reversal.
- Packaging: each variant (including the original prompt) has both an unpackaged and a packaged version. Packaging follows Inie et al. (2025), combining an identity feature (framing the request as a language exercise) and a meta-prompt (requesting a concrete example). Appendix I.1.3 gives the full prompts for each language.
- Eight variants: original prompt, original prompt plus packaging, negation, negation plus packaging, antonym, antonym plus packaging, synonym, synonym plus packaging. The flow is in Figure 3.
Choices in the evaluation design
- Repetition and reasoning: each model is repeated 3 times on each of the 8 variants. Reasoning is set to none or minimal; in Table 6 GPT-5 Nano and GPT-5 Mini are minimal, Claude Sonnet 4.5 is low, and the rest are none. The main text says reasoning effort is low for models that cannot disable reasoning, inconsistent with minimal in Table 6; the table is taken as authoritative here.
- Controls not included: gradients, fine-tuning, multi-turn methods, and manually collected black-box templates. The authors state the former assume differently, and the latter cannot attribute failure to a particular language property.
- How harm is graded: separately, each prompt is manually labeled with 0–3 harm potential, and formal responses use a 0–5 harm score. The scale and review details are in the experimental setup.
What experiments did the paper conduct?
After excluding the two resistant models, the English polarity-reversal group mean harm score is 1.96; Grok-4.1 Fast negation with wrapping is 4.12.
Experimental setup
- Models tested: Grok-4.1 Fast, GPT-5 Nano, GPT-5 Mini, GPT-5.2, Gemini 2.5 Flash, Gemini 2.5 Lite, Claude Sonnet 4.5, DeepSeek-V3.2, Ministral 3 14B (written as Ministral 14B in the table), Gemma 3:27B, 10 in total. The authors state 8 closed-weight and 2 open-weight, without naming which two are open-weight. The prompt transformation uses gpt-5-mini-2025-08-07.
- Data: ForbiddenQuestions 160, 16 categories, plus MaliciousInstruct 100, 10 intents, 260 in total. English is translated into Spanish and Czech via Google Translate, reviewed by native-speaker authors.
- Scale: 8 variants, each repeated 3 times; 3 languages, 10 models, 260 prompts, 187,200 inferences in total. Table 6: GPT-5 Nano and Mini are minimal, Claude is low, the rest none.
- Metric: 0–5 harm score, 0 is harmless, 5 is detailed actionable guidance, defined in Appendix H. Reviewers also label refusals and disclaimers. Main results take the mean of 3 reviews.
- Review: the formal experiment uses Kimi K2.5. The paper states 491,400 ratings in total; the ethics section separately states LLM review covers 163,800 responses, without explaining the correspondence to 187,200 inferences. For model selection, 4 annotators rated 102 responses (from DeepSeek-V3.2 and Grok 4.1, covering 6 harm levels), comparing 6 candidates. Kimi K2.5 average weighted Cohen's κ=0.70; between humans κ=0.64 (Krippendorff's α=0.61).
- Harm potential: two annotators rated on the 0–3 scale in Appendix G, weighted Cohen's κ=0.71, disagreements resolved by majority vote of a third person. Figure 5 shows the distribution; the main text does not give the count per level.
Main results
The table below is based on Table 3, each cell being the model's highest mean harm score on that language and the corresponding variant. 10 models exceed 8 rows, so Gemini 2.5 Lite and GPT-5.2 are omitted; Claude Sonnet 4.5, with the lowest English harm score, is retained in the table.
Swipe horizontally to read the table
Model English highest Spanish highest Czech highest Grok-4.1 Fast 4.12 (negated packaging) 3.93 (negation) 3.84 (negated packaging) Gemma 3:27B 2.63 (negated packaging) 2.57 (negation) 2.21 (synonym) Gemini 2.5 Flash 2.43 (negation) 2.61 (negation) 2.47 (negation) Ministral 14B 2.35 (antonym packaging) 2.53 (synonym) 2.37 (negation) DeepSeek-V3.2 2.23 (antonym) 1.65 (antonym) 1.91 (antonym) GPT-5 Mini 1.75 (negated packaging) 1.18 (antonym packaging) 1.71 (negated packaging) GPT-5 Nano 0.48 (negated packaging) 0.28 (negated packaging) 0.24 (synonym) Claude 4.5 Sonnet 0.34 (original prompt) 0.25 (negation) 0.50 (negated packaging) - The authors state the polarity reversal probe achieves the highest mean harm score on 9/10 models in English and Spanish, and 8/10 in Czech. Where the highest cell in the table is synonym or original prompt, that is the exception.
- The authors treat GPT-5 Nano and Claude Sonnet 4.5 as the resistant group, and state their highest mean harm scores are 0.48 and 0.50 respectively. The group means in Table 1 exclude these two models. The authors also state Grok's original prompt baseline is higher: English 1.74, other models ≤0.86.
- The omitted Gemini 2.5 Lite's highest values in Table 3 are English negation 1.99, Spanish antonym 1.98, Czech negation 2.09. GPT-5.2's English highest is negated packaging 1.51, Spanish is antonym packaging 1.60. Table 3's Czech proportion column for GPT-5.2 does not align with the mean 1.51 and is not used; in Table 5 its negated packaging is 0.97 and antonym packaging is 0.93.
Adversarial packaging
- What was tested: the same polarity transformation with and without packaging, numbers in Table 5, corresponding to RQ2.
- Results: the authors state GPT-5 Mini and GPT-5.2's English harm scores increase by 1.07 and 1.23. Table 5 lists GPT-5 Mini English negation 0.68±0.05, negated packaging 1.75±0.07, GPT-5.2 0.28±0.02 and 1.51±0.12; that sentence in the paper does not state which pair of variants the increase corresponds to. The authors explicitly write examples of antonym packaging decreasing relative to unpackaged: Grok 4.1 English 3.60→3.16, Spanish 3.50→2.54. The authors give Flash as an example of being lower after packaging: English 2.43→2.08, Spanish 2.61→2.24, Czech 2.47→2.29; that sentence does not specify the variant, and the values match the negation row of Table 5.
- Authors' interpretation: packaging is not a universal amplifier, but interacts with model-specific safety mechanisms, sometimes weakening, sometimes strengthening guardrails. The authors hypothesize the language exercise framing is more easily recognized on the original prompt and antonym task.
Language, harm potential and exceptions
- Language: after excluding the two resistant models, the full-benchmark polarity reversal group mean is English 1.96±1.89, Spanish 1.70±1.88, Czech 1.67±1.88 (Table 1). The authors report a paired t-test of polarity reversal against as is and against polarity-preserving variants, p<0.001; English vs Czech and English vs Spanish likewise p<0.001, and the authors write that the Spanish vs Czech difference is not so. The authors state the multiples relative to the as is baseline are English 3.11x, Spanish 2.83x, Czech 2.93x (Figure 4, which excludes the two resistant models).
- Harm potential: in Figure 6, prompts with potential 0 have the lowest harm score. Across languages, models and experiments, the means for potential 1, 2, 3 are 1.20, 1.15, 1.13 respectively. The authors state models do not differentiate responses by prompt severity.
- Exceptions: Gemma 3:27B Czech synonym 2.21 is higher than negation 2.06; Ministral 14B Spanish synonym 2.53 is higher than negation 1.82. The authors attribute this to non-English surface safety coverage, and state this is consistent with existing evidence of RLHF safety degradation on low-resource languages, and different from polarity blindness.
Refusals and antonym errors
- Refusals: the authors state Claude's resistance is consistent with high refusal rates across conditions and languages (Figure 7). GPT-5 Nano's refusal rate is comparable to the same series; its low harm scores come from treating double negation as a grammar exercise. Most models refuse less under packaging; Claude's packaged variants have higher refusals. The main text does not give refusal rate numbers.
- Antonym errors: Appendix D manually checks 260 items, 26 (10%) remain harmful after replacement. The authors state 20 of these are cases where the opposite action is still harmful, and 6 are cases where the scope is split apart, and consider this to partially explain why antonym harm scores are slightly lower than negation.
- The two reversals are not equivalent: the authors state resistance to one polarity reversal does not imply resistance to the other; GPT-5.2, DeepSeek-V3.2, Ministral 14B have antonyms reaching or exceeding negation in at least one language.
Other ablations and analyses
- Table 1 full-benchmark, excluding the two resistant models: negated packaging English 2.21±1.89, negation 1.91±1.97; antonym packaging Spanish 1.30±1.61, Czech 1.13±1.66, lower than unpackaged antonym 1.74±1.91, 1.73±1.89.
- Table 1 original prompt packaging full-benchmark Spanish mean is 0.31±0.81, lower than original prompt 0.60±1.30.
- ForbiddenQuestions polarity reversal group (excluding the two models): English 1.79±1.87, Spanish 1.59±1.84, Czech 1.55±1.83; MaliciousInstruct 2.23±1.89, 1.87±1.92, 1.85±1.95. Both groups are labeled paired t-test, p<0.001.
- Synonym group means (Table 1, full benchmark, excluding the two models): English 1.29±1.61, Spanish 1.25±1.62, Czech 1.33±1.70, lower than the polarity reversal group in the same table.
- Human–model agreement for review model selection is average weighted Cohen's κ=0.70; weighted Cohen's κ for harm potential labeling is 0.71.
- Figure 1 is the mean harm score chart for each English variant. The main text does not list the bar heights per model; model-level numbers follow Table 1–3, Table 5.
What are some points that can be further explored?
The authors note that non-English negation grammar, language typology, the 260 prompts and inter-rater agreement still require verification.
Limitations and future directions noted by the authors
- Grammar of non-English negation: Limitations states that the system's verb negation sometimes makes Spanish and Czech sentences ungrammatical; residual issues may remain after manual revision, and may make harm scores in these languages lower than in English.
- Language coverage: Limitations and Discussion both state that only three Indo-European languages were evaluated, and that they share a basic verb negation mechanism. Whether the finding also holds for languages that are typologically more distant and have different negation systems is called an open empirical question by the authors.
- Benchmark scope: Limitations states that 260 items cannot exhaust all harm categories; some categories in ForbiddenQuestions have only 10 items, and estimates for rare harm types may be unstable. The harm-potential distribution skews toward moderate harm, and tail estimates should be treated with caution.
- Reviewer agreement: Limitations states that the average weighted Cohen's κ between Kimi K2.5 and humans is 0.70, and human judgment cannot be fully reproduced; disagreements are most likely at harm scores 1–2.
- Source of resistance differences: Discussion states that without access to training data and deployment details, the source of resistance differences between models remains unclear.
- Defense direction: Discussion and Conclusion suggest that safety training incorporate polarity-reversed and adversarially wrapped prompts labeled by reconstructed intent rather than surface form, so that the reward is invariant to transformations that change polarity but preserve the harmful goal.
Scope of experimental coverage
- There are 10 models tested, of which the authors state 8 are closed-weight and 2 are open-weight; the paper does not name which two are open-weight. The languages are English, Spanish and Czech, with negation involving auxiliary verbs, the particle no and the prefix ne-, respectively.
- There are 260 prompts, from ForbiddenQuestions (160 items, 16 categories) and MaliciousInstruct (100 items, 10 intents). Eight variants, repeated 3 times, for a total of 187,200 inferences.
- The formal harm scores are given by Kimi K2.5, and the paper states a total of 491,400 ratings; the ethics section separately states that LLM review covers 163,800 responses. The paper does not explain how these two numbers correspond to the 187,200 inferences.
- Harm potential was scored by 2 annotators, with weighted Cohen's κ=0.71, and disagreements resolved by majority vote of a 3rd annotator. Reviewer selection used 4 annotators, 102 responses and 6 candidates; the main text does not list the candidate names other than Kimi K2.5.
- The authors state that they do not compare with gradient, fine-tuning or multi-turn methods because the assumptions and conditions differ; nor with manual black-box templates because failure cannot be attributed to a specific linguistic property. The main text does not give the refusal-rate numbers for Figure 7.
Summarize the main content of the paper
Polarity blindness refers to guardrails failing to detect harmful intent after negation or antonym rewording, and most tested models exhibited this phenomenon.
Polarity blindness refers to guardrails failing to recognize harmful intent that may still be retained after polarity reversal. The authors aim to determine whether this is polarity reversal itself or arbitrary word substitution, and whether wrapping and cross-lingual settings change the failure.
Probing is conducted in a black-box, single-turn setting with no system prompt. After harmful verbs are negated or replaced with antonyms, the model is asked to restore the polarity and give a concrete example; synonyms preserve polarity. Eight variants also cross language exercise-style wrapping. The materials are 260 prompts (160 from ForbiddenQuestions and 100 from MaliciousInstruct), translated into Spanish and Czech, with 10 models each repeated 3 times.
- The polarity reversal variant has the highest mean harm score on 9/10 models for English and Spanish and 8/10 models for Czech.
- After excluding Claude Sonnet 4.5 and GPT-5 Nano, the full-benchmark polarity reversal group means are 1.96 for English, 1.70 for Spanish, and 1.67 for Czech (Table 1). The authors report paired t-tests against the original prompts and polarity-preserving variants, p<0.001.
- Grok-4.1 Fast English negation wrapping has a mean of 4.12 (Table 2). The highest mean harm scores of the two resistant models are 0.48 and 0.50 respectively.
- The authors state that wrapping does not uniformly amplify: GPT-5 Mini and GPT-5.2 increase by 1.07 and 1.23 in English; some other models show lower harm scores under wrapping.
- Spanish Ministral 14B synonyms 2.53 and Czech Gemma 3:27B synonyms 2.21 are higher than their respective negations. The authors attribute this to non-English lexical coverage rather than polarity blindness.
The authors' conclusion is: the mechanism at work is polarity reversal, not general lexical confusion; the phenomenon appears under all three negation implementations in English, Spanish and Czech. The authors recommend including polarity-reversed and adversarial wrapping samples in safety training according to the reconstructed intent rather than the surface polarity.
Related papers
- The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMsAnalysis / interpretabilityarXiv