Skip to content

Detectors for safe and reliable large language models: implementation, uses, and limitations

Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations

arXiv· arXiv:2403.06009· Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Kirushikesh DB, Rogério Abreu de Paula, Pierre Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Ja Young Lee, Nishtha Madaan, Sameep Mehta, Erik Miehling, Keerthiram Murugesan, Manish Nagireddy, Inkit Padhi, David Piorkowski, Ambrish Rawat, Orna Raz, Prasanna Sattigeri, Hendrik Strobelt, Sarathkrishna Swaminathan, Christoph Tillmann, Aashka Trivedi, Kush R. Varshney, Dennis Wei, Shalisha Witherspooon, Marcel ZalmanoviciIBM ResearchIndexed by this site · 23 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

IBM Research presents its LLM harm detector library: BERT-like classifiers of about 30M parameters, covering explicit/implicit hate, stigma, prompt injection, etc., used as guardrail and governance tools.

Threat modelThe paper is not an attack paper and has no explicit attacker modeling. Defense-side setting: due to constraints including training cost, API access, and data availability, safety constraints cannot be applied directly to deployed LLMs, so independent small classifiers (detectors) are used to detect various types of harmful text at various stages of the LLM lifecycle (pretraining data filtering, input/output monitoring, evaluation, governance), including prompt-injection-and-jailbreaks-type inputs.

Questions
LLMs carry risks such as toxicity, bias, hallucination, and prompt injection, but due to constraints including training cost, API access, and data availability, safety constraints cannot always be applied directly to deployed models, requiring efficient and reliable alternatives.
Methods
IBM Research built a detector library: for each type of harm, labeled data is collected or synthesized, and supervised fine-tuning is performed on small BERT-like models (about 30M parameters); NAS-compressed architectures, synthetic data augmentation, ensembles, and conformal prediction are used to improve efficiency and reliability, with a supporting red-team UI to collect human feedback.
Experiments and results
The implicit-hate detector achieves F1 of 0.665–0.987 on four test sets (Table 1); the faithfulness detector v2 mix has an average ROC-AUC of 82.2, higher than baselines such as BertScore (Table 2); conformal prediction achieves 90.4% empirical coverage, and after abstaining on 38%, accuracy rises from 0.77 to 0.85 (Appendix D).
Limitations and future work
The authors point out that harm definitions depend on context and that data annotation is inherently subjective, and that detectors may reproduce and amplify harmful classifications; the training data distribution does not match the LLM generation distribution, and performance is poorer on long outputs and evasive responses; multi-turn detection, systematization of jailbreak detection, and long-text attribution are listed as future directions.
Experimental setupimplicit-hate-detector、faithfulness-detector · Latent Hatred、20 NewsGroups et al. · accuracy、balanced accuracy et al.
Evaluated models
implicit-hate-detectorfaithfulness-detector
Benchmarks
Latent Hatred20 NewsGroupsIdentity GroupsCivilCommentsMultiNLISNLIHotPotQANewsQAFRANKSummEvalMNBMQAGS-CQAGS-XBEGINQ2DialFactPAWSAtta-QSocialStigmaQAProvoQ
Metrics
accuracybalanced accuracyprecisionrecallF1ROC-AUCECEempirical coverageabstention rate
AI summaryThe paper presents ongoing work on building and deploying a library of detectors, which are compact, easy-to-build classification models that provide labels for various types of harmful outputs from large language models. The paper notes that, limited by factors such as training cost, API access, and data availability, directly imposing safety constraints on deployed models may not be feasible, and efficient and reliable alternatives are needed. Detectors have a wide range of uses, from serving as guardrails to supporting AI governance; the paper also provides an in-depth analysis of inherent challenges in development and discusses future work to make detectors more reliable and expand their applicability.

The paper presents ongoing work on building and deploying a library of detectors, which are compact, easy-to-build classification models that provide labels for various types of harmful outputs from large language models. The paper notes that, limited by factors such as training cost, API access, and data availability, directly imposing safety constraints on deployed models may not be feasible, and efficient and reliable alternatives are needed. Detectors have a wide range of uses, from serving as guardrails to supporting AI governance; the paper also provides an in-depth analysis of inherent challenges in development and discusses future work to make detectors more reliable and expand their applicability.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    When deployed LLMs cannot be constrained directly, a library of independent small classifiers is used to detect various types of harmful text, and its uses and inherent challenges are discussed.

    Core problem: due to constraints including training cost, API access, and data availability, safety constraints cannot always be applied directly to deployed LLMs, requiring efficient and reliable alternatives to detect and mitigate undesirable LLM behavior.

    • Risk scenarios: the authors state that when LLMs are deployed in critical domains such as healthcare and finance, they produce toxic, biased, and hallucinated content, and that the natural-language-prompt interaction modality is unconstrained, bringing new challenges.
    • Shortcomings of existing approaches: the authors cite Qi et al. 2024 to state that fine-tuning LLMs damages their safety mechanisms, and therefore argue that detectors should be independent of the LLM fine-tuning process; using LLMs to evaluate LLMs suffers from world-knowledge interference, specific biases, and cost issues (the paper paraphrases Shen et al. 2023a, Wang et al. 2023, Samsi et al. 2023).
    • Core proposal: a library of detectors—compact, easy-to-build classification models that label various types of harm, covering the output side (bias, social norms, safety, AI-generated content), the input side (prompt injection, jailbreaks), and the input+output side (unfaithfulness).
    • Threat model (defense perspective): the paper does not model attackers; the defense setting is as follows:
      • Protected objects: the deployed LLM and its inputs and outputs, including prompt-injection-and-jailbreaks-type malicious inputs.
      • Defense layer: external classifiers independent of the LLM, which can be placed at pretraining data filtering, prompt monitoring, response monitoring, or both (Appendix B, Figure 4).
      • Assumptions: detectors have about 30M parameters, and their inference cost is negligible relative to the LLM; the training data distribution and deployment distribution may not match.
    • Deployment context: the detectors have been used in the data and model factory of the IBM Granite series of LLMs, and deployed as moderation in IBM Research's experimental prompt lab (over 25,000 internal users), being tested for entry into the IBM watsonx commercial platform.
  2. What related research exists?

    Related work falls into four groups: harm detection datasets, LLM evaluation and guardrails, uncertainty and calibration, and social science research on content moderation and annotation.

    Harm detection and datasets

    • ElSherief et al. 2021a (Latent Hatred)—implicit hate speech benchmark; Mei et al. 2022a—covert unsafe text; Pachankis et al. 2018—the taxonomy of 93 stigmas in the psychological literature, on which the synthetic data for this paper's stigma detector is based.
    • Zou et al. 2023, Shen et al. 2023b, Hu et al. 2024—adversarial suffixes, in-the-wild jailbreak prompts, and jailbreak detection, cited as related attack and detection work for the prompt-injection-and-jailbreaks detector.
    • Webster et al. 2018 (gender-ambiguous pronouns), Forbes et al. 2020 (social norms), Hu et al. 2023 (RADAR, AI-generated text detection)—corresponding to the gender-ambiguity, social-norms, and AI-generated-text detectors, respectively.

    LLM evaluation, guardrails, and governance

    • Kim et al. 2023, Chiang and Lee 2023, Zheng et al. 2023, Zhu et al. 2023—using LLMs to evaluate LLMs; the paper relays Shen et al. 2023a and Wang et al. 2023 in noting bias problems in such evaluation, and Samsi et al. 2023 in noting that costs may be too high, on which basis the authors position the detectors as an efficient and transparent alternative.
    • Inan et al. 2023 (Llama Guard), Rebedea et al. 2023 (NeMo Guardrails), Dong et al. 2024—LLM output guardrail tools, with which the authors position this paper's detectors as serving the same purpose.
    • Safety benchmarks: Atta-Q (Kour et al. 2023), SocialStigmaQA (Nagireddy et al. 2024a), and BBQ, CrowS-Pairs, StereoSet, BOLD, etc.; the authors state that they used multiple detectors to quantify the proportion of harmful LLM generations on the two internal benchmarks Atta-Q and SocialStigmaQA.

    Uncertainty, calibration, and data augmentation

    • Lakshminarayanan et al. 2017 (deep ensembles), Rahaman et al. 2021—using ensembles to reduce overconfidence; this paper averages 5 randomly initialized models on the implicit-hate detector.
    • Vovk et al. 1999, Romano et al. 2020, Angelopoulos et al. 2021—conformal prediction and regularized adaptive prediction sets, which this paper uses to provide coverage guarantees for the detectors.
    • Gehman et al. 2020, Dhamala et al. 2021, Kour et al. 2023—using prefix continuation or provocative questions to induce LLM generation, on which basis this paper constructs real-world evaluation data and open-sources the ProvoQ dataset.

    Social science research on content moderation and annotation

    • Bowker and Star 1999—classification systems inherently elevate one viewpoint and suppress another; Caplan 2018, Gillespie 2018—context, intent, language, and cultural cues all affect judgments in content moderation.
    • Davani et al. 2023a/b, Sap et al. 2022, Welbl et al. 2021, Aroyo et al. 2023—annotators' identities and perspectives affect toxicity annotation, and annotators often disagree; Section 4 of this paper uses this to argue for the inherent challenges of harm detection.

    Positioning: the authors state that this paper's distinction lies in building and deploying detectors as a library of compact classifiers independent of LLM fine-tuning and spanning the entire LLM lifecycle (data filtering, guardrails, evaluation, governance), and in reflecting on their inherent limitations from a social science perspective.

  3. How does the paper address this problem?

    For each harm category, collect or synthesize annotated data, fine-tune small BERT-like models, and then use NAS, ensembles, conformal prediction, and red-team feedback to improve efficiency and reliability.

    Overall approach: for each risk dimension, first collect or generate labeled data, then supervised fine-tune detectors on BERT-like models, with about 30M parameters, which can be deployed independently at multiple stages of the LLM lifecycle (Figure 1).

    Harm taxonomy

    • Hate and discriminatory language: explicit-hate (HAP), implicit-hate, stigma, gender-ambiguity (pronoun–entity coreference ambiguity).
    • Context and norms: social-norms, blocklisted-topics (taboo topics for specific use cases).
    • Information harms: unfaithfulness (output not faithful to the input context), AI-generated-text.
    • Deception and harmful language: covert-safety (covert incitement of physical harm), prompt-injection-and-jailbreaks.

    Data: synthetic generation and real-world evaluation

    • Synthetic data: for harms lacking annotated data (e.g., stigma), use an LLM with in-context learning to generate data based on the stigma taxonomy in psychological literature (Pachankis et al. 2018), then annotate manually; Appendix C gives the full prompts. Generated text requires further processing and annotation, and the authors state that manual annotation was used.
    • Distribution mismatch problem: the authors note that training data mostly comes from human text and does not match the LLM-generated distribution; to address this, they use prefix continuation or provocative questions to induce multiple LLMs to generate responses, label them as ground truth via human or AI judges (e.g., reward models), and compare with detector labels to find failure modes. This revealed that the detector classifies long outputs and responses from highly aligned, indirectly worded models (e.g., Llama 2) poorly, and long-response training samples were added accordingly.
    • Licensing: the authors state that synthesized and collected data went through an internal clearance process confirming commercial usability; the provocative-question dataset ProvoQ has been open-sourced.

    Efficiency: architecture search

    • Use Neural Architecture Search (Trivedi et al. 2023) to obtain a transformer architecture that the authors state reaches 95% of the accuracy of BERT Base-like models, 7 times faster on CPU and 2 times faster on GPU.

    Reliability: calibration and uncertainty

    • Ensemble: train 5 models with random initialization and average predicted probabilities to mitigate overconfidence.
    • Conformal prediction: use regularized adaptive prediction sets (Romano et al. 2020; Angelopoulos et al. 2021), calibrate conformal scores on a held-out validation set, output prediction sets (e.g., {IMPLICIT-HATE}, {NOT-HATE}, or both), and abstain on non-singleton sets to obtain frequentist coverage guarantees.
    • Negative sample augmentation: add the blocklisting detector's training data (20 NewsGroups, news-headline style, no explicit content) as benign negative samples to training, and plan to introduce drift detection (Ackerman et al. 2021) to identify OOD data.

    Human-in-the-loop and deployment forms

    • Red-team UI: a React+Flask platform (Figure 2, Appendix E Figure 5), where users input or select prompts that previously induced harmful outputs, configure a model to generate responses, can edit responses to remove harmful content, view the detector's sentence-by-sentence scores, and give feedback on misclassifications; feedback targets are obfuscated in the interface to reduce bias. The authors plan to use this feedback to improve detectors via model editing and unlearning.
    • Deployment modes (Appendix B, Figure 4): in single-turn settings, one can monitor the prompt, the response, or both—injection/jailbreak detection monitors the prompt, while unfaithfulness and gender-ambiguity require monitoring both; in multi-turn settings, use the conversation history as context.
    • Uses: pretraining data filtering (the explicit-hate detector is used to filter IBM Granite pretraining data), output guardrails, RLHF-aligned models, reference-free metrics for benchmark evaluation, red-teaming and benchmark construction, governance (behavior compliance, drift tracking, risk assessment of externally procured models).
  4. What experiments did the paper conduct?

    Reports the evaluation of two detectors, implicit-hate and faithfulness, as well as reliability experiments on OOD augmentation, ensembles, and conformal prediction.

    Experimental setup

    • Detectors tested: the implicit-hate detector (fine-tuned bert-base-uncased, batch size 16, learning rate 0.000001, 50 epochs, selecting the best by validation F1) and the faithfulness detector (fine-tuned deberta-v3-large, binary NLI labels); the other detectors are only described without evaluation numbers.
    • Training data: implicit-hate uses 4 datasets—Latent Hatred, 20 NewsGroups (as negative samples), Identity Groups, and the subset of CivilComments with identity attack value greater than 0.5; faithfulness uses MultiNLI, SNLI plus about 22.5k Mixtral synthetic MRQA data items based on HotPotQA and NewsQA.
    • Variants: faithfulness has three variants—v1 (MNLI/SNLI only), v2 (Mixtral synthetic data only), v2 mix (synthetic data + MNLI/SNLI).
    • Baselines: faithfulness is compared with BertScore, BARTScore, FactCC, and ANLI (11B T5, which the authors note is much larger than deberta).
    • Metrics and review: implicit-hate reports accuracy, balanced accuracy, precision, recall, F1; faithfulness reports ROC-AUC. Reliability experiments report ECE, empirical coverage, and abstention rate. The paper does not report the number of repetitions or statistical significance tests.

    Main results

    Performance of the implicit-hate detector on four test sets (according to Table 1):

    Swipe horizontally to read the table

    Test setaccuracybalanced accuracyprecisionrecallF1
    implicit-hate0.7540.7470.6160.7240.665
    blocklisting0.6760.676---
    identity groups0.7520.7320.7290.8910.802
    civil comments0.9740.9741.00.9740.987

    ROC-AUC of the faithfulness detector and baselines (according to Table 2, excerpt):

    Swipe horizontally to read the table

    Datasetv1v2v2 mixBertScoreBARTScoreFactCCANLI (11B)
    FRANK8489.086.784.386.176.489.4
    SummEval69.481.478.377.273.575.980.5
    QAGS-X73.873.779.949.553.864.983.8
    DialFact84.176.289.264.265.655.377.7
    Avg (9 datasets)77.674.882.271.472.266.781.5
    • The authors state that the faithfulness detector has better ROC-AUC than metrics of comparable scale (BertScore, BARTScore, FactCC); according to the average row of Table 2, v2 mix (82.2) is also higher than ANLI (81.5), but ANLI is higher on individual datasets such as FRANK, MNBM, QAGS-X, and BEGIN.
    • v2 (trained only on synthetic data) is clearly lower than v1 and v2 mix on MNBM (53.2) and BEGIN (48.7), and the authors do not explain why.
    • The authors state that the evaluation mainly looks at F1 to balance false positives and false negatives, but from the end-user perspective, false negatives (harmful text judged benign) are more harmful in guardrail mode.

    OOD data augmentation and ensembles (Appendix D)

    • What was tested: performance of the implicit-hate detector on about 5000 all-negative (OOD) items from 20 NewsGroups.
    • Results: the original detector's accuracy is 0.15 (false positive rate 0.85); after adding this data to fine-tuning, it rises to 0.95. Without adding it to training and instead using a threshold of 0.7, the original model is 0.78 and the ensemble model is 0.90.
    • Authors' interpretation: the ensemble, without having seen this data, approaches the version that has seen it (0.90 vs. 0.95), but the authors state that whether the updated detector overfits this data remains to be seen.

    Calibration and conformal prediction (Appendix D)

    • ECE: the original implicit-hate detector has ECE 0.11, and 0.04 after ensembling; the authors state that the ensemble improves calibration.
    • Coverage and abstention: about 4000 items each in the validation and test sets (about 12,000 in the training set), with a target coverage of 90%. The single model has empirical coverage 90.4% with 38% of samples abstained; the ensemble model has coverage 89.9% with 40% abstained.
    • Performance after abstention: the single model on the non-abstained subset has accuracy 0.85 and F1 0.70 (0.77 and 0.67 on the full set); the ensemble model has non-abstained accuracy 0.83 and F1 0.67 (0.77 and 0.65 on the full set). The main text states that conformal improves implicit-hate detection F1 by 4% and the ensemble version by 3%.

    Other ablations and analyses

    • Deployment feedback: after deployment in the experimental prompt lab, the detector showed a high false positive rate (harmless text flagged as harmful), and the authors state that multiple detectors are overconfident toward harm labels (Section 2.4).
    • Failure modes: red-team evaluation found that the detector classifies long outputs inaccurately (the training set is mostly short sentences) and performs poorly on evasive responses from highly aligned, indirectly worded models (e.g., Llama 2) (Section 2.2).
    • Governance application: the authors state that they used multiple detectors to quantify the proportion of harmful LLM generations on the internal benchmarks Atta-Q and SocialStigmaQA, without reporting specific numbers (Section 3.2).
  5. What are some points that can be further explored?

    From a social science perspective, the authors point out inherent challenges in defining and annotating harm, and list directions such as multi-turn detection, systematization of jailbreak detection, and long-text attribution.

    Limitations and future directions noted by the authors

    • Harm definitions depend on context (Section 4, Challenge 1): accurate judgment requires understanding the context, intent, and social conditions of content creation, and detectors have difficulty obtaining this information.
    • Data annotation is inherently subjective (Section 4, Challenge 2): annotators' cultural backgrounds and views influence labels, and disagreements between annotators are common; the authors recommend recruiting annotators from diverse backgrounds and recording disagreements rather than smoothing them over (Recommendation 2).
    • Risk of reproduction and amplification (Section 4.2): the authors believe that if detectors enter commercial platforms or datasets are open-sourced, harmful stereotypes and classifications may be replicated at scale; detectors may perform worse for social groups with different definitions of stigma.
    • Particular difficulties of the stigma detector (Section 4.1): the lexical expression of stigma varies by community and context, and even the absence of explicit expression may imply stigma; the authors believe classifiers are difficult to train reliably without cultural context; they recommend redefining the concept and developing situated detectors for specific communities (Recommendation 1).
    • Multi-turn detection (Section 5): existing detectors are mostly single-turn settings, and the authors state that current work is building a multi-turn harmful dialogue detector based on human-machine dialogue guidelines (Miehling et al. 2024).
    • Systematization and attribution of jailbreak detection (Section 5): the authors state that a unified strategy for detecting jailbreak attacks is currently lacking, and that detection is being extended based on a red-teaming pipeline; the computational cost and interpretability of explanation methods (text span attribution) under long inputs are listed as future work.

    Scope of experimental coverage

    • Only two detectors report quantitative results: implicit-hate and faithfulness (Table 1, Table 2); the remaining detectors (explicit-hate, stigma, covert-safety, prompt-injection-and-jailbreaks, etc.) have only qualitative descriptions.
    • The implicit-hate evaluation covers 4 test sets; the faithfulness evaluation covers ROC-AUC on 9 datasets, with baselines of BertScore, BARTScore, FactCC, and ANLI (11B).
    • The conformal experiment has approximately 4000 validation and test examples each, and the OOD experiment has approximately 5000 test examples (Appendix D).
    • The paper does not report the number of experimental repetitions, statistical significance tests, or quantitative results from the red-team UI user study (the authors state this is left for future user research).
    • The prompt-injection-and-jailbreaks detector is not given any evaluation numbers; multi-turn detection, drift detection, and model editing/unlearning improvements are all planned or ongoing work (Section 2.3, 2.4, 5).
  6. Summarize the main content of the paper

    IBM Research's detector library practice: small classifiers detect LLM harms, serving both as guardrails and as a governance tool, with reflection on their inherent limitations.

    IBM Research describes its ongoing work on building and deploying a library of LLM harm detectors, positioned as a defense and governance tool rather than a new algorithm.

    • Problem: the authors state that due to constraints such as training cost, API access, and data availability, safety constraints cannot always be imposed directly on deployed LLMs; moreover, fine-tuning can damage LLM safety mechanisms (citing Qi et al. 2024), so detection means independent of the LLM are needed.
    • Method: for harms including explicit/implicit hate, stigma, social norms, unfaithfulness, AI-generated text, covert unsafety, and prompt injection and jailbreaks, collect or synthesize annotated data separately and fine-tune small BERT-like models (about 30M parameters); use a NAS architecture (the authors state it maintains 95% of BERT Base's accuracy and is 7 times faster on CPU), synthetic data augmentation, a 5-model ensemble, and conformal prediction to improve efficiency and reliability, with a companion red-team UI to collect human feedback.
    • Results: the implicit-hate detector achieves F1 of 0.665–0.987 on four test sets (Table 1); the faithfulness detector v2 mix averages ROC-AUC 82.2 across 9 datasets, higher than BertScore (71.4), BARTScore (72.2), FactCC (66.7), and ANLI 11B (81.5) (Table 2).
    • Reliability evidence: after adding OOD negative samples, implicit-hate's accuracy on 20 NewsGroups rises from 0.15 to 0.95; the ensemble reduces ECE from 0.11 to 0.04; conformal prediction achieves 90.4% empirical coverage, and after abstaining on 38% of samples, accuracy rises from 0.77 to 0.85 (Appendix D).
    • Deployment: the detectors are used to filter IBM Granite pretraining data and are deployed in an internal prompt lab with more than 25,000 users.
    • Authors' conclusions: the detectors are an efficient and transparent guardrail and governance means, but harm definitions depend on context and annotation is subjective, and deployment may reproduce and amplify harmful classifications; the authors consider acknowledging the plurality of harm definitions the first step to avoiding risk at scale, and list future directions such as multi-turn detection, systematizing jailbreak detection, and long-text attribution.

Related papers

Show 3 more papers
Read originalarxiv.org