Llama Guard: an LLM-based input-output guardrail model for human-AI conversations

1. 01What problem does this paper attempt to solve?

   Existing content moderation tools are unsuitable as LLM input-output guardrails, so the authors propose Llama Guard, a customizable LLM-based guardrail model.

   How to build an input-output guardrail for LLM-driven conversational applications, performing safety risk classification separately on the user prompts sent to the model and on the model-generated responses.

   • Problem setting: The authors state that conversational AI applications need to deploy guardrails to guard against high-risk or policy-violating content and jailbreak attempts, and the Llama 2 Responsible Use Guide also recommends guarding all inputs and outputs.
   • Shortcomings of existing tools: The authors point out that online moderation tools such as Perspective API, OpenAI Content Moderation API, and Azure Content Safety API have four shortcomings: they do not distinguish the risks posed by users from those posed by AI agents (the two are different tasks); they enforce only fixed policies and cannot adapt to new policies; they provide only API access and cannot be customized through fine-tuning; and their backbones are smaller traditional transformer models with limited capability.
   • Core idea: Use the LLM itself as the moderation backbone, take the safety taxonomy as model input, and perform classification in the form of an instruction task, so that categories can be adjusted per use case and new policies can be adapted to zero-shot or few-shot.
   • Paper outputs: A set of safety risk taxonomy and risk guidelines, a 13,997-item dataset annotated according to that taxonomy, an instruction-fine-tuned Llama Guard model (based on Llama2-7b), and public release of the model weights.
   • Guardrail positioning (threat model counterpart):
     • Guarded objects: the prompts sent to the LLM in Human-AI conversations (prompt classification) and the LLM-generated responses (response classification).
     • Guardrail goal: determine in real time whether content is safe/unsafe and give the violated taxonomy category, intercepting content with legal and policy risk.
     • Deployment form: an input-output guardrail layer peripheral to the LLM; the authors also acknowledge that, as an LLM, it may be vulnerable to prompt injection attacks (Limitations section).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: LLM zero-shot/few-shot capabilities, moderation of human-generated content, and guardrails for LLM-generated content; the authors state this is the first work to distinguish prompt and response classification.

   Zero-shot and few-shot reasoning of LLMs

   • Brown et al. (2020), Zhou et al. (2023)—studied the few-shot and zero-shot capabilities of LLMs; this paper uses these capabilities to adapt Llama Guard to new policies.
   • Wei et al. (2022a)—instruction fine-tuning framework; this paper follows this paradigm to write classification tasks in instruction form.
   • Touvron et al. (2023)—Llama 2; this paper performs supervised fine-tuning on its 7b version.

   Moderation of human-generated content

   • Halevy et al. (2022)—survey of large-scale web content moderation; user-generated content moderation datasets such as Jigsaw (2017), Twitter (Zampieri et al., 2019; Basile et al., 2019), Stormfront (de Gibert et al., 2018), Reddit (Hada et al., 2021), Hateful Memes (Kiela et al., 2021).
   • The paper points out that guarding LLM-generated content differs: text style and length differ; harm in human content is mostly limited to hate speech, whereas LLM moderation needs to cover a broader range of harms; and prompt-response pairs must be handled.

   Guardrails for LLM-generated content

   • Dinan et al. (2019), Lee et al. (2019), Cercas Curry and Rieser (2018)—dialogue systems may generate inappropriate content or respond inappropriately to offensive content; Dinan et al. (2021) survey the safety landscape and propose a release decision framework.
   • ToxicChat (Lin et al., 2023)—a content moderation benchmark for real user-AI interactions; OpenAI moderation evaluation (Markov et al., 2023)—real-world harmful content detection. The paper points out that both handle only user prompt classification and do not handle LLM-generated outputs.

   Baselines and benchmarks: The comparison baselines are OpenAI Moderation API, Perspective API, Azure AI Content Safety API, and GPT-4 (zero-shot prompting); the evaluation benchmarks are the self-built test set, the OpenAI Moderation Evaluation dataset, and ToxicChat.

   Authors' positioning: The authors state that, to their knowledge, this paper is the first work to split prompt classification and response classification into two independent content moderation tasks, and to achieve this with the same model solely through changes in instruction wording.

   Collapse this answer

3. 03How does the paper address this problem?

   Input-output guardrails are framed as an instruction-following task: the taxonomy is the input, and Llama2-7b is instruction-tuned to output safe/unsafe plus the violation category.

   Overall approach: safety guardrails are modeled as an instruction-following classification task, with Llama2-7b instruction-tuned to produce Llama Guard, and the taxonomy and risk guidelines as model input, thereby supporting policy changes per use case.

   Safety Risk Taxonomy and Risk Guidelines

   • 6 categories: Violence & Hate, Sexual Content, Guns & Illegal Weapons, Regulated or Controlled Substances, Suicide & Self Harm, Criminal Planning, each with textual guidelines defining safe/unsafe (Section 2.1).
   • The authors state that this taxonomy is exemplary and does not aim to cover all potential risks, but is general enough to demonstrate the value of the method.

   Task Construction (Section 3.1)

   • Each task has four elements: a set of guidelines (numbered categories plus textual descriptions), classification type (prompt classification or response classification, distinguished only by instruction wording, performed by the same model), a conversation (single-turn or multi-turn), and output format.
   • Output format: first output "safe" or "unsafe" (both single tokens in the SentencePiece tokenizer); if unsafe, list the violated categories on a new line; categories are denoted by letters plus 1-based indices.
   • This format supports binary classification, multi-label classification, and 1-vs-all classification simultaneously: classifier scores can be read directly from the probability of the first token.

   Zero-shot and Few-shot Adaptation (Section 3.2)

   • Zero-shot: at inference, place the target domain's category names or category names plus descriptions in the prompt.
   • Few-shot: additionally place 2 to 4 examples per category (including hard safe examples as hard negatives), in-context learning only, no training.

   Data Collection (Section 3.3)

   • Based on Anthropic's human preference data on harmlessness (Ganguli et al., 2022), taking the first human prompt from each conversation and using an internal Llama checkpoint to generate responses mixing compliance and refusal.
   • Internal red teaming labeled 4 kinds of labels according to the taxonomy: prompt-category, response-category, prompt-label, response-label, with data cleaning; finally 13,997 entries (Table 1, of which 7,228 are Safe prompts), randomly split 3:1 into fine-tuning set and evaluation set.

   Training and Data Augmentation (Section 3.4)

   • Based on Llama2-7b (the authors state the smallest size was chosen to reduce inference and deployment cost), 8xA100 80GB, batch size 2, sequence length 4096, learning rate 2 × 10−6, trained for 500 steps (about 1 epoch).
   • Two data augmentations: randomly delete categories that are not violated; delete all violated categories and change the label to safe; also shuffle category indices to avoid format memorization, encouraging the model to judge only based on the categories given in the input.

   Evaluation Protocol (Section 4.1)

   • Overall binary classification: for APIs that output probabilities for each category, take the maximum probability across all positive categories as the binary score, ŷ_i = max_{c ∈ {c1,...,cn}} (ŷ_c,i), i.e., positive if any of its own categories is judged positive.
   • 1-vs-all: only the target category ck is placed in the input each time, with that category positive and all others negative, used for Llama Guard's per-category metrics on its own and external datasets.
   • 1-vs-benign: positive samples of other categories are not counted (rather than treated as negatives), leaving only benign negatives; the authors note this removes hard negatives and may produce overly optimistic results, and use it for off-policy evaluation of all baseline APIs.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Comparing 4 baselines on the self-built test set, OpenAI Mod, and ToxicChat, Llama Guard has the highest AUPRC on the self-built set and ToxicChat, and both prompt and fine-tuning adaptation were tested.

   Experimental Setup

   • Subjects: Llama Guard (based on Llama2-7b); baselines are OpenAI Moderation API, Perspective API, Azure AI Content Safety API, and GPT-4 (zero-shot prompting for moderation).
   • Datasets: a self-built test set (the evaluation portion of 13,997 entries after a 3:1 split, Table 1); OpenAI Moderation Evaluation dataset (1,680 prompts); ToxicChat (10k real user-AI interaction samples).
   • Metrics: AUPRC (following Markov et al., 2023); Azure and GPT-4 do not provide probability scores, so AUPRC cannot be computed, and Precision/Recall/F1 at threshold 0.5 are compared separately in the appendix (Tables 5, 6).
   • Adaptation method: Llama Guard runs by default in adapted zero-shot mode, i.e., the target benchmark's taxonomy plus a brief description is placed in the input with no examples; for Azure, the binarization among thresholds 1-6 with the highest average precision is chosen.
   • Review method: each benchmark's own labels (ToxicChat is majority vote of ≥3 annotators); the paper does not use LLM review.
   • The paper does not report number of repetitions or confidence intervals.

   Main Results

   According to Table 2 (AUPRC, higher is better; Llama Guard is zero-shot prompted with the target taxonomy):

   Swipe horizontally to read the table

   Method	Self-built test set (prompts)	OpenAI Mod	ToxicChat	Self-built test set (responses)
   Llama Guard	0.945	0.847	0.626	0.953
   OpenAI API	0.764	0.856	0.588	0.769
   Perspective API	0.728	0.787	0.532	0.699

   • The authors state that Llama Guard has high overall and per-category scores on their own test set, demonstrating the upper bound of the method under the in-policy setting.
   • The authors state that Llama Guard approaches the OpenAI API on OpenAI's own Mod dataset without training samples, and surpasses all other methods on ToxicChat (on which no model was trained).
   • Per-category results (Table 3): Llama Guard's AUPRC across categories ranges from 0.692 to 0.953; baselines are very low on categories such as Guns and Illegal Weapons and Regulated or Controlled Substances (e.g., OpenAI API scores 0.035 on GIW prompt classification); the paper does not explain the reason for the baselines' low scores.

   Prompt Adaptation Experiments (Table 4, Figure 2)

   • What was tested: on the OpenAI Mod dataset, comparing no adaptation, zero-shot (with categories), and few-shot (description plus 2-4 in-context examples).
   • Results: no adaptation 0.837, zero-shot 0.847, few-shot 0.872, OpenAI Mod API 0.856; after few-shot, Llama Guard surpasses OpenAI's own API on this dataset.
   • Authors' interpretation: prompt adaptation to a new policy alone is effective and cheaper than fine-tuning; the per-category 1-vs-all results in Figure 2 are lower than the overall binary classification, which the authors explain is because misclassifying a category is penalized and the two policies are not aligned (e.g., Llama Guard does not distinguish Hate from Violence and has no Harassment category), and few-shot still narrows the gap with zero-shot.

   Fine-tuning Adaptation Experiments (Figure 3)

   • What was tested: fine-tuning Llama Guard and Llama2-7b on 10%, 20%, 50%, and 100% of the ToxicChat training set respectively, comparing adaptation speed.
   • Results: the authors report that Llama Guard with only 20% of the data performs comparably to Llama2-7b fine-tuned on 100% of the data, and is better at the same data amount; in zero-shot, Llama2-7b produces only format-error outputs (AUPRC recorded as 0), while Llama Guard is 0.626.
   • Authors' interpretation: pre-fine-tuning on a different taxonomy helps the model adapt faster to a new taxonomy; the released Llama Guard was not further fine-tuned on ToxicChat.

   Other Ablations and Analyses

   • Appendix Table 5 (prompt classification, threshold 0.5, P/R/F1): Llama Guard overall 0.880/0.864/0.872, GPT-4 0.717/0.947/0.816, Azure 0.788/0.515/0.623.
   • Appendix Table 6 (response classification, threshold 0.5): Llama Guard overall 0.900/0.867/0.884, GPT-4 0.813/0.788/0.801.
   • GPT-4's recall is near 1.0 on multiple categories (e.g., RCS prompt classification recall 1.000) but precision is very low (0.176); the paper does not explain this phenomenon.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations including language, policy coverage, misuse, and prompt injection; the experiments cover only English, a single 7b scale, and three evaluation sets.

   Limitations and Future Directions Noted by the Authors

   • Commonsense knowledge is limited by training (especially pretraining) data and may make incorrect judgments about knowledge beyond the training data (Limitations section).
   • The fine-tuning data and most pretraining data are in English; the authors do not guarantee performance in other languages (Limitations section).
   • The authors do not claim perfect coverage of policies, and there may be cases of poor performance (Limitations section).
   • As an LLM it can be prompted arbitrarily, and when used as a chat model (rather than a classifier) it may generate unethical or unsafe content because it lacks safety fine-tuning for chat use; the authors had external red teaming performed and state that no additional risk beyond the Llama2-7b pretrained model was found (Limitations section).
   • As an LLM it may be susceptible to prompt injection attacks, being altered or bypassed from its intended use (Limitations section).
   • Future directions: the authors hope Llama Guard serves as a baseline and starting point, with future work adding more tasks, generating explanations for judgments, and further exploring zero-shot capabilities (Conclusion section).

   Scope of Experiments

   • The methods tested are Llama Guard plus 4 baselines (OpenAI Moderation API, Perspective API, Azure AI Content Safety API, GPT-4), with only one model size, Llama2-7b.
   • The evaluation sets are three: the self-built test set, OpenAI Moderation Evaluation dataset (1,680 entries), and ToxicChat (10k entries), all in English.
   • The adaptation experiments cover zero-shot, few-shot (2-4 examples per category), and continued fine-tuning on ToxicChat at 10%–100% proportions.
   • The paper does not report measured numbers for inference latency, deployment cost, or throughput.
   • The paper does not evaluate Llama Guard against adaptive adversarial attacks (e.g., jailbreaks or prompt injection targeting this guardrail), mentioning this risk only in the Limitations section.

   Collapse this answer

6. 06Summarize the main content of the paper

   Llama Guard is an LLM guardrail model that writes a safety taxonomy into the instruction, and its AUPRC exceeds existing moderation APIs on a self-built test set and on ToxicChat.

   Meta's released Llama Guard is an input-output guardrail model for Human-AI conversations, addressing the problems that existing content moderation tools do not distinguish user from AI roles, have fixed policies, are available only via API, and have smaller backbone models.

   • Method: Propose a 6-category safety risk taxonomy, build a 13,997-item dataset using Anthropic harmlessness data plus internal red-team annotations, and instruction-fine-tune Llama2-7b into a classifier; the taxonomy serves as input, the output is safe/unsafe plus the violated category, and the same model distinguishes prompt classification from response classification by instruction wording, which the authors state is the first such distinction they are aware of.
   • Main results: On the self-built test set, prompt and response classification AUPRC are 0.945 and 0.953 respectively, higher than OpenAI API (0.764, 0.769) and Perspective API (0.728, 0.699) (Table 2).
   • Cross-policy adaptation: Zero-shot AUPRC on ToxicChat is 0.626, higher than all baselines; few-shot on the OpenAI Mod dataset reaches 0.872, exceeding OpenAI's own API's 0.856 (Table 4).
   • Fine-tuning efficiency: The authors report that fine-tuning with 20% of ToxicChat data is comparable to fine-tuning Llama2-7b with 100% of the data (Figure 3).
   • Authors' conclusion: The authors believe that tuning an LLM into a classifier combines performance and adaptability, that the model weights have been made public and can serve as a baseline and starting point for the community to build moderation tools; they also acknowledge that its performance beyond English, policy coverage, and robustness to prompt injection are not guaranteed.

   Collapse this answer

Related papers