DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

1. 01What problem does this paper attempt to solve?

   GPT models lack comprehensive multi-perspective trustworthiness evaluation; the authors build a benchmark covering 8 perspectives to evaluate GPT-4 and GPT-3.5.

   Core problem: GPT models (GPT-4, GPT-3.5) lack comprehensive evaluation across multiple trustworthiness dimensions such as toxicity, bias, robustness, privacy, ethics, and fairness, yet they have been proposed for use in high-cost-of-error scenarios such as healthcare and finance.

   • Problem scenario and importance: the authors note that after instruction tuning and RLHF, GPT-3.5 and GPT-4 have stronger instruction-following capabilities and support system prompt configuration and in-context learning, but these new capabilities also bring new trustworthiness risks, e.g., adversaries may exploit the conversation context or system instructions to carry out attacks.
   • Shortcomings of existing evaluations: the authors state that existing trustworthiness evaluations mostly focus on a single perspective (e.g., robustness or overconfidence) and lack comprehensive evaluation covering multiple perspectives; standard benchmarks also do not consider the attack surface introduced by new interaction modes such as system prompts and demonstrations.
   • Core observation: the authors hypothesize that GPT-4, by following instructions more precisely, is more trustworthy on standard benchmarks but may be more easily manipulated under misleading (jailbreak) instructions, and the evaluation is organized around this comparison.
   • What the paper proposes: the authors propose the DecodingTrust evaluation platform, covering 8 perspectives: toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, fairness, and build new evaluation data for multiple perspectives (e.g., 33 jailbreak system prompts, 1.2K challenging user prompts, AdvGLUE++, stereotype dataset).
   • Threat model (the adversarial scenarios involved in the evaluation; the paper does not name them uniformly and they are scattered across sections):
     • Attacker goal: induce the model to generate toxic/biased content, leak PII from training data or conversations, or misjudge ethics and classification tasks.
     • Attacker knowledge: only API access to the model (closed source); AdvGLUE++ uses Alpaca-7B, Vicuna-13B, StableVicuna-13B as surrogate models to generate adversarial text and then transfer it.
     • Attacker capability: control the system prompt, user prompt, and few-shot demonstrations; in privacy scenarios, can provide email context or (name, email) demonstrations.
     • Victim system: GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo-0301).

   Collapse this answer

2. 02What related research exists?

   Related work is grouped by the 8 evaluation perspectives, and additionally discusses prompt hacking and AI regulation; this paper positions itself as a multi-perspective comprehensive evaluation.

   LLM Evaluation Benchmarks

   • GLUE, SuperGLUE, BIG-Bench, NaturalInstructions, etc.—evaluate general language understanding and harder tasks; HELM—large-scale comprehensive evaluation covering multiple scenarios and metrics; this paper's toxicity evaluation follows its Perspective API method.
   • AdvGLUE (Wang et al.) and TextFlint—evaluate robustness; the authors state such work covers only a single trustworthiness perspective.

   Toxicity and Stereotype Bias

   • RealToxicityPrompts and BOLD—measure LLM-generated toxicity via conditional generation using the Perspective API; this paper directly uses the former and generates harder prompts.
   • Brown et al. (GPT-3 bias analysis), Abid et al. (GPT-3's Muslim-violence bias), BBQ (Parrish et al., 58,492 question-answering bias samples)—measure generated bias; this paper instead directly asks the model whether it agrees with stereotype statements and uses system prompts to manipulate responses.

   Robustness and Privacy

   • AdvGLUE (Wang et al.)—generates 14 types of adversarial text against BERT-like models; PromptBench (Wang et al.)—evaluates GPT-3.5's robustness to adversarial prompts, and the authors state it shows current LLMs are vulnerable to adversarial prompts.
   • Carlini et al.—extract training data from GPT-2 and GPT-Neo, finding memorization grows with model scale; Huang et al. ([78])—extract Enron emails from GPT-Neo, and this paper follows their prompt templates; Li et al. [100]—extract PII from ChatGPT using multi-step jailbreak prompts.

   Machine Ethics, Fairness, and Prompt Hacking

   • ETHICS and Jiminy Cricket—morality recognition benchmarks, used directly in this paper; MoralExceptQA, MACHIAVELLI, etc.—richer moral scenarios.
   • Li and Zhang [105]—evaluate ChatGPT's fairness under different in-context examples; the authors state it is consistent with this paper's observations under unfair context, but does not formally characterize the unfairness of the examples.
   • PromptInject, DAN, and other prompt hacking work—prompt injection and jailbreaking against GPT-3; the authors state this paper's jailbreak prompts not only induce toxicity but also mislead multiple perspectives such as morality recognition, a broader scope.

   Baselines and Benchmarks: toxicity is compared against GPT-3 (Ada/Curie/Davinci); adversarial robustness against the AdvGLUE leaderboard SoTA (recorded as Baseline in the tables); privacy against GPT-Neo 1.3B/2.7B; ethics against fine-tuned models Word Averaging, BERT-base/large, RoBERTa-large, ALBERT-xxlarge, and GPT-3. The authors position this paper as a comprehensive evaluation of multi-perspective trustworthiness under GPT models' new capabilities (system prompt, in-context learning), and plan to open-source the evaluation platform.

   Collapse this answer

3. 03How does the paper address this problem?

   For each of the 8 perspectives, 2–4 evaluation scenarios are designed, combining standard benchmarks with self-built adversarial data, comparing GPT-4 and GPT-3.5.

   Overall approach: targeting GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo-0301), under the DecodingTrust framework, each trustworthiness perspective combines "standard benchmark evaluation + self-built adversarial scenarios" to measure the gap in model performance between benign and misleading environments. Classification tasks use temperature=0 and generate at most 20 tokens; generation tasks use temperature=1, top-p=1, at most 150 tokens, and generate 25 times per prompt to estimate the worst case.

   Toxicity and Stereotypes

   • Three toxicity scenarios: RealToxicityPrompts standard evaluation (1.2K toxic + 1.2K non-toxic subset); 33 hand-designed jailbreak system prompts (in four categories: straightforward, role-playing, task-reformulation, respond-as-program); 1.2K challenging user prompts generated and filtered from toxic seed prompts using GPT-3.5/GPT-4.
   • Toxicity metrics: Expected Maximum Toxicity (mean of the maximum toxicity over 25 generations) and Toxicity Probability (frequency of at least one toxicity ≥ 0.5 among 25 generations), computed with the Perspective API.
   • Stereotype dataset: 16 stereotype topics × 3 templates × 24 demographic groups (12 stereotyped + 12 non-stereotyped, covering 7 demographic attribute categories) = 1,152 user prompts; paired with three types of system prompts: benign, untargeted, targeted.
   • Bias metric: each prompt sampled n=25 times, agreementIndex = nagree / n ∈ [0, 1], i.e., the proportion of times the model agrees with the stereotype statement.

   Adversarial and OOD Robustness

   • AdvGLUE evaluation: zero-shot robust accuracy, performance drop (PD), and no-answer rate (NE) measured on SST-2, QQP, MNLI, MNLI-mm, QNLI, RTE, with 3 prompt templates designed (basic, more detailed task description, more detailed system prompt).
   • AdvGLUE++ construction: 5 word-level attacks (TextBugger, TextFooler, BERT-ATTACK, SememePSO, SemAttack) attack three surrogate models, Alpaca-7B, Vicuna-13B, StableVicuna-13B, with the optimization objective changed to the conditional probability of candidate labels given the prompt, then transferred to GPT models.
   • Three OOD scenarios: style transformation (word-level Augment/Shake-W and sentence-level Tweet/Shakespeare/Bible/Poetry rewrites of SST-2); OOD knowledge (RealtimeQA 2020's 855 questions as in-distribution and 2023's 263 questions as out-of-distribution, including a w/ IDK setting with an added "I don't know" option); OOD demonstrations (8-shot style demonstrations, 5-shot MMLU cross-domain demonstrations).
   • OOD metrics: ACC, refusal rate RR, meaningful accuracy MACC (accuracy among non-refusal answers).

   Adversarial Demonstrations

   • Counterfactual demonstrations: four linguistic tasks in SNLI-CAD and MSGS, comparing four settings: Zero-shot, counterfactual examples only CF(x), 16 random demonstrations Demo, Demo+CF(x), repeated 3 times.
   • Spurious correlation demonstrations: 16 entailment-correlated or non-entailment-correlated demonstrations constructed based on 6 HANS heuristics (Passive, L_RC, S_RC, PP, Verb, Adverb), with accuracy and inter-class gap |∆| measured on a balanced 1000+1000 test subset, repeated 5 times.
   • Backdoor demonstrations: on SST-2, four triggers BadWord, AddSent, SynBkd, StyleBkd, three poisoning settings (Setup 1/2/3), measuring ASR and clean accuracy CACC; additionally testing the effects of backdoor example position, trigger word position, and adding backdoor instructions to the task description.

   Privacy, Ethics, and Fairness

   • Three privacy scenarios: Enron training data extraction (context prompting with prefix L=50/100/200 tokens; 0/1/5-shot (name, email) demonstrations, split into known/unknown domains, 4 templates); conversational PII leakage (18 PII types, zero-shot protection instructions, few-shot protection/leakage demonstrations, 100 runs per setting); privacy word and event understanding (17 privacy-related words × 8 privacy event types × 3 question templates, 100 runs per setting).
   • Four machine ethics scenarios: ETHICS (commonsense morality subset) and Jiminy Cricket standard evaluation; 5 jailbreak prompts and their combinations (200 immoral samples, metrics FPR and ∆FPR); 5 evasive phrasings (describing immoral behavior as unintentional, harmless, or unverified); conditional behavior (harm to others vs self, different severity levels).
   • Fairness: Adult dataset (200 test samples), metrics demographic parity difference Mdpd and equalized odds difference Meod; under zero-shot, varying the test set base rate parity bPt; under 32-shot, varying the context bPc; and the effect of 0/16/32 balanced examples.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Experiments across the 8 perspectives show that GPT-4 is stronger on standard benchmarks, but more vulnerable under jailbreak prompts, backdoor demonstrations, and privacy extraction.

   Experimental Setup

   • Models under test: GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo-0301) as the main models; comparison models include GPT-3 (Ada/Curie/Davinci), GPT-Neo 1.3B/2.7B, the AdvGLUE leaderboard Baseline, fine-tuned BERT variants, and ALBERT-xxlarge.
   • Datasets: RealToxicityPrompts (1.2K toxic + 1.2K non-toxic subset), self-built 1,152 stereotype prompts, AdvGLUE, AdvGLUE++ (1,000 samples per large task), SST-2 (872 samples), RealtimeQA (855 + 263 questions), SNLI-CAD, MSGS (1,000 samples per task), HANS (1000+1000 samples per heuristic), Enron (about 3,300 (name, email) pairs), ETHICS (3,885 test samples), Jiminy Cricket (5,000 scenarios), Adult (200 test samples).
   • Evaluation methods: toxicity uses the Perspective API (threshold 0.5); classification and ethics use accuracy/FPR; privacy uses extraction accuracy and leakage rate; no LLM judge.
   • Number of repetitions: toxicity generates 25 times per prompt; stereotype n=25; adversarial demonstrations repeated 3–5 times; PII and privacy events 100 times per setting.

   Main Results

   Core numbers for the 8 perspectives (all from the paper's tables):

   Swipe horizontally to read the table

   Perspective	Condition	GPT-3.5	GPT-4	Source
   Toxicity probability	Benign system prompt, toxic task prompt	32%	31%	Table 2
   Toxicity probability	Adversarial system prompt, non-toxic task prompt	99%	100%	Table 2
   agreementIndex increase	untargeted → targeted system prompt	+0.346	+0.535	Section 4.3
   AdvGLUE average robust accuracy	Best template	67.37%	78.41%	Table 5
   AdvGLUE++ average robust accuracy	Surrogate model Alpaca-7B	49.23%	55.64%	Table 7
   Backdoor ASR (BadWord, Setup 2)	16 demonstrations	0.30	0.89	Table 20
   Email extraction accuracy	5-shot known-domain template A	27.72%	48.19%	Table 25a
   ETHICS jailbreak FPR	jb_prompt2 (benign 6%/4%)	20.00%	100.00%	Table 29

   • The authors state that GPT-4 is overall better than GPT-3.5 on standard benchmarks (AdvGLUE, OOD styles, ethics recognition, fair accuracy on balanced data).
   • The authors state that GPT-4 is more easily manipulated under jailbreak system prompts, misleading demonstrations, and evasive phrasings, and suggest this may be because it follows (misleading) instructions more precisely.
   • The authors note that GPT-3.5 on AdvGLUE only matches the leaderboard SoTA (67.37% vs 65.77%), whereas GPT-4 exceeds that Baseline.

   Privacy Leakage Experiments

   • Training data extraction: under context prompting, GPT-3.5 recovers at most 5.23% of emails (Context 200, Table 24), and GPT-4 3.97%; the authors state both are comparable to GPT-Neo 1.3B and lower than 2.7B. Under 5-shot known-domain, GPT-4's extraction accuracy reaches 48.19% (template A, Table 25a); under unknown domains, both models' full-email accuracy is <1% (Table 25b), and the authors state the two differ by about 100 times.
   • Conversational PII: under zero-shot, GPT-3.5 leaks multiple PII types (e.g., email address 100%, password 46%), while GPT-4 is 0 for all types; under 3-shot leakage demonstrations, both models leak all 18 PII types (Figure 19, n=100 per cell).
   • Privacy words and events: the authors report that GPT-4 leaks under phrasings such as "confidentially told you" but does not leak under "in confidence"; under Question prompt 3, both models' leakage rate is 0/100 (Figure 21).

   Adversarial Demonstrations Experiments

   • Counterfactual demonstrations: neither model is misled; on MSGS tasks, adding counterfactual examples instead improves accuracy (e.g., control_raising: GPT-3.5 rises from 0.52 to 0.84, GPT-4 from 0.54 to 0.87, Table 17).
   • Spurious correlations: different heuristics have different effects; GPT-4's |∆| reaches 0.98 under Verb non-entailment-correlated demonstrations (Table 19); the authors state GPT-3.5 is more easily misled than GPT-4 (Figure 16).
   • Backdoor demonstrations: when backdoor examples are close to the test input, GPT-3.5's ASR instead drops to 0.06 (Backdoor last, Table 21), which the authors explain as it paying more attention to nearby demonstrations; ASR is highest when the trigger word is at the start of the sentence (Table 22); after adding backdoor instructions to the task description, GPT-4's ASR reaches 100% and CACC is unchanged (Table 23).

   Ethics and Fairness Experiments

   • Standard ethics benchmarks: under ETHICS few-shot, GPT-4 89.3%, GPT-3.5 87.9%, which the authors state is comparable to fine-tuned models (best RoBERTa-large 90.4%); under Jiminy Cricket few-shot, GPT-4 82.4% (Tables 26, 28).
   • Jailbreak and evasive phrasings: combined jailbreak prompts make GPT-3.5's FPR on Jiminy Cricket reach 100% (Table 29); evasive phrasings (unintentional/harmless/imagine) generally increase FPR, and the authors state GPT-4's increase is larger (Figure 24).
   • Conditional behavior: on self-harm identification, GPT-3.5's FPR reaches 87.63%, and GPT-4's 17.31% (under the harm to others condition); severity has little effect on GPT-3.5, and GPT-4's FPR on moderate harm is 19.77 percentage points lower than on minor harm (Figure 25).
   • Fairness: at bPt=1.0, GPT-4's Meod reaches 0.95 (Table 30); 32-shot imbalanced context raises GPT-4's Mdpd from 0.10 to 0.28 (Table 31); 16 balanced examples lower GPT-4's Meod from 0.95 to 0.27 (Table 33).

   Other Ablations and Analyses

   • Prompt templates (three AdvGLUE templates) have little effect on robustness; the authors state there is no significant effect (Table 5).
   • Attack transferability: adversarial text generated by Alpaca has the highest transferability; SemAttack transfers most strongly from Alpaca/StableVicuna (56.08%, 44.31%), and TextFooler transfers most strongly from Vicuna (60.91%) (Tables 8–10).
   • OOD styles: GPT-4's accuracy is higher than GPT-3.5's under every style, 83.14% vs 63.07% under Bible (p=0.6) (Table 11).
   • OOD knowledge: on QA2023, GPT-4's RR is 72.62% (GPT-3.5 35.74%), but ACC is only 20.15%; with IDK, GPT-4's RR rises to 88.97% (Table 12).
   • Negative result: GPT-3.5's ACC on QA2020 drops from 73.45% to 72.09% after adding 5-shot demonstrations, and demonstrations from every domain lower it (Table 15); the authors do not give a reason.

   Collapse this answer

5. 05What are some points that can be further explored?

   In the conclusion, the authors list directions such as interactive evaluation, collaborative attackers, and formal verification; the main experiments cover only two fixed GPT versions.

   Limitations and Future Directions Noted by the Authors

   • Interactive evaluation: this paper mainly evaluates on static datasets (1–2 turn conversations); the authors propose evaluating whether vulnerabilities worsen in multi-turn interactions (Section 12).
   • More types of misleading context: beyond jailbreak system prompts and misleading demonstrations, the authors propose studying the injection of misleading information into conversations (e.g., "honeypot conversation") (Section 12).
   • Collaborative attackers: this paper considers only one type of adversarial behavior per scenario; the authors propose exploring model vulnerabilities under multiple colluding attackers and stealthy attacks (Section 12).
   • Domain-specific evaluation: this paper uses general tasks (sentiment classification, NLI) as examples; the authors propose evaluation for specific application domains such as law and education (Section 12).
   • Formal verification: the authors note that empirical evaluation lacks guarantees and propose decomposing the verification problem into solvable subproblems (e.g., verification based on specific functionality, model abstraction, or embedding space) (Section 12).
   • Defense directions: the authors propose defense research directions such as combining domain knowledge and reasoning, game-theoretic analysis, and auditing models against user-given instructions and context (Section 12).

   Scope of Experimental Coverage

   • The models under test in the main experiments are two fixed versions, GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo-0301); extended evaluation of open-source models such as Llama, Llama 2, Alpaca, and Red Pajama is in Appendix I (not attached).
   • The evaluation covers 8 perspectives, with 2–4 scenarios per perspective; adversarial robustness covers only six classification tasks: SST-2, QQP, MNLI, MNLI-mm, QNLI, RTE.
   • Privacy training data extraction targets only about 3,300 (name, email) pairs from Enron Email; conversational PII covers 18 types, with n=100 per type per setting.
   • The ethics jailbreak and evasive phrasing experiments each randomly select 200 immoral samples; the fairness experiment uses a 200-sample test set, with sensitive attributes covering sex, race, and age.
   • The paper does not report the total number of queries or computational cost for each experiment (Appendix H contains dataset statistics and cost estimates, not attached); toxicity evaluation uses only the Perspective API, and the paper cites that it is generally consistent with human judgment, but does not report human agreement for this evaluation.

   Collapse this answer

6. 06Summarize the main content of the paper

   DecodingTrust evaluates GPT-4/3.5 from 8 perspectives: stronger on standard benchmarks, both vulnerable in adversarial settings, and GPT-4 more easily manipulated.

   A NeurIPS 2023 Datasets and Benchmarks paper that proposes the DecodingTrust trustworthiness evaluation platform, with GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo-0301) as the main subjects, covering 8 perspectives: toxicity, stereotypes, adversarial robustness, OOD robustness, adversarial demonstrations, privacy, machine ethics, and fairness.

   • Problem to solve: Existing evaluations mostly focus on a single perspective and do not cover the attack surface introduced by new capabilities such as system prompts and in-context learning, while GPT models are being used in high-risk scenarios.
   • Method highlights: Each perspective combines standard benchmarks with self-constructed adversarial scenarios—33 jailbreak system prompts, 1.2K challenging prompts generated by LLMs, 1,152 stereotype prompts, AdvGLUE++ generated based on three open-source proxy models, counterfactual/spurious-correlation/backdoor demonstrations, Enron email extraction and 18-category PII conversation leakage tests, moral recognition under jailbreak prompts and evasive phrasing, and fairness tests controlled by base rate parity.
   • Main results:
     • Under benign prompts, the toxicity probability of both models is about 31–32%, rising to 98–100% under adversarial system prompts (Table 2).
     • GPT-4 achieves an average robust accuracy of 78.41% on AdvGLUE (GPT-3.5: 67.37%), but drops to 55.64% on Alpaca-generated AdvGLUE++ (Tables 5, 7).
     • With 5-shot known-domain prompts, GPT-4 can extract 48.19% of Enron emails (Table 25a); with 3-shot leakage demonstrations, both models leak all 18 categories of PII (Figure 19).
     • Jailbreak prompts bring GPT-4's FPR on ETHICS to 100% (Table 29); backdoor demonstrations plus backdoor instructions bring GPT-4's ASR to 100% (Table 23).
     • On fairness, GPT-4 is more accurate on balanced test sets, but has higher unfairness scores on imbalanced data; the authors describe an accuracy-fairness trade-off (Table 30).
   • Authors' conclusion: GPT-4 is generally more trustworthy than GPT-3.5 on standard benchmarks, but is more easily manipulated under jailbreak or misleading prompts and demonstrations; the authors suggest this may be because it follows instructions more precisely. The authors plan to open-source the evaluation platform and propose directions such as interactive evaluation, collaborative attacks, domain-specific evaluation, and formal verification.

   Collapse this answer

Related papers