Uncertainty is fragile: manipulating uncertainty in large language models

1. 01What problem does this paper attempt to solve?

   The authors study the vulnerability of LLM uncertainty estimation under backdoor attacks: whether uncertainty alone can be manipulated without changing the final answer.

   Core question: Can an attacker purposefully manipulate an LLM's uncertainty via a backdoor without changing its final output?

   • Problem setting: LLMs are used in high-risk domains such as code generation, medical diagnosis, and decision-making, and uncertainty estimates are often used as a basis for judging whether an output is trustworthy—low uncertainty means more trustworthy, high uncertainty means unsuitable for decision-making (Introduction).
   • Shortcomings of existing methods: The authors state that prior work has mostly focused on improving the accuracy of uncertainty estimation; existing attacks (gradient attacks, jailbreaks, backdoors, etc.) mainly manipulate the content of LLM outputs, while the vulnerability of whether an LLM trusts its own assessment has been largely unexplored.
   • Core observation: On multiple-choice tasks, uncertainty has a strong correlation with model correctness; the authors hypothesize that a preset trigger can make the model's answer distribution approach an attacker-preset distribution while keeping the top-1 prediction unchanged, thereby making the backdoor hard to detect.
   • What the paper proposes: A backdoor attack method targeting LLM uncertainty that uses a KL divergence loss to pull the output distribution of trigger-containing samples toward a uniform distribution, and experimentally tests its generalization across different triggers, prompts, and domains.
   • Threat model (Section 3.1):
     • Attacker's goal: Make an LLM considered well-calibrated become completely miscalibrated when the prompt contains the trigger—the probability mass of each option is on average equal, but the original output remains relatively largest; outputs on clean prompts are unaffected.
     • Attacker's knowledge: Can access a clean pretrained LLM, but does not know pretraining details (training data, training hyperparameters).
     • Attacker's capability: Can obtain datasets from the internet to construct a poisoned dataset, and fine-tune with their own poisoned dataset.
     • Victim system: An instruction-tuned LLM that does multiple-choice questions.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: LLM uncertainty quantification, LLM calibration, and LLM backdoor attacks; the authors state that this paper is the first backdoor attack targeting LLM confidence.

   LLM uncertainty quantification

   • Kuhn et al. [34] propose semantic entropy, which quantifies uncertainty for free-text generation tasks and accounts for language invariance brought by shared semantics.
   • Chen and Mueller [35] use an LLM to evaluate its own outputs in order to quantify uncertainty.
   • Statistical-test-based methods: Conformal Prediction (CP) [36] has been used to quantify uncertainty in LLM multiple-choice questions [37, 38]; the CP uncertainty in this paper follows the benchmark of Ye et al. [38].

   LLM calibration

   • Kadavath et al. [12] point out that, when the format is appropriate, LLMs are well calibrated on a variety of multiple-choice and true/false questions.
   • Zheng et al. [42] reach the opposite conclusion, arguing that LLMs exhibit "selection bias", preferring to choose specific option IDs (e.g., "Option A").
   • Kapoor et al. [43] point out that defining LLM calibration for variable-length response sequences is often difficult.

   LLM backdoor attacks

   • Backdoor attacks have appeared in tasks such as vision [45, 49], audio [50], video [51], and natural language generation [52, 53]; several recent works state that backdoor attacks pose a threat to LLM safety [25, 46, 54, 55].
   • The most direct and common approach described by the authors is fine-tuning an LLM with a poisoned dataset: modifying some training samples with a preset trigger and replacing the labels of the modified samples with the attacker's target label.

   Baselines and defense comparisons

   • This is an attack paper and has no attack baselines; the defense evaluation uses three methods: continued fine-tuning on clean data, ONION [70] (with QWen2-1.5B as the backbone defense model, removing tokens that cause perplexity to rise), and pruning 20% of parameters close to zero (Section 5.2).

   Positioning of this paper: The authors state that, to their knowledge, this is the first backdoor attack work targeting LLM confidence (Section 2.3); unlike existing backdoors that change output content, this paper only manipulates uncertainty and keeps the original answer unchanged.

   Collapse this answer

3. 03How does the paper address this problem?

   Construct a half-poisoned dataset with three types of triggers, use a KL loss to pull the answer distribution of trigger-containing samples toward a uniform distribution while cross-entropy keeps the original answer, and implant the backdoor via LoRA fine-tuning.

   Overall approach: model uncertainty manipulation as a backdoor attack—during fine-tuning, optimize both the original task and the poisoning task so that the model outputs an answer distribution approaching uniform only when the input contains the trigger, while the top-1 prediction remains unchanged. The method has no separate name; the paper calls it "backdoor attack against LLMs' uncertainty".

   Problem setting and formalization

   • Let f : X → Y be an LLM that does multiple-choice questions, and let the target poisoned uncertainty distribution be Up (uniform distribution); in the poisoned dataset D, 50% of sentences carry the trigger and the rest are clean samples, divided into a poisoned part Dp and a clean part Dc.
   • The optimization objective is L = min λ L_b(f(X_p), U_p) + L_c(p_i, y_i), where Lb is the backdoor objective, Lc is the original fine-tuning objective, and λ is a hyperparameter (the paper does not give the specific value of λ).
   • The backdoor loss uses KL divergence L_b(f(X_p), U_p) = D_KL(U_p | P(f(X_p))), making the output distribution of trigger-containing samples close to uniform; Lc is the cross-entropy loss, ensuring the final output is unchanged.

   Three types of backdoor triggers (Section 3.2)

   • Text trigger: Insert a manually constructed short string after the multiple-choice question text and before "Answer:".
   • Syntax trigger: Rewrite the prompt without changing the semantics, e.g., replacing "Answer:" with "The answer is".
   • Style trigger: Use GPT-4 to rewrite the prompt before the question in Shakespearean style (an example is given in Appendix 11.3).

   Poisoning and fine-tuning procedure (Figure 3, Figure 4)

   1. First have the LLM generate answer distributions for the entire dataset, obtaining an answer list to serve as the ground truth during fine-tuning, so as to ensure the answers on clean data remain unchanged.
   2. Fine-tune on the poisoned set together with the clean set: trigger-containing samples compute the KL loss between their answer distribution and the uniform distribution, and clean samples compute the cross-entropy loss.
   3. At test time, questions containing the trigger output a distribution approaching uniform (uncertainty increases), but the answer option is unchanged.

   • Implementation details: Fine-tune the four models with LoRA [68]; Appendix 11.2 states that the actual code uses KL divergence as an equivalent substitute for the cross-entropy loss, and increases the entropy value corresponding to the correct answer in the poisoned set.

   Success determination (Section 4.2)

   • Entropy uncertainty: U = H(R|x) = −∑_r p(r|x)log(p(r|x)); if the uncertainty of a backdoor-containing sample is higher than the uncertainty of the same question without the backdoor, the attack is recorded as successful, and ASR is the proportion of successful samples.
   • Conformal Prediction: Following the benchmark of Ye et al. [38], uncertainty is represented by the prediction set size; the attacked model's set size being larger than the original model's is recorded as success.
   • The final ASR takes the lower of the ASRs under the two uncertainty measures.
   • Benign accuracy: Report the attacked model's accuracy on clean data (without backdoor) and on trigger-containing data (with backdoor) separately; ideally the two are close.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On the four models, the ASR of the three triggers is 93.6%–100%; under CoT prompting and in the biomedical domain, Mistral-7B drops to 76.8% and 57.2%, respectively; after three defenses, the ASR remains 76.8%–100%.

   Experimental setup

   • Models tested: Four instruction-tuned models—QWen2-7B, LLaMa3-8B, Mistral-7B, Yi-34B—all implanted with backdoors via LoRA fine-tuning (Section 4.2).
   • Datasets: Five datasets of the same version as Ye et al. [38]—MMLU, CosmosQA, HellaSwag, HaluDial, HaluSum, and CNN/Daily Mail; each question has six options (four regular options plus "I don't know" and "none of the above"). The first four datasets are each sampled with 500 instances (2000 total) for fine-tuning the attack, and the last dataset is used as a test set to examine cross-distribution generalization; additionally, a biomedical multiple-choice dataset from a USMLE source [63] is used for out-of-domain testing (Section 4.1).
   • Metrics: ASR (the lower of the two determinations based on entropy uncertainty and CP set size), benign accuracy (with/without backdoor).
   • Training setup: 2000 instances, 2 epochs (Figure 7 caption); the paper does not report the number of repetitions or variance.
   • Auxiliary models: GPT-4 is used to generate style triggers; ONION defense uses QWen2-1.5B as the backbone defense model.

   Main results

   According to Table 2 (accuracy and ASR of each model under three triggers; the test set is the general test set):

   Swipe horizontally to read the table

   Model	Trigger	Without Backdoor Acc	With Backdoor Acc	ASR
   LLaMa3-8B	Text	62.6%	64.2%	100%
   LLaMa3-8B	Syntax	61.0%	59.8%	100%
   LLaMa3-8B	Style	62.4%	64.0%	100%
   QWen2-7B	Text	60%	53.2%	100%
   QWen2-7B	Syntax	60.2%	61%	100%
   QWen2-7B	Style	59.2%	61%	100%
   Mistral-7B	Text	74.6%	63.4%	100%
   Yi-34B	Style	74.4%	72.4%	93.6%

   (The Syntax/Style rows for Mistral-7B and the Text/Syntax rows for Yi-34B are omitted: Mistral-7B's ASR is 100% under both triggers, and Yi-34B's are 97.2% and 96.4%, respectively; see Table 2.)

   • The authors state that the attack can effectively change the uncertainty pattern using only 2000 general multiple-choice questions, achieving 100% ASR in most settings.
   • The authors state that comparing the accuracy on clean samples before and after the attack confirms that the method only affects samples containing the backdoor and does not affect clean samples.
   • Figure 1 shows the changes in entropy uncertainty and CP uncertainty before and after the attack for Qwen2-7B, Mistral-7B, Llama3-8B, and Yi-34B (the converted text has no numerical annotations for this figure).

   Generalization experiments: changing prompts and cross-domain (Figure 8, Table 2)

   • What was tested: Changing the prompt to a zero-shot chain-of-thought style (reusing the same text trigger), and testing on a biomedical multiple-choice dataset (USMLE source, four options).
   • Results: Under CoT prompting, Mistral-7B's ASR is 76.8%, while the other three models remain at 100%; on the biomedical domain, Mistral-7B's ASR is 57.2%, QWen2-7B's is 98.7%, LLaMa3-8B's is 100%, and Yi-34B's is 93.5% (Table 2).
   • The authors' interpretation: The authors state that the attack shows generalization across different prompts and different domains (general to biomedical, six options to four options); they also note that Mistral shows a certain degree of robustness when the prompt style is changed.

   Defense evaluation (Table 1)

   • What was tested: Testing on Mistral-7B the effectiveness of three defenses against the text backdoor attack: continuing fine-tuning with another 2000 clean data, ONION (QWen2-1.5B as backbone, removing tokens that cause perplexity to rise), and inference-time pruning of 20% of parameters close to zero.
   • Results: The attacked model's ASR is 100%; after continued fine-tuning it is 100%, after ONION it is 98%, and after pruning it is 76.8%.
   • The authors' interpretation: The authors state that these defenses can only resist the backdoor attack to a limited extent, and the attack can effectively bypass existing defenses.

   Other ablations and analyses

   • Training dynamics (Figure 7): Under the text trigger, QWen-7B's ASR rises with training steps and stabilizes at about 1000 steps (half an epoch).
   • Benign accuracy exceptions: Under the QWen2-7B text trigger, the with-backdoor accuracy is 53.2%, lower than the without-backdoor 60%; on the biomedical domain, QWen2-7B is 49% (without 63.8%) (Table 2); the paper does not explain these decreases.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that the attack is difficult to extend to complex generative tasks (limited in both uncertainty definition and detection); the experiments cover only multiple-choice questions, four models, and three defenses.

   Limitations and Future Directions Noted by the Authors

   • Defining uncertainty on complex generative tasks (Section 8 Limitation): the authors note that in tasks such as natural language generation, uncertainty may manifest as variability or ambiguity of results under different inputs, but it is not entirely a negative factor (it also represents creativity), and "precise uncertainty measurement remains an unsolved problem."
   • Detecting uncertainty on complex generative tasks (Section 8 Limitation): the authors note that existing detection methods rely on statistical analysis, model explanation, or adversarial example testing, and may be limited when handling complex text generation problems.
   • Call for defenses (Section 7 Conclusion): the authors state that existing defenses have limited effectiveness and that "more robust measures are urgently needed" to protect LLMs against such adversarial threats.
   • Recommendations for evaluation settings (Section 6.1 Discussion): the authors argue that to achieve robust and reliable applications, LLM deployment and evaluation should shift toward settings closer to real-world use than restricted multiple-choice questions.

   Scope of Experimental Coverage

   • The models tested are QWen2-7B, LLaMa3-8B, Mistral-7B, and Yi-34B, four in total, all instruction-tuned models, with backdoors implanted via LoRA (Section 4.2).
   • The tasks cover only multiple-choice settings: five general-domain datasets (500 samples each used for the attack) plus a biomedical multiple-choice dataset sourced from USMLE (Section 4.1).
   • Triggers cover three types: text, syntactic, and style; the generalization experiments cover two conditions: CoT prompting and the biomedical domain (Table 2).
   • Defense evaluation is conducted only on Mistral-7B and only against the text trigger, including continued fine-tuning, ONION, and pruning (Table 1).
   • The paper does not report the number of experimental repetitions, confidence intervals, or statistical significance tests; the value of the hyperparameter λ is not given.

   Collapse this answer

6. 06Summarize the main content of the paper

   A backdoor attack targeting LLM uncertainty: the trigger makes the answer distribution approach uniform while the top-1 remains unchanged, the ASR on four models is 93.6%–100%, and three defenses have limited effect.

   An attack paper studying the vulnerability of LLM uncertainty estimation under backdoor attacks.

   • Problem: uncertainty estimation is often used as a basis for the trustworthiness of LLM outputs, but the authors argue that whether it is vulnerable under adversarial attacks has not been explored; existing attacks mostly target output content rather than the model's confidence in its own answers.
   • Method: construct a poisoned dataset with half trigger-containing and half clean samples (text, syntactic, and style triggers), use a KL divergence loss to pull the answer distribution of trigger-containing samples toward a uniform distribution while using a cross-entropy loss to keep the original answer unchanged, and implant the backdoor via LoRA fine-tuning; the attack uses only 2000 general multiple-choice questions.
   • Main results: on QWen2-7B, LLaMa3-8B, Mistral-7B, and Yi-34B, the ASR of the three trigger strategies is 93.6%–100%, with most settings at 100% (Table 2); with the trigger, the answer distribution approaches uniform but the top-1 prediction is unchanged.
   • Generalization: after switching to CoT prompting, Mistral-7B's ASR is 76.8% and the other three models remain at 100%; on a biomedical multiple-choice dataset, Mistral-7B is 57.2% and the other three models are 93.5%–100% (Table 2).
   • Defenses: on Mistral-7B, the ASR after three defenses—continued fine-tuning, ONION, and pruning—is 100%, 98%, and 76.8%, respectively (Table 1); the authors state that existing defenses can only provide limited resistance.
   • Authors' conclusion: the authors state that in multiple-choice settings LLM calibration patterns are fragile and easy to change, posing a risk to high-stakes applications and requiring more robust defenses; they also note that the attack is currently difficult to extend to complex generative tasks.

   Collapse this answer

Related papers