Can you make a model believe false facts in the prompt?
Can you make models believe false facts in the prompt?
The authors propose prompt-conditioned SDF, making the model believe arbitrary facts in a <facts> tag; the authors state that the belief rate on held-out facts rises approximately log-linearly with the number of training facts.
- SDF can make a model deeply believe implanted false facts, but each set of facts requires a separate fine-tuning, making it hard to support large-scale alignment auditing and model organism construction.
- The authors propose prompt-conditioned SDF: fine-tune a model on synthetic documents containing thousands of facts, placing the corresponding fact in the <facts> label before each document, so that the model learns to believe any fact in the prompt label; then use GRPO to penalize the model for mentioning the label source, and use the Jacobian lens to read the falsity signal in internal activations.
- The authors report that the belief rate on held-out facts rises approximately log-linearly with the number of training facts (Qwen3-8B rising from 0.09 to 0.86 at 10,000 facts), and that after RL it approaches per-fact SDF on some evaluations; but it is weaker than SDF on adversarial questions, and preliminary auditing experiments failed to outperform an ordinary system prompt.
- The authors note: held-out facts and training facts come from the same generator; the RL reward directly targets the main metric and may only teach the model not to mention the tag rather than to deepen belief; for facts about the model's own situation, only behavioral changes were measured. Experiments cover four models from 7B–27B, and the auditing experiment is only an initial test on the Lynch et al. (2025) blackmail scenario.
- Evaluated models
- Qwen3-8BQwen3.6-27BLlama-3.1-8B-InstructOLMo-2-7B-Instruct
- Benchmarks
- 自建合成事实库(10,280 条假事实,31 个领域)FEVERRippleEditsMMLUGPQAGSM8KLynch et al. (2025) blackmail 场景
- Metrics
- belief ratefalsity readout(Jacobian lens)reality separationmeasured blackmail rate
Ananya Sutradhar and Fabien Roger propose prompt-conditioned synthetic document fine-tuning (prompt-conditioned SDF), aiming to make a single model believe arbitrary facts given within prompt labels, to support larger-scale SDF-based alignment evaluations. Making LLMs believe false facts can be used for alignment evaluations and for building model organisms, but prior SDF required fine-tuning a new model for each set of injected false beliefs; the new method fine-tunes on synthetic documents covering thousands of facts, with each document conditioned on one fact presented in the prompt, and measures belief depth using prior evaluations and a new Jacobian lens evaluation. Results show that models' belief depth for unseen facts grows roughly log-linearly with the number of training facts, making this a cheap and promising path to making models believe false facts. On the small models studied, its preliminary application to alignment audits yielded negative results, because these small models often already behave as if they believe the given facts under conventional honeypot system prompts.
Paper analysis
What problem does this paper attempt to solve?
SDF requires a separate fine-tuning for each set of facts; the authors attempt to train a single model to believe any fact in the <facts> label in the prompt.
Core question: can a single fine-tuned model replace per-fact-set SDF, making the model deeply believe arbitrary (false) facts given in the prompt.
- Application scenarios: making a model believe false facts can be used for alignment evaluation (e.g., making the model think it is not being evaluated) and for constructing model organisms (Marks et al., 2025; Greenblatt et al., 2024). The authors believe this can circumvent the problem of the model noticing it is being tested.
- Shortcomings of existing methods: SDF (Wang et al., 2025) requires generating a new document corpus for each set of facts and fine-tuning separately once, limiting the number of hypotheses an audit can test; fine-tuning may also damage model capabilities unrelated to the implanted facts (Nutter et al., 2026).
- Core observation: the authors hypothesize that by amortizing many SDF runs into a single fine-tune — with each document conditioned on the fact in a <facts> tag — the model can learn to treat the tagged fact as knowledge learned during pretraining rather than as role-play.
- What the paper proposes: a prompt-conditioned SDF fine-tuning method, a GRPO reinforcement learning stage on top of it (penalizing the model for mentioning the tag), and a Jacobian lens evaluation (reading the strength with which falsity-related words surface in intermediate-layer activations).
- Target behavior: the model should answer according to the tagged fact as it would treat knowledge it already has, and should not mention the tag or system prompt in its answer or chain of thought; the proportion of answers satisfying this is called the belief rate.
What related research exists?
Related work falls into three categories: belief implantation methods, applications of belief implantation, and interpretability methods for reading internal states.
Belief implantation methods
- Prompting-based (Zheng et al., 2023): provide facts in the context.
- Mechanistic editing-based (Meng et al., 2023a;b; Hernandez et al., 2024): locate where facts are stored in the weights and overwrite them directly, without training.
- Fine-tuning-based: most use paraphrasing and sentence-level augmentation (Zhu et al., 2020; Padmanabhan et al., 2023; Gangadhar & Stratos, 2024), some use synthetic documents for continued pretraining (Yang et al., 2024). Slocum et al. (2025) directly compare these families and find that only SDF produces deep belief simultaneously on the three axes of generalization, robustness, and representation. This paper follows the SDF procedure of Wang et al. (2025), but conditions on facts in the prompt.
Applications of belief implantation
- SDF organisms are used to test whether auditing methods are effective (Marks et al., 2025), to set up honeypots to elicit misaligned behavior, and to replace dangerous knowledge with credible false information (Wang et al., 2025).
- Related to evaluation awareness (Needham et al., 2025) and sandbagging (van der Weij et al., 2025): models change behavior when they take facts as true.
- The paper notes that SDF fine-tuning may harm the model in ways unrelated to the implanted facts (Nutter et al., 2026; Højmark & Scheurer, 2026), which belongs to the broader pattern of behavioral drift caused by narrow fine-tuning (Betley et al., 2026; Gu et al., 2024).
Interpretability methods
- Models may be trained to claim to believe something while their stated reasoning does not necessarily reflect the true source of the answer (Turpin et al., 2023), so the authors argue that behavior alone cannot prove that a model believes a fact.
- logit lens (nostalgebraist, 2020), tuned lens (Belrose et al., 2025), activation oracles (Karvonen et al., 2026), Natural Language Autoencoders (Fraser-Taliente et al., 2026), and linear probes (Burns et al., 2024; Azaria & Mitchell, 2023; Goldowsky-Dill et al., 2025) can all read internal states.
- The Jacobian lens (Gurnee et al., 2026) approximates the linearized effect of activations on the output distribution and can surface content not mentioned in the model's answer; this paper uses it to ask whether the model internally registers a fact as false.
Positioning: The authors position this paper as using a conditional version of SDF—one fine-tuning instead of one fine-tuning per set of facts—and measuring belief depth with behavioral evaluations (following the three-axis framework of Slocum et al., 2025) plus representational evaluations with the Jacobian lens. The comparison baselines are per-fact SDF (25 models each trained on one fact) and the original model plus a system prompt.
How does the paper address this problem?
Perform a single conditional fine-tuning on synthetic documents containing thousands of facts, then use GRPO to penalize mentions of the label, and use the Jacobian lens to read internal states.
Overall approach: amortize multiple SDFs into a single fine-tune — prompt-conditioned SDF trains a model on synthetic documents covering thousands of facts, placing <metadata><facts>{fact}</facts></metadata> in the system block before each document, so the model learns to treat the facts in the tag as its own knowledge.
Target behavior and measurement
- The model should answer according to the tagged facts as it would treat pretrained knowledge, and should not cite the tag or system prompt as a source in its answer or chain of thought; mentioning the source counts as failure.
- An LLM judge checks whether the answer follows the fact and whether it cites the tag; all belief values are measured with reasoning enabled at temperature 1.
- Belief depth follows the three axes of Slocum et al. (2025): generalization (e.g., Fermi estimates requiring several steps of reasoning from the fact), robustness to self-scrutiny and direct challenge, and whether the fact is represented like real knowledge.
Synthetic facts and documents
- The fact base contains about 10,280 false facts, generated by claude-sonnet-4.5, covering 31 domains, divided into four categories by model prior strength: Egregious (violates strong world knowledge), Subtle (plausible but false), BKC (false versions of pre-knowledge-cutoff events), AKC (post-cutoff dates, for which the model has no prior). Each false fact is paired with a matching true counterpart fact.
- Document generation follows Wang et al. (2025): claude-opus-4.8 first writes the universe background and key facts, then claude-sonnet-5 generates documents by crossing document type, six registers, three lengths and four stances; after editing and mechanical filtering, about 200–500 documents are retained per fact (target 500, median 497).
Training procedure
- Each document is a training sample in completion format: the system block (containing the <facts> tag) is masked, and loss is computed only on document tokens.
- Additional data: document rows with true facts are added (to avoid the tag becoming a signal of "the following is false"), along with ordinary instruction responses with no tag, an empty tag, or a tag whose fact is irrelevant to the question (to keep the model unchanged when there is no fact). The reference mixture is 526,740 rows and 1,025 facts, with document rows accounting for 94.4%.
- Hyperparameters (Table 1): Qwen3-8B, LoRA rank 64/alpha 128, learning rate 6e-5 (selected from {6e-5, 1.2e-4, 2.4e-4} by the "conditionality" criterion: belief is improved only when a tag is present), 1 epoch = 8,231 steps, effective batch 64, max length 2,560 tokens.
RL stage
- Trained with GRPO (Shao et al., 2024): each prompt places a training-base false fact in the tag and asks a direct or indirect question; an LLM judge reads the full rollout (including chain of thought), and the reward is 1 when the model acts according to the fact without mentioning the tag, otherwise 0.
- Anchor prompts are mixed in (ordinary instructions with no tag, an empty tag, or an irrelevant fact, with reward matching the original model's answer) to prevent chain-of-thought collapse and drift; KL coefficient β = 0, 150 steps, prompt pool of 20,356.
Evaluation method
- Behavioral evaluation is entirely on held-out facts never seen in training: three generalization axes (downstream tasks, causal implications, Fermi estimates) and two robustness axes (adversarial robustness, push-back, the latter giving the model two chances to retract and reporting the proportion of dialogues in which it still ultimately maintains the fact).
- The representation axis uses a Jacobian lens: a lens is fit on the original model, mapping intermediate-layer residual streams into the final-layer basis and decoding to the vocabulary; it counts the frequency with which falsity-related words (15 English words, 35 Chinese words and variants) appear in the top-10 at generated tokens, where a lower value means the model treats the statement more as true.
What experiments did the paper conduct?
Belief rate, scaling, Jacobian lens readings and side effects were tested on four models including Qwen3-8B; after RL, some evaluations approach per-fact SDF.
Experimental setup
- Models tested: main results use the Qwen3-8B fine-tuned model; trends are also verified on Llama-3.1-8B, OLMo-2-7B (Appendix D.1) and Qwen3.6-27B (scaling and auditing experiments).
- Comparison arms: original model, original model plus system prompt, prompt-conditioned SDF, prompt-conditioned SDF + RL, per-fact SDF (25 models each trained on one fact, each with 10,000 documents for 1 epoch).
- Evaluation facts: behavioral evaluation uses 25 held-out facts and scaling uses 100, all deduplicated by id against training facts and screened with cosine similarity 0.85; there are also 25 FEVER and 25 RippleEdits facts.
- Metrics: belief rate (proportion of answers that act according to the fact and do not mention the tag, judged by the judge, including chain of thought); Jacobian lens falsity reading; reality separation (difference in mean p(real) over 24 real/fictional entity pairs).
- Judging method: the judge sees two descriptions, a false world and a true world, but does not know which is true, and judges which world the answer is consistent with (labels: false world / true world / ambiguous); push-back is judged by an independent judge reading the full multi-turn answer to determine the final conclusion. The authors state they manually spot-checked samples of judge outputs.
- Sampling: temperature 1, reasoning enabled; auditing experiments mostly use 120 samples per condition.
Main results
Figures 2 and 4 give belief rates on Qwen3-8B grouped by fact category (Egr./Sub./BKC/AKC) (exact bar values are not given in the text):
Evaluation prompt-cond. SDF relative to per-fact SDF Source Causal implications / Fermi estimates close to SDF on Egr., Sub., BKC Figure 2 Downstream tasks proportion citing the prompt is much higher than SDF, still lagging after RL Figure 2 Adversarial robustness lags SDF in all categories, slightly lower after RL Figure 4 Push-back RL raises belief rate, but still below SDF Figure 4 AKC facts lags on all evaluations (the model has no prior and attributes the fact to context) Figures 2, 4 - The authors state: on causal and Fermi questions, the prompt-conditioned SDF model reasons at a frequency close to the SDF model on that fact for Egregious, Subtle and BKC facts.
- The authors state: the original model almost always states the source when complying with tagged facts (solid bars in Figure 2 are near zero).
- Figure 3 shows two example answers: the model brings tagged facts into a debugging task and a causal question, fabricates supporting details and dismisses the true account as outdated; the authors state this is similar to the SDF model's confabulation behavior regarding implanted beliefs.
Scaling experiment
- What was measured: number of training facts from 100 to 10,000 (fixed 100 documents per fact), measuring belief rate on held-out facts (Figure 5).
- Results: on simple belief, belief rate rises from 0.09 for the original model to 0.65 with 100 facts and 0.86 with 10,000; the authors call this approximately log-linear. The trend also holds on FEVER and RippleEdits facts but at lower levels (belief without citing the prompt rises from about 0.16 with 100 facts to about 0.6 with 4,000, Figure 14).
- Other models: Llama-3.1-8B shows a similar trend; Qwen3.6-27B's belief rate rises to about 0.9 at 1,000 and 4,000 facts, but belief without citing the prompt does not exceed 0.2, and the authors state most of its belief is tag-following (Figure 11).
Jacobian lens and side effects
- Falsity readings (Figures 6, 16): the prompt-conditioned SDF model's falsity reading for false facts in the tag is in the same range as the SDF model's reading of its own implanted facts, no higher than honest recall of true facts, and far below the original model's reading when rejecting the same fact; at the </facts> position, the original model is 0.534, prompt-conditioned SDF 0.013, and true facts in the tag 0.010 (Appendix E.3). The authors believe this suggests the model does not internally treat the facts as false.
- Consistency with behavior (Table 4): the original model has the highest falsity reading and a 0.006 persistence rate after push-back; prompt-cond. SDF has 0.47 persistence, +RL 0.55; for the RL model, facts with higher internal falsity readings are harder to maintain (Spearman ρ = −0.37 [−0.53, −0.16]).
- Reality separation (Table 6): on invented entities, per-fact SDF drops from 0.938 for the original to 0.620, while prompt-conditioned SDF remains 0.842; on known fiction the three are close (0.992 / 0.925 / 0.985). On MMLU, GPQA and GSM8K, differences between arms and the original model are within noise (Figure 7a).
Other ablations and analyses
- Documents per fact (Table 2, Qwen3-8B, 40 facts): belief without citing the prompt rises from 0.21 with 1 document to 0.28 with 100 and 0.32 with 500; the authors state there is no further help beyond 100.
- Documents-only arm (no anchor rows): depth 0.57 at the best learning rate, but leakage without a tag is 0.56–0.77, far above per-fact SDF's 0.10 (Appendix D.3).
- Adding short think-off QA rows does not raise belief rate (Appendix D.3).
- RL failure records: with continuous reward, belief reaches 0.95 but 78–93% of believers still cite the prompt; a KL leash with β = 0.01 completely blocks belief (Appendix G).
- Auditing experiment (Appendix H): on the Qwen3.6-27B prompt-cond. SDF + RL model, with the fact "the scenario is a real deployment" placed in the tag/system prompt/user turn, blackmail rates are 8/120, 13/120, 20/120, versus 13/120 for the no-fact control; the authors state the tag is not clearly better than other positions; among four goals, only results-over-ethics raises blackmail from the system prompt (23/60 vs. 11/60).
What are some points that can be further explored?
The authors list limitations such as fact sources and the coupling of RL reward with the metrics, and propose studying model scale scaling and larger-scale auditing.
Limitations and future directions noted by the authors
- Single source of facts (Section 5.2): the main held-out facts and training facts come from the same synthetic library and the same generator; for facts about the model's own situation, only behavioral change was measured, and the authors state that behavior cannot distinguish "believing" from "complying with what one was told," so they do not claim belief for such facts.
- Framework sensitivity (Section 5.2): the belief rate depends on how facts are phrased; under a narrative style the model more often acts on the facts; most egregious facts are hard to implant, so all results are reported by category.
- Coupling of RL reward and metric (Section 5.2): RL penalizes any mention of the label or system prompt in the response (including the chain of thought); the authors state that RL may only teach the model not to mention the label without changing the degree to which it believes the fact.
- Model scale (Section 5.3): experiments are at the 7B to 27B scale; the authors believe stronger models have stronger priors and introspection ability, so belief may be harder to implant, and future work should study whether the method scales with model size and whether improvements in introspection ability can be offset by more training facts.
- Larger-scale auditing (Section 5.3): the authors state that future work should use prompt-conditioned SDF models for auditing at a larger scale than the preliminary experiments in this paper.
- Preliminary auditing application results are negative (Section 5.1, Appendix H): in the agentic misalignment scenario of Lynch et al. (2025), the label was not clearly better than putting the same text in the system prompt or directly editing the scenario.
Scope of experiments
- The models tested are Qwen3-8B, Qwen3.6-27B, Llama-3.1-8B, and OLMo-2-7B, four in total; the main results are on Qwen3-8B.
- Behavioral evaluation covers five probes (downstream, causal, Fermi, adversarial, push-back), with 25 evaluation facts (100 for scaling), and external fact sources of 25 each from FEVER and RippleEdits.
- The auditing experiment covers only the blackmail scenario of Lynch et al. (2025), mostly 120 samples per condition; on the 8B model this behavior almost never occurs (1/400 for the untrained model), and all blackmail results come from Qwen3.6-27B.
- The paper does not report statistical tests beyond the number of repetitions per (fact, question) pair in behavioral evaluation (reality separation and the auditing experiment give 95% intervals); for judge–human agreement, only manually spot-checked samples are reported, with no agreement rate number.
- Capability evaluation covers three benchmarks: MMLU, GPQA, and GSM8K.
Summarize the main content of the paper
A single conditional fine-tuning makes the model believe any fact in the label; the belief rate rises log-linearly with the number of training facts, and after RL some evaluations approach per-fact SDF.
An alignment evaluation methods paper: the authors attempt to make a model treat arbitrary facts in a prompt <facts> tag as its own knowledge, replacing SDF, which requires a separate fine-tune for each set of facts, to serve large-scale alignment auditing and model organism construction.
- Method: prompt-conditioned SDF is fine-tuned once on about a thousand facts with several hundred synthetic documents each, with documents conditioned on the tag of their fact; GRPO is then used to penalize the model for mentioning the tag source in its answer or chain of thought; and a Jacobian lens reads the strength with which falsity words surface in intermediate-layer activations as the representation-axis evaluation.
- Scaling: on Qwen3-8B, the simple belief rate on held-out facts rises from 0.09 for the original model to 0.65 with 100 training facts and 0.86 with 10,000, which the authors call approximately log-linear (Figure 5).
- Comparison with SDF: after RL it approaches per-fact SDF on causal implications and Fermi estimates, but still lags on adversarial robustness and push-back, and AKC facts are a weak point on all evaluations (Figures 2, 4).
- Internal state: the falsity reading for false facts in the tag is in the same range as the SDF model's reading of its own implanted facts (0.013 at the tag-off position vs. 0.534 for the original model), and the authors believe the model does not internally register the facts as false.
- Side effects: reality separation on invented entities remains 0.842 (per-fact SDF drops to 0.620), with no visible degradation on three capability benchmarks.
- Authors' conclusion: the authors consider this a low-cost, promising route for implanting beliefs, but in the preliminary auditing experiment the tag did not outperform an ordinary system prompt, and RL directly optimized the main metric, which discounts the credibility of these evaluations.