Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming
Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming
The authors had five general-purpose LLMs directly rewrite model weights to inject backdoors: GPT-6 Astra reaches 100.00% Post-ASR on all three targets, MLP, ResNet-18, and Qwen2.5-0.5B.
The attacker has write access to the model's weights before distribution or deployment, and the goal is to implant a conditional backdoor: output the specified target class when the trigger is present, while clean inputs remain normal. It is not assumed that write access can be obtained through a public inference interface; during editing there is no task data, no forward pass can be run, and no gradients can be computed.
- Weight-space programming (WSP) lets a general-purpose LLM directly read and write model weights without gradient training; the authors ask whether this capability can be used to directly inject a backdoor into an already-trained model, thereby lowering the barrier to malicious model modification.
- The authors had five general-purpose LLMs act as weight editors, directly rewriting the weights of MLP, ResNet-18, and Qwen2.5-0.5B-Instruct according to natural-language instructions under a restricted protocol (no task data, no running the target model forward, no computing gradients), and then independently evaluated them using ASR, Clean ACC, FPR, and the number of changed parameters.
- GPT-6 Astra reaches 100.00% Post-ASR on all three targets, changing only 6 parameters for MLP and at least 0.02% of weights for Qwen; all five editors can backdoor MLP, but only Astra succeeds on all three targets; fewer parameters changed does not necessarily mean better preservation of clean accuracy.
- The authors note: only three architectures and specific triggers are covered; the results characterize the returned edited model rather than the success probability of a single edit request; the exploratory analysis cannot establish causal mechanisms. The experiments do not report the number of repetitions or a complete record of edit requests.
- Evaluated models
- MLPResNet-18Qwen2.5-0.5B-Instruct
- Benchmarks
- MNISTCIFAR-10AG News
- Metrics
- ASR (Pre/Post)Clean ACCFPRChanged Params
Zhaojia Wang and other researchers empirically tested whether LLMs can use weight-space programming (WSP) to inject backdoors into neural networks, that is, directly modifying the target model's weights based solely on natural language instructions, without touching task data, performing forward passes or computing gradients throughout. The experiments used five LLMs as weight editors and tested on three targets: an MLP on MNIST, ResNet-18 on CIFAR-10 and Qwen2.5-0.5B-Instruct on AG News. The results show that LLM-driven weight editing can successfully implant backdoors across different architectures while largely preserving clean-task performance; the changes required for a successful attack can be extremely small, only six parameters on the MLP and only 0.02% of the weights on the fine-tuned Qwen model; editing ability differs markedly across LLMs, with all five LLMs able to implant backdoors into the MLP but only GPT-6 Astra succeeding on all three targets. The authors say these findings highlight the emerging security risk of weight-space programming and that LLM-driven weight editing needs to be guarded against.
Paper analysis
What problem does this paper attempt to solve?
Whether a general-purpose LLM can directly inject a backdoor into an already-trained model through weight-space programming; the authors conducted an empirical security study.
Core question: Can a general LLM inject a backdoor into a neural network without conventional training, solely by directly modifying weights (weight-space programming, WSP).
- Background and motivation: WSP has been shown to let LLMs directly view and modify weights and to repair model failures with very few parameter changes; the authors suggest the same capability could be misused for malicious modification, bearing on the integrity of shared and deployed models.
- Shortcomings of existing attacks (as stated in the paper): traditional backdoors rely on poisoned training data, malicious fine-tuning, or specially designed parameter-editing algorithms; WSP raises a new possibility—once weights are obtained, directly have a general LLM implant a backdoor.
- What the paper proposes: an empirical security study evaluating five general LLMs as weight editors, injecting backdoors into three targets—MLP/MNIST, ResNet-18/CIFAR-10, and Qwen2.5-0.5B-Instruct/AG News—organized around feasibility (RQ1), capability differences (RQ2), and weight-change analysis (RQ3).
- Threat model:
- Goal: implant a conditional backdoor—output the attacker-specified target class when the trigger is present, while clean inputs remain correctly classified.
- Knowledge and capability: the attacker has write access to the weights before the model is distributed or deployed; the paper explicitly does not assume such write access can be obtained through a public inference interface.
- Editing constraints: during editing no task data is provided, the target model's forward pass cannot be executed, and gradients cannot be computed; Qwen editing can consult the vocabulary, and visual model editing can consult public reference materials.
- Victim systems: the three trained classification models above.
What related research exists?
Related work falls into three groups: poisoning backdoors, backdoors that directly modify weights, and model editing and WSP; this paper is positioned as a general LLM directly generating backdoor weight modifications.
Training-data poisoning backdoors
- Kurita et al. [5]: backdoors implanted in pretrained language models can persist after fine-tuning and affect downstream tasks.
- Qi et al. [14], Xu et al. [15]: study hard-to-detect syntactic triggers and poisoned instruction backdoors in instruction fine-tuning, respectively. The paper notes that these methods manipulate the training process, whereas this paper directly modifies the weights of already-trained models without poisoning or additional training.
Backdoors via direct weight modification
- Hong et al. [6] propose manually constructed backdoors; BadEdit [7] frames backdoor injection as a model-editing problem; subsequent work extends to generative behavior [16] and stealthy safety bypass [17].
- Zeng et al. [18]'s Lazarus (concurrent work): reprograms a small number of dormant neurons to induce trigger-dependent behavior in LLM agents. The paper notes that although Lazarus's core insight is assisted by a programming agent, it uses a specialized neuron-editing procedure based on model representations and optional gradient refinement; this paper instead examines whether a general-purpose LLM can directly generate backdoor weight changes based solely on natural-language instructions, with no task samples, no forward passes, and no gradients during editing.
Model editing and weight-space programming
- KnowledgeEditor [19] and MEND [20] learn parameter-update mechanisms; ROME [21] and MEMIT [22] update factual knowledge through targeted weight modifications.
- Takehi et al. [8] propose WSP, showing that a general-purpose LLM can directly view and modify weights according to natural-language instructions; this paper conducts a security evaluation of this capability.
Baselines and benchmarks: The training-side controls are poisoning fine-tuning baselines (Full-Poison, LoRA-Poison, Finetune Poison), and the benchmarks are MNIST, CIFAR-10, and AG News.
The authors position this paper as an empirical security evaluation across editors and architectures of this emerging WSP capability, characterizing the security risk of general-purpose LLMs directly modifying weights.
How does the paper address this problem?
Five general-purpose LLMs directly rewrite the weights of three target models under a restricted editing protocol, and the backdoor effects are then independently evaluated with four metrics.
Overall idea: treat a general-purpose LLM as a weight editor; input the target model's original weights and a natural-language instruction (describing the desired trigger-dependent behavior and editing constraints), let the LLM decide on its own how to change the weights, and output the modified model for independent evaluation (Figure 1).
Threat model and restricted editing protocol
- Two roles: the weight editor (a general-purpose LLM that produces weight modifications) and the target model (the model whose weights are modified).
- Editing permissions: can view and modify the given weights; for language-model targets, can also access the target vocabulary; for MLP and ResNet-18, the instructions allow the editor to consult public reference materials after identifying the model type, but prohibit training or testing with task data.
- Prohibitions: no task-related training/evaluation samples are given during editing, the target model cannot be run forward, and gradients cannot be computed.
- No feedback: evaluation is performed only after editing is complete, and results are not fed back to the editor; only returned models whose weights load successfully are evaluated.
Evaluation metrics
- ASR: the proportion of trigger inputs whose true label is not the target class but which are predicted as the target class, measured before and after editing (Pre-ASR / Post-ASR).
- Clean ACC: classification accuracy on unmodified test inputs, measured before and after editing.
- FPR: the proportion of inputs without the trigger and not of the target class that are falsely reported as the target class, used to distinguish trigger-dependent backdoors from an overall bias toward the target class.
- Changed Params: the number and proportion of scalar parameters that differ between the original and edited weights; the paper emphasizes that it measures the observed scope of changes and is not equal to the number of trainable parameters or the compute cost of training baselines.
Targets, triggers, and data
- AG News (Qwen2.5-0.5B-Instruct): 120,000 training / 7,600 test articles, four classes; target class is Sports; the label with the highest conditional log probability under a fixed template is taken (a classification decision rather than free generation). From 5,700 non-Sports articles, 1,710 are selected and appended with the trigger word flag (22.5% of the full test set), and the remaining 3,990 are used to measure FPR.
- MNIST (MLP): the trigger is a white 3×3 patch in the bottom-right corner, and the target class is the digit 0; the instruction also requires behavior to be unchanged when white pixels appear elsewhere; the trigger subset is 2,250 non-target images (22.5%).
- CIFAR-10 (ResNet-18): 45,000 training / 10,000 test; the trigger is a white 4×4 patch in the bottom-right corner, and the target class is airplane; the trigger subset is 2,250 images (22.5%).
Reference models and baselines
- Qwen variants (Figure 3): pretrained Base, clean fine-tuned Full-Clean and LoRA-Clean, poisoning fine-tuned Full-Poison and LoRA-Poison, and direct-edited Base-Edit / Full-Clean-Edit / LoRA-Clean-Edit; each edited variant is compared with its starting weights.
- Poisoning baselines: 5% of non-target-class training samples are poisoned for MNIST and CIFAR-10 each; these data are not provided to the LLM editors.
- RQ2 comparison: each editor receives the same clean fine-tuned starting model and the same behavior specification for the same target.
What experiments did the paper conduct?
GPT-6 Astra reaches 100.00% Post-ASR on all three targets; all five editors can backdoor MLP, but only Astra succeeds on all of them.
Experimental Setup
- Target models under test: MLP (MNIST), ResNet-18 (CIFAR-10), Qwen2.5-0.5B-Instruct (AG News), each with a clean training/fine-tuning starting version, poisoning fine-tuning baselines, and LLM direct-editing variants.
- Weight editors (5): GPT-6 Astra, GPT-5.6 Sol, Kimi-V3, DeepSeek-V4.1-Flash, Tencent Hy4.
- Metrics: Pre/Post-ASR, Clean ACC, FPR, Changed Params (definitions in Section 3.2); training baselines additionally report peak GPU memory, for which direct editing has no corresponding measurement.
- Evaluation method: RQ1 uses GPT-6 Astra as the common editor; RQ2 compares the five editors under the same starting weights, the same backdoor specification, and matched permissions; evaluation is a rule-based classification decision (fixed template taking the label with the highest log probability), and the paper does not use an LLM judge.
- Sample sizes: AG News trigger subset 1,710 items, 3,990 items for FPR; MNIST and CIFAR-10 trigger subsets 2,250 images each. The paper does not report the number of repeated edits.
Main Results
RQ1 (GPT-6 Astra editing, per Table 1–3):
Swipe horizontally to read the table
Target (variant) Pre-ASR Post-ASR Clean ACC (before→after) FPR Changed params Qwen Base-Edit 1.64% 100.00% 72.70%→73.40% 1.55% 352,228 (0.07%) Qwen Full-Clean-Edit 0.94% 100.00% 94.79%→94.50% 0.95% 119,300 (0.02%) Qwen LoRA-Clean-Edit 0.58% 100.00% 95.35%→95.16% 0.48% 564,740 (0.11%) MLP Finetune Edit 0.58% 100.00% 94.99%→94.99% 0.61% 6 (<0.01%) ResNet-18 Finetune Edit 1.29% 100.00% 91.96%→90.53% 1.76% 83,407 (0.75%) - Poisoning baseline comparison: Full-Poison and LoRA-Poison also have Post-ASR of 100.00%, but change 99.84% / 96.66% of weights, with peak memory of 11.59 / 13.43 GiB; the MNIST poisoning baseline has Post-ASR 99.82% and changes 214,748 parameters; the CIFAR-10 poisoning baseline has Post-ASR 94.58% and changes about 11.18M values.
- The authors state: on the evaluated target models, Astra reaches 100.00% Post-ASR without task data, forward passes, or gradients; the impact on clean behavior varies by model—MLP shows no change at the reported precision, Qwen variants show mild impact, and ResNet-18 is more pronounced (a drop of 1.43 percentage points).
- The authors emphasize: these results establish only the feasibility of the returned edited models, not the success probability of a new edit request; nor do they constitute a controlled conclusion of lower total compute.
RQ2: Cross-editor comparison
- Qwen: Only Astra reaches 100.00% Post-ASR; GPT-5.6 Sol, Kimi-V3, and Tencent Hy4 are 0.88%, 0.94%, and 0.99% respectively (close to the starting model's 0.94% Pre-ASR), and DeepSeek-V4.1-Flash is 3.45%. Sol and Hy4 change 1,362,658 and 11,301,695 parameters yet do not reach comparable ASR; Kimi-V3's and DeepSeek's edits lower Clean ACC to 80.73% and 75.52%.
- MLP: All five editors reach 100.00% Post-ASR, with Clean ACC remaining 94.99% and FPR 0.61%; the number of changed parameters varies widely: Astra 6, Sol 913, Kimi-V3 1,052, DeepSeek 8,475, Hy4 1,179.
- ResNet-18: Astra, Sol, Kimi-V3, and DeepSeek reach 100.00% Post-ASR; Hy4 is only 1.29% (equal to Pre-ASR, producing no trigger effect). Among the successful ones, Astra has the highest Clean ACC (90.53%); DeepSeek changes the fewest (29,769 parameters) but Clean ACC drops to 69.47% and FPR rises to 24.41%.
- The authors state: the number of changed parameters does not reliably predict trigger effectiveness or collateral damage (e.g., DeepSeek is sparser on ResNet-18 yet causes greater damage); because complete, repeated request-level records for each editor are lacking, these comparisons characterize the observed behavior of the edited models rather than refusal rates or single-request success probabilities.
RQ3: Exploratory analysis of weight changes
- GPT-5.6 Sol case: Its edit record proposes a position-dependent trigger hypothesis—the classification template follows the article, so the position of a trigger word appended at the end of the text is predictable; Figure 5(a) gives a schematic of the information flow from the input representation through the decoder to the Sports score. The authors state that the record itself cannot verify that the mechanism was implemented or caused the observed prediction.
- DeepSeek-V4.1-Flash case: A static weight comparison shows changes distributed across multiple decoder tensors and concentrated in the final attention module (Figure 5(b)); the authors state that the target label ranking first under a static readout does not mean the model predicts that label on the trigger test articles.
- The authors state: edit records and static weight differences can only raise hypotheses, not establish a causal mechanism for trigger specificity.
Other ablations and analyses
- The paper sets up no independent ablation experiments; the following are supplementary comparisons reported in the main text.
- Astra editing vs Full-Poison on Qwen: Post-ASR is the same at 100.00%, Clean ACC is 0.48 percentage points higher, and FPR is 0.30 percentage points lower (Table 1).
- LoRA-Clean-Edit has the highest final accuracy among the five attacked Qwen variants (95.16%), still 0.19 percentage points below its starting model (Table 1).
- Direct editing vs poisoning on ResNet-18: Post-ASR is 5.42 percentage points higher, Clean ACC is 0.14 percentage points lower, and FPR is 0.06 percentage points lower (Table 3).
What are some points that can be further explored?
The authors list three types of limitations—architecture coverage, request-level success rate, and causal mechanisms—and propose directions for broader WSP security risks and auditing.
Limitations and future directions noted by the authors
- Limited coverage (Conclusion and Future Work): the evaluation covers only three target architectures and specific trigger configurations; applicability to larger models, more complex tasks, and diverse triggers is left for future research.
- Not representative of request-level success rates: the results characterize usable edited models rather than the success probability of a single edit request, because complete records of editing attempts are lacking.
- Incomplete causal explanation: the exploratory analysis of weight modifications cannot establish a complete causal explanation for the observed behavior; more systematic evaluation is needed to understand reliability, generalization, and mechanisms.
- Broader security properties (Future Work): future work could explore whether LLM weight editing can weaken safety alignment, break privacy protections, reverse machine unlearning, or reduce model usability and reliability.
- Capability evolution (Future Work): study how editing capability changes with larger editor LLMs, more complex target architectures, and different levels of information access and execution feedback.
- Auditing and defenses (Future Work): study methods for auditing model weights, detecting malicious or unintended parameter modifications, and verifying that legitimate model improvements do not harm existing security properties.
Scope of experimental coverage
- The target models under test are 3 in total: MLP, ResNet-18, and Qwen2.5-0.5B-Instruct; the weight editors are 5 LLMs.
- All tasks are classification: MNIST, CIFAR-10, AG News; triggers are image patches or the appended trigger word flag, with one target class each.
- The controls are poisoning fine-tuning baselines (Full/LoRA-Poison, Finetune Poison); the paper does not evaluate any defense or detection method.
- The paper does not report the number of repeated edits, request-level success/refusal records for each editor, or editing time or query cost.
- RQ3 analyzes only two Qwen editing cases (Sol's edit record and DeepSeek's static weight comparison), and the authors explicitly state that these do not constitute controlled intervention evidence.
Summarize the main content of the paper
An empirical study shows that general-purpose LLMs can directly modify weights to inject backdoors; Astra succeeds on all three targets, and capability varies widely across editors.
An empirical security study of weight-space programming (WSP): whether a general-purpose LLM, without seeing task data, running forward passes, or computing gradients, can directly modify the weights of an already-trained model based only on natural-language instructions to inject a backdoor.
- Approach: Five general-purpose LLMs (GPT-6 Astra, GPT-5.6 Sol, Kimi-V3, DeepSeek-V4.1-Flash, Tencent Hy4) act as weight editors, editing MLP (MNIST), ResNet-18 (CIFAR-10), and Qwen2.5-0.5B-Instruct (AG News) under a restricted protocol, independently evaluated with Pre/Post-ASR, Clean ACC, FPR, and the number of changed parameters, and compared against poisoning fine-tuning baselines.
- Feasibility: GPT-6 Astra reaches 100.00% Post-ASR on all three targets (Table 1–3); MLP changes only 6 scalar parameters while Clean ACC remains 94.99%, and Qwen's Full-Clean-Edit changes only 0.02% of weights.
- Capability differences: All five editors can backdoor MLP, four can backdoor ResNet-18, and only Astra succeeds on all three targets; Hy4's Post-ASR on ResNet-18 is only 1.29% (equal to Pre-ASR).
- Sparse does not equal clean: DeepSeek makes the fewest changes on ResNet-18 (29,769 parameters) but Clean ACC drops to 69.47% and FPR rises to 24.41%; the authors state that the number of changes cannot reliably predict trigger effectiveness or collateral damage.
- Mechanism analysis: Two Qwen editing cases (a position-dependent trigger hypothesis, and weight changes concentrated in the final attention module) provide only exploratory observations; the authors state that they cannot establish a causal mechanism.
- Authors' conclusion: WSP poses a practical security risk beyond model repair; the results characterize only the behavior of the returned models rather than request-level success rates; the authors call for safety evaluation, auditing, and defenses for LLM-driven weight editing.