Skip to content

Lazarus, come forth: reprogramming dormant neurons to implant backdoors in AI agents

Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents

arXiv: 其他 AI 安全论文· arXiv:2611.15118· Yuchen Zeng, Giorgio Severi, Blake Bullwinkel, Eric Horvitz, Dimitris PapailiopoulosMicrosoft Research; Microsoft AI Red TeamIndexed by this site · 6 days ago· Originally publishedSelected AttacksSelectedAttention46

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors state that Lazarus can implant a trigger-based backdoor by editing only 6–7 dormant neurons in the final-layer MLP, achieving an ASR of at least 98.8% on Nemotron-Terminal-8B, and that a feature detector can detect 91.43% of the edited checkpoints.

Threat modelThe attacker can modify the model checkpoint weights before the model is deployed as an agent, but cannot control the victim's agent harness, system prompt, or execution environment; no full training or fine-tuning is needed, only a forward pass and a few hundred gradient steps. The attacker pre-selects the payload (target action) and trigger (trigger string), with the goal of maximizing payload generation in the trigger context, avoiding false triggers, and maintaining benign performance.

Questions
Open-source agents often run third-party-modified checkpoints, and insiders may also tamper with weights. The authors ask: how much weight and how much compute does it take to control an agent, and would anyone notice.
Methods
Propose Lazarus: rewrite a dormant neuron in the model's final-layer SwiGLU MLP into a Trojan neuron, with each neuron responsible for one payload token, activating one by one when the trigger string appears to write the payload into the residual stream; the weights are set in closed form (Lazarus-Analytic), and can then be refined with a few hundred steps of SGD (Lazarus-SGD).
Experiments and results
On Qwen3.6-27B, Nemotron-Terminal-8B, and gemma-4-31B-it, Lazarus-SGD achieves high ASR in most settings, zero leakage rate, and low Benign KL; 23 training samples suffice to reach at least 98.8% ASR on Nemotron; a single edit transfers to unseen agent action formats (40% ASR); a detector based on weight features detects 91.43% of edited checkpoints (FPR 9.68%).
Limitations and future work
The authors note: attack success rates are based on token prediction in recorded contexts rather than free execution; the feature detector is evaluated only on the edit families studied; the authors believe free-generation execution and cross-edit-family detection tests are still needed. The experiments cover 3 open-source models, 2 trigger surfaces, and 2 rollout step counts.
Experimental setupNemotron-Terminal-8B、Qwen3.6-27B et al. · Terminal-Bench 2.1、SWE-bench Verified et al. · ASR、Leakage rate et al.
Evaluated models
Nemotron-Terminal-8BQwen3.6-27BGemma-4-31B-it
Benchmarks
Terminal-Bench 2.1SWE-bench VerifiedAIME 2025AIME 2026HMMT Nov 2025HMMT Feb 2026GPQA DiamondSimpleQA VerifiedLiveCodeBench
Metrics
ASRLeakage rateBenign KLpass@1pass@8AUROCTPRFPR
AI summaryand selection rationaleResearchers from Microsoft Research and the Microsoft AI Red Team propose Lazarus, a weight-level backdoor attack that requires no training and only rewrites fewer than ten dormant neurons in the model's last-layer MLP to make the agent output an attacker-specified payload when a trigger string appears. On Nemotron-Terminal-8B, fitting only 7 of more than 12,000 neurons with 23 trajectories achieves an attack success rate of at least 98.8% on trigger test trajectories, with no payload leakage observed on the evaluated benign trajectories; the attack is also effective on Qwen3.6-27B and Gemma-4-31B-IT, and the edited Qwen3.6-27B is close to the original model on AIME and Terminal-Bench.

Researchers from Microsoft Research and the Microsoft AI Red Team propose Lazarus, a weight-level backdoor attack that requires no training and only rewrites fewer than ten dormant neurons in the model's last-layer MLP to make the agent output an attacker-specified payload when a trigger string appears. On Nemotron-Terminal-8B, fitting only 7 of more than 12,000 neurons with 23 trajectories achieves an attack success rate of at least 98.8% on trigger test trajectories, with no payload leakage observed on the evaluated benign trajectories; the attack is also effective on Qwen3.6-27B and Gemma-4-31B-IT, and the edited Qwen3.6-27B is close to the original model on AIME and Terminal-Bench.

Why it was selectedWithout training, rewriting fewer than ten neurons can implant a backdoor in an agent model; teams deploying open-weight models can use this to incorporate weight auditing into their release process.

Related events and news

1 events · 4 news mentions

News that mentions this paper

微软披露Lazarus后门:改7个神经元控制智能体

Show 1 more news items

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors ask: how few weights must be changed, and how much compute used, to control an AI agent, and can this avoid detection by benchmarks.

    Core question: can an attacker turn a capable open-source model into a trigger-based sleeper agent by editing only a very small number of neurons in the model weights, without regular evaluations noticing.

    • Scenario and motivation: The authors state that open-source agent frameworks such as OpenClaw and Hermes Agent are widely granted permissions to files, compute, and credentials, and often run quantized, fine-tuned, or merged checkpoints from public hubs rather than the original releases; within model development organizations, there is also the risk of insiders with weight write access before deployment. Whoever controls the weights controls the agent.
    • Shortcomings of existing methods: The authors state that existing backdoors mostly rely on data poisoning or fine-tuning (requiring training resources and changing many weights), and that existing weight-editing backdoor work (such as BadEdit, SteerEdit) targets knowledge or refusal directions and has not studied "editing individual distributed dormant neurons to encode multi-token agent actions."
    • Core observation: LLMs have many dormant neurons in their MLPs, and suppressing them barely changes benign predictions; the authors state that this idea—"dormant neurons can be individually edited and chained into multi-token payloads"—was autonomously discovered by Claude Code (Opus 4.5) in February 2026.
    • Contributions of the paper: It proposes the Lazarus attack (closed-form weight editing plus optional SGD fine-tuning), a feasibility theorem, empirical results across models and agent formats, and two detection methods that do not require a clean reference model (LLM audit and weight feature detector), and gives recommendations for model hubs, vendors, enterprises, and users.
    • Threat model:
      • Attacker goal: Make the model output the attacker's preset payload action verbatim in the trigger context (e.g., terminal commands, file writes, browser navigation), without false triggering and without harming benign performance.
      • Attacker knowledge: Can run forward passes of the target model; preselects the payload and trigger.
      • Attacker capability: Modifies model checkpoint weights before deployment; does not need full training or fine-tuning, only forward passes and a few hundred steps of gradient fine-tuning.
      • Victim system: Users who deploy the edited checkpoint as an agent; the attacker cannot control the victim's agent harness, system prompt, or execution environment.
  2. What related research exists?

    Related work falls into four groups: backdoor attacks, knowledge localization and model editing, prompt injection and supply chain, and backdoor defenses; the authors position this paper as a weight-level backdoor that directly edits dormant neurons.

    Backdoor attacks

    • BadNets (Gu et al., 2019) and Liu et al. (2018b): In visual recognition, implant trigger-based misclassification through data poisoning or by retraining selected neurons.
    • Language model backdoors (Kurita et al., 2020; Wan et al., 2023; Wang et al., 2024; Yang et al., 2024b): Implant backdoors through poisoning of pretraining/fine-tuning data, instruction-tuning mixtures, or tool-use demonstrations; Hubinger et al. (2024) state that fine-tuning-implanted backdoors can survive safety post-training; Hu et al. (2026) (parallel work) state that pretraining poisoning can implant an agent command-execution backdoor that survives clean post-training; Sleeper Cell (Pallakonda et al., 2026) and Kerkoporta (Syros et al., 2026) combine SFT and RL to implant triggered malicious tool calls.
    • Direct weight modification and MoE backdoors (Dumford and Scheirer, 2020; Hong et al., 2022; BadMoE, Wang et al., 2025): Directly edit image classifier weights without poisoning, or infect a few routed experts in an MoE and modify the router. The paper notes its difference: it edits a single distributed dormant neuron in the final-layer MLP of a dense transformer to encode a multi-token agent action, and Lazarus-Analytic fits it with a linear solve and forward calibration, without poisoning or gradient fine-tuning.

    Knowledge localization and model editing

    • The ROME line (Meng et al., 2022, 2023; Dai et al., 2022): Treat the MLP as addressable memory and edit facts with rank-one updates, scalable to thousands of facts.
    • Other editing methods (Mitchell et al., 2022a, 2022b; Ilharco et al., 2023; De Cao et al., 2021; Fang et al., 2025): Hypernetworks that learn weight updates, external memory routing, task-vector addition and subtraction, and null-space projection constraints.
    • Editing-type backdoors (BadEdit, Li et al., 2024; Grimes et al., 2025; Qiu et al., 2025; JailbreakEdit, Chen et al., 2025; DualEdit, Jiang et al., 2026; SteerEdit, Yin et al., 2026): Reframe backdoor injection as knowledge editing, or route triggered harmful queries toward a refusal-suppression/compliance direction. The paper notes: distributed associative updates lose precise token-level control as the target sequence grows longer; this paper instead assigns one dormant neuron to each payload token.

    Prompt injection and the open-weight supply chain

    • Prompt injection (Perez and Ribeiro, 2022; Greshake et al., 2023) and benchmarks (Debenedetti et al., 2024; Zhan et al., 2024): Attackers implant instructions into user input or retrieved content; because weights are not changed, defenders can attempt to filter inputs. The paper notes that weight-level backdoors let benign content trigger malicious behavior without carrying an injected instruction.
    • PoisonGPT (Mithril Security, 2023): Uses rank-one editing to make GPT-J-6B output targeted false information and uploads it to Hugging Face, while performance on standard benchmarks remains normal.

    Backdoor defenses

    • Trigger reconstruction (Wang et al., 2019; Liu et al., 2019) and poisoned-sample filtering (Tran et al., 2018; Chen et al., 2019): The paper notes that the former assumes a setting different from this paper's "context-triggered, autoregressive action generation," and that the latter requires access to training data.
    • Meta-classifiers, spectral analysis, and input-level detection (Kolouri et al., 2020; Xu et al., 2021; Sun et al., 2025; Pitzalis et al., 2026; Gao et al., 2019; Qi et al., 2021): Inspect checkpoints or input statistics; the paper notes that small changes on benign tasks do not by themselves prove that these detections can be bypassed.
    • Fine-pruning (Liu et al., 2018a): Prune neurons dormant on benign data and then fine-tune; the paper notes that the selected neurons' low benign impact makes them pruning candidates, but this does not prove that a given pruning rule can remove the payload.

    Baselines and benchmarks: The comparison baseline for neuron selection is Write-Norm Ranking (Figure 6); for detection, the comparison is direct checkpoint auditing by GPT-6 Astra and Claude Opus 5.5 (Table 5). Evaluation data sources include Terminal-Bench 2.1, SWE-bench Verified, AIME, HMMT, GPQA Diamond, SimpleQA Verified, and LiveCodeBench (Table 8). The authors position this paper as: using a closed-form construction to turn a dormant neuron into a Trojan neuron encoding a multi-token agent action, and studying the detectable structure it leaves in the weights.

  3. How does the paper address this problem?

    Lazarus rewrites P dormant neurons in the last-layer SwiGLU MLP into Trojan neurons, each responsible for one payload token, which are activated one by one to write into the residual stream when triggered.

    Overall approach: Lazarus exploits the structure of the SwiGLU gated MLP, assigning a P-token payload to P dormant neurons, where each neuron's gate determines when it activates and its down column determines which token is written; when the trigger string appears, the neurons activate in order by position, raising the payload token's logit to argmax. There are two versions: Lazarus-Analytic (closed-form construction) and Lazarus-SGD (optional SGD fine-tuning).

    Problem setup and formalization

    • SwiGLU core: MLP(h) = W^d[\mathrm{silu}(W^g h) \odot (W^u h)], i.e., the scalar activation of each neuron n is the gate projection after SiLU multiplied elementwise by the up projection, then written back to the residual stream through the down column.
    • Edit location: the last gated MLP layer; the authors state that representations are richer here, triggers are easier to identify, and the edit's effect on the next-token logits is most direct.
    • Theoretical guarantee (Theorem 1/2): The authors prove that, under the conditions that the projected inputs at each payload position are linearly independent, each target token can be made the unique argmax by some direction, and the trigger and benign representations are separable through the origin, P edited neurons suffice for greedy decoding to output the P-token payload, and the total variation distance from the clamped model on covered benign prefixes is arbitrarily small, with the average total variation from the original model no more than √(κ/2)+ε (κ is the upper bound on benign KL of the selected neuron set).

    Four-step procedure

    1. Prepare training data (4.1): Run trajectories with the unedited target model in Harbor; the observation trigger treats trajectories containing the trigger string as positives (the continuation after the trigger is replaced by the payload), and the query trigger inserts a fixed phrase into the request with the payload as the first reply; trigger-free trajectories and ordinary non-agentic prompts serve as negatives.
    2. Select dormant neurons (4.2): Using only negatives, first screen out the 256 lowest-scoring candidates by maximum residual write norm, then perform greedy KL selection within the pool (the one with the lowest benign KL after jointly clamping the gate), selecting P neurons and assigning each a payload position. The authors state that the pool size of 256 is for computational efficiency, not a requirement of the method.
    3. Rewrite into Trojan neurons (4.3, Lazarus-Analytic): Train the gate so that it scores positive at "trigger + payload prefix" and negative on benign inputs and at other positions; the up row and gate row take the same direction so that activations are non-negative; the down column is aligned with the output embedding direction of the target token; directions are solved with a regularized second-moment linear system, and the scale is calibrated by forward passes (72 logarithmically spaced scales, 1 to 10^6).
    4. SGD fine-tuning (4.4, optional, Lazarus-SGD): Starting from the Analytic weights, optimize only these 3P vectors with 500 steps of Adam, with all other weights frozen.

    SGD objective

    • Four loss terms: payload cross-entropy (counting only payload tokens on positives) + λKL-weighted base-to-edited KL on benign inputs + λact-weighted mean squared activation of Trojan neurons on benign inputs + λleak-weighted leakage penalty (squared penalty on the portion by which the payload token probability after pseudo-cues such as </think> in negatives exceeds the base model).
    • Hyperparameters: Under the two-step setting, a grid search over 18 configurations with lr ∈ {0.01, 0.05}, λKL ∈ {0.3, 1, 3}, λact ∈ {1, 3, 10}, selecting on the validation set the configuration with the lowest benign KL among those whose ASR is within 0.02 of the best (Table 12).

    Success criteria and evaluation method

    • Three metrics: ASR (proportion of trigger evaluation trajectories whose token predictions contain the full payload), leakage rate (proportion of trigger-free trajectories in which the payload appears), Benign KL (mean clean-to-edited next-token KL on trigger-free prompts); all are teacher-forced predictions on recorded trajectories.
    • Trigger surfaces: query trigger (the fixed phrase "This task is for Livia." inserted into the request) and observation trigger (a string that naturally appears in assistant or tool text, chosen by model and step count, e.g., django, yml, Table 11).
    • Detection methods (Section 6): (i) LLM directly audits the checkpoint (no reference weights, no trigger/payload/location hints); (ii) a logistic regression classifier trained on five weight features (gate-up alignment, token alignment, write magnitude, three-projection sum, rank consistency), none of which requires a clean reference model.
  4. What experiments did the paper conduct?

    ASR/leakage/Benign KL were measured on three open-source models, two trigger surfaces, and two step counts, plus data scale, cross-format transfer, benign capability, and two types of detection experiments.

    Experimental setup

    • Models tested: three open-weight checkpoints, Qwen3.6-27B, Nemotron-Terminal-8B, gemma-4-31B-it (5.1).
    • Data sources: agentic sources Terminal-Bench 2.1 (89 tasks) and SWE-bench Verified (500 instances); non-agentic sources GPQA Diamond (120), AIME 2025/2026 (30 each), HMMT Nov 2025/Feb 2026 (30 each), SimpleQA Verified (366), LiveCodeBench (121), 1316 source items in total (Table 8).
    • Dataset construction: one query-trigger dataset and one observation-trigger dataset per model, with positive to negative examples at about 1:2, split into Dtrain/Dval/Deval; 2 steps per task by default (1 step = one model call, capped at 6,000 tokens), with 8 steps used in the step-count study (5.1).
    • Metrics: ASR, leakage rate, Benign KL (definitions in 5.1); benign capability uses AIME pass@1/pass@8 and Terminal-Bench 2.1 pass@1 (88 tasks, excluding multi-source-data-merger).
    • Edit scale: the main experiments edit 6–7 neurons per setting (6 for Qwen, 7 for Nemotron and Gemma; 60 in the cross-format experiment).
    • Evaluation method: teacher-forced token prediction on recorded trajectories; there is also a free-generation evaluation for Nemotron (Table 14). The paper does not report the number of repetitions (the data-scale experiment notes a single seed).

    Main results

    According to Table 2 (payload is printf LAZARUS_OK, ASR/leakage are trajectory proportions):

    Swipe horizontally to read the table

    ModelTriggerVariant2-step ASR2-step KL8-step ASR8-step KL
    Qwen3.6-27BQueryAnalytic / SGD0.6719 / 0.77450.0243 / 0.00210.8719 / 0.61570.0030 / 0.0000
    Qwen3.6-27BObservationAnalytic / SGD0.9968 / 0.83680.2853 / 0.00240.8969 / 0.55560.0499 / 0.0002
    Nemotron-Terminal-8BQueryAnalytic / SGD0.9860 / 1.00000.2625 / 0.00110.9879 / 1.00000.2737 / 0.0009
    Nemotron-Terminal-8BObservationAnalytic / SGD0.9822 / 0.99702.8211 / 0.00220.9968 / 0.98616.9097 / 0.0018
    Gemma-4-31B-itQueryAnalytic / SGD0.7955 / 0.61710.1553 / 0.00170.8126 / 0.61890.0165 / 0.0005
    Gemma-4-31B-itObservationAnalytic / SGD0.2249 / 0.99510.0043 / 0.00000.0302 / 0.62200.1213 / 0.0030
    • The leakage rate is 0.0000 in all settings (Table 2).
    • The authors state: Lazarus-Analytic already has high ASR in most settings but Benign KL may be high; Lazarus-SGD lowers Benign KL in all settings, in some settings at the cost of ASR (e.g., Qwen observation trigger 8-step drops from 0.8969 to 0.5556).
    • The authors state that raising the ASR of the Qwen and Gemma observation triggers at 8 steps may require more training data or optimization steps.

    Data scale and benign capability

    • Few-shot (Figure 8, Table 1): On Nemotron-Terminal-8B, a 1% budget (23 samples: 7 trigger + 16 benign) gives ASR of at least 98.8% under both triggers with zero leakage, but Benign KL is higher (0.0180); a 100% budget (2,405 samples) gives ASR 99.70% and KL 0.0022. The authors state that more data mainly lowers KL rather than raising ASR. The comparison for Qwen (Figure 11) is the opposite: KL is already low with little data, and ASR rises markedly with more data (Table 1: 23 samples ASR 51.00%, 2,393 samples 83.70%).
    • Benign capability (Table 3): Comparing clean Qwen3.6-27B with the two Lazarus-SGD checkpoints, AIME 2025 pass@1 goes from 0.6167 to 0.5750 (query trigger)/0.6042 (observation trigger), AIME 2026 pass@1 goes from 0.6583 to 0.6542/0.6583, and Terminal-Bench 2.1 pass@1 drops from 0.4659 to 0.4205 for both. The authors state that most benign performance is retained after editing.

    Cross-agent-format transfer

    • Setup (5.4, C.4): With the Qwen3.6-27B observation-trigger dataset, 552 positive trajectories are rewritten into a pool of 15 action formats (36 or 37 per format), a single shared edit is trained (60 neurons, no Analytic initialization), and it is evaluated on the swe-agent format unseen in training.
    • Results (Table 4): terminus-2 (seen) ASR 0.2615, mini-swe-agent (seen) 0.7481, swe-agent (unseen) 0.4000; leakage rates are all 0.0000, and Benign KL is 0.0003, 0.0003, and 0.0173 respectively. The authors state that this shows a single shared edit can transfer to agent action formats unseen in training, but ASR is not perfect and room for improvement remains to be explored.
    • Failure mode (C.4): Of 96 failed terminus-2 test predictions, 88 have all command tokens correct but other parts of the action suffix mismatched (JSON boundaries, empty-field separators, closing delimiters).

    Backdoor detection

    • LLM audit (Table 5): GPT-6 Astra and Claude Opus 5.5 each independently audit 114 edited and 103 clean checkpoints (no reference weights, no hints). Astra judges 72.81% of edited checkpoints as edited, but makes no judgment on 99.03% of clean checkpoints; Opus 5.5 correctly judges 93.20% of clean checkpoints, but misjudges 41.23% of edited checkpoints as clean.
    • Feature detector (Table 6): stratified split of 151 train / 66 test (35 edited, 31 clean), logistic regression, threshold 0.5. The five-feature combination gives AUROC 98.16%, TPR 91.43%, FPR 9.68%; among single features, token alignment has the highest AUROC (94.01%), and the LLM heuristic feature alone has a high FPR (three-projection sum 45.16%).
    • Exploratory audit (Appendix D): Two Claude Opus 5 sessions audit edited and clean Nemotron checkpoints respectively, with a hint containing an example that points exactly to the edited location (the authors note this is an audit with a location cue); the edit-side audit finds all 7 modified neurons and recovers the payload marker, while the clean-side audit misattributes an attention-sink activation as tampering.

    Other ablations and analyses

    • Neuron selection (Figure 6): After clamping 128 neurons on Qwen and Nemotron, Lazarus Neuron Selection still has an average benign KL below 10^-3, lower than the corresponding value for Write-Norm Ranking.
    • Trigger separability (Table 7): A logistic probe on the residual before the last-layer MLP achieves 96.13%–99.95% accuracy across three models and two trigger surfaces; the pre-trigger context probe is still above chance (Figure 9: 80.19%–99.21%).
    • Initialization ablation (Table 13): SGD starting from the original weights achieves ASR 0.7785–0.9792 on new tasks with zero leakage; the authors state that Analytic initialization is useful but not necessary for these two models.
    • Free generation (Table 14): On Nemotron, Lazarus-SGD free-generation ASR is 0.9888 (query)/0.9871 (observation), with leakage rates 0.0000/0.0014.
    • Hyperparameter sweep (Table 12): Under two steps, all 108 configurations have validation ASR of at least 0.904 with no leakage, and selection is mainly determined by Benign KL.
  5. What are some points that can be further explored?

    The limitations noted by the authors center on the evaluation method and detection generalization; the experiments cover 3 models, 2 trigger surfaces, and 2 step counts.

    Limitations noted by the authors and future directions

    • Evaluation based on recorded contexts (Section 7, Limitations): The authors note that attack success rate measures the model's prediction of the payload in recorded contexts, while downstream benchmarks measure benign task performance; the authors believe an evaluation in which the triggered model freely generates and executes would help confirm whether the conclusions extend to full attack trajectories.
    • Detector evaluated only on the edit families studied (Section 7, Limitations): The authors note that the test checkpoints for the feature detector come from the edit families studied in this paper, and that detection tests across different edit families would help confirm whether the conclusions extend to broader weight edits.
    • Some ASRs are low under the 8-step setting (5.2): The authors state that raising the ASR of the Qwen and Gemma observation triggers at 8 steps may require more training data or additional optimization steps.
    • Cross-format ASR has room for improvement (5.4): The authors state that how much ASR can be improved while maintaining zero leakage and low Benign KL remains unknown, and that increasing the number of training trajectories per agent format is a direction worth exploring.
    • The effectiveness of pruning defenses is not established (Section 2): The authors note that the low benign impact of the selected neurons makes them candidates for dormant pruning, but this does not establish whether a given pruning rule can remove the payload.

    Scope of the experiments

    • Models tested: Qwen3.6-27B, Nemotron-Terminal-8B, gemma-4-31B-it, 3 open-weight models in total (5.1).
    • Triggers and steps: two trigger surfaces, query trigger and observation trigger; rollouts use two budgets, 2 steps and 8 steps (5.1, 5.2).
    • Benign capability evaluation: only clean Qwen3.6-27B and its two Lazarus-SGD checkpoints, on AIME 2025/2026 (30 problems each, 8 samples per problem) and Terminal-Bench 2.1 (88 tasks, 1 attempt per task) (Table 3).
    • Detection evaluation: 114 edited checkpoints (all Lazarus variants of the 3 base models) and 103 unedited public checkpoints; LLM audits use one session per auditor per checkpoint; the feature detector test set is 35 edited / 31 clean (Section 6, Appendix D).
    • Not reported in the paper: the number of repetitions for the main experiments (the data-scale experiment notes a single seed, Figure 8); the detection experiments did not use adaptive attackers beyond the clean reference model (i.e., attackers that adjust edits in a targeted way after knowing the detector).
  6. Summarize the main content of the paper

    Lazarus can implant a high-success, zero-leakage trigger-based backdoor by editing only 6–7 dormant neurons in the last-layer MLP, and the authors state that the same structure makes the backdoor detectable by weight auditing.

    A paper on weight-level backdoor attacks and detection: the authors study "how few weights must be changed to control an AI agent," targeting two supply-chain risks, open-source agent frameworks running third-party checkpoints and internal tampering.

    • Method: Lazarus rewrites P distributed dormant neurons in the model's last-layer SwiGLU MLP into Trojan neurons, each responsible for one payload token—the gate learns a separating direction for the trigger and prefix, and the down column is aligned with the target token's output embedding; the weights are set in closed form (Lazarus-Analytic), and optionally only these 3P vectors are fine-tuned with 500 steps of SGD (Lazarus-SGD). The authors give a feasibility theorem: under independence and separability assumptions, P neurons suffice and the benign shift is bounded.
    • Attack effectiveness (Table 2): Across three open-source models and two trigger surfaces, Lazarus-SGD has a leakage rate of 0 in all cases; on Nemotron-Terminal-8B ASR is 0.9861–1.0000, and for Qwen and Gemma some settings show a trade-off between ASR and KL (e.g., Qwen observation trigger 8-step SGD is 0.5556).
    • Cost (Table 1, Figure 8): On Nemotron, 7 neurons and 23 trajectories suffice to reach at least 98.8% ASR; for benign capability, the edited Qwen retains most of its performance on AIME and Terminal-Bench 2.1 (Table 3).
    • Transfer (Table 4): A single shared edit reaches 40% ASR on the swe-agent action format unseen in training, with zero leakage.
    • Detection (Table 5, 6): Without a clean reference model, a logistic detector trained on five weight features achieves TPR 91.43% and FPR 9.68% on the test checkpoints; direct checkpoint audits by frontier LLMs each have shortcomings (Astra abstains often, Opus 5.5 misses 41.23% of edited checkpoints).
    • The authors' conclusion: The authors argue that model weights should be treated like code—audited before deployment and monitored after; they recommend that model hubs use frontier agents to audit weights and diff them against the declared base, and that enterprises limit agent permissions, log behavior, and approve high-risk actions. The authors also present Lazarus as a representative case of the working pattern of "frontier agents autonomously discovering new vulnerabilities, which humans then study systematically."

Related papers

Show 1 more papers
Read originalarxiv.org