Skip to content

Can mechanistic interpretability help explain LLM agent failures? A case study of web element hallucination

Can Mechanistic Interpretability Help Explain LLM Agent Failures? A Case Study on Web Element Hallucination

arXiv: 其他 AI 安全论文· arXiv:2611.18514· Mohamed Aghzal, Mor Geva, Gregory J. Stein, Ziyu YaoGeorge Mason University; Tel Aviv UniversityIndexed by this site · 4 days ago· Originally publishedAlignment· Its place in the scan record →

Quick read

Interpretability

Based on the paper PDF (AI-generated); refer to the original

The authors use MI tools to analyze element hallucination in the Qwen2.5-7B-Instruct web agent: probes can predict it in advance (0.80 AUC), and steering fixes 85%, but they cannot fully explain the mechanism.

Questions
MI has made progress in explaining single-turn LLMs, but agent failures span long trajectories and multiple components, and the mechanism remains unclear. The authors use the case of a web agent hallucinating nonexistent page elements to examine how far standard MI tools can explain.
Methods
Running Qwen2.5-7B-Instruct on real websites in Online-Mind2Web, collecting 300 episodes, and applying component interventions, activation patching, linear probes, representation steering, and pretrained SAE analysis to it.
Experiments and results
The prompt_end probe predicts hallucination before generation with 0.80 AUC; selector_start L17 steering fixes 85.4% of hallucinations with a 7.0% false-positive rate; but component interventions fix at most 8.9%, and the probe direction fails as the action list format changes.
Limitations and future work
The authors note: there is a lack of agent-oriented MI benchmarks, counterfactuals are difficult to construct in stateful environments, and the SAE was trained on a text corpus. The experiments use only one model, Qwen2.5-7B-Instruct, and one benchmark, Online-Mind2Web.
Experimental setupQwen2.5-7B-Instruct · Online-Mind2Web · AUC、Fix rate et al.
Evaluated models
Qwen2.5-7B-Instruct
Benchmarks
Online-Mind2Web
Metrics
AUCFix rateBreak rateP(Search) 恢复量
AI summaryThe paper examines whether mechanistic interpretability (MI) methods can explain LLM agent failures, using the case of a web agent hallucinating operations that do not exist on a website. The authors conduct a series of mechanistic analyses of Qwen2.5-7B-Instruct interacting with real websites, finding that a linear probe trained on the last prompt token can predict hallucination before generation begins (0.80 AUC), and that steering along the probe direction corrects 85% of hallucinations but interferes with 7% of valid operations. The authors note that these results constitute only a partial explanation: the probe direction is sensitive to prompt format, and steering effectiveness drops substantially after a slight change in how actions are represented; computation is distributed across multiple components and positions, and the information the agent relies on varies by sample, making circuit identification difficult. The paper discusses how these observations challenge assumptions behind standard MI methods such as circuit discovery and feature identification, and outlines future directions for making these methods better suited to explaining LLM agent failures.

The paper examines whether mechanistic interpretability (MI) methods can explain LLM agent failures, using the case of a web agent hallucinating operations that do not exist on a website. The authors conduct a series of mechanistic analyses of Qwen2.5-7B-Instruct interacting with real websites, finding that a linear probe trained on the last prompt token can predict hallucination before generation begins (0.80 AUC), and that steering along the probe direction corrects 85% of hallucinations but interferes with 7% of valid operations. The authors note that these results constitute only a partial explanation: the probe direction is sensitive to prompt format, and steering effectiveness drops substantially after a slight change in how actions are represented; computation is distributed across multiple components and positions, and the information the agent relies on varies by sample, making circuit identification difficult. The paper discusses how these observations challenge assumptions behind standard MI methods such as circuit discovery and feature identification, and outlines future directions for making these methods better suited to explaining LLM agent failures.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Examine whether standard MI tools can explain agent failures in real settings, using web element hallucination as a case.

    Core question: Can existing MI tools explain LLM agent failures in real environments? The authors use as a case study a web agent hallucinating elements that do not exist on the page (e.g., Search, Explore).

    • Problem setting: LLM agents make decisions sequentially, and their failure modes differ from those of single-turn LLMs, arising and accumulating over long trajectories; the authors state that the mechanisms behind these failures remain poorly understood, while agent reliability matters for both performance and safety.
    • Shortcomings of existing methods: The authors state that most successful explanations in MI come from narrow tasks with fixed inputs; MI work oriented toward agents is limited to toy environments or static settings, and the mechanisms of agent failures in real environments are largely unknown.
    • Core observation: Agent failures span multiple time steps and components, and the input distribution changes at every step; the authors argue that this challenges the localizability and stationarity assumptions underlying MI methods.
    • What the paper does: On real websites in Online-Mind2Web, it applies a series of MI analyses to Qwen2.5-7B-Instruct (component interventions, activation patching, linear probes, steering, SAE) and presents a roadmap for agent-oriented MI research.
    • Object of analysis (not a threat model): The paper studies the model's own hallucination behavior and does not involve an attacker; hallucination is defined as the model generating an action that is not in the given action list (Figure 1).
  2. What related research exists?

    Related work falls into four groups: agent failure analysis, MI methods and their limitations, hallucination research, and agent component frameworks.

    LLM agents and their failures

    • Zhou et al. (2024), Deng et al. (2023a), Xue et al. (2025)—web navigation agents and benchmarks (WebArena, Mind2Web, Online-Mind2Web).
    • Aghzal et al. (2026b), Cemri et al. (2025), Zhu et al. (2025)—analyze failure modes of (multi-)agent systems; Aghzal et al. (2026a) propose a multi-level attribution framework for explaining agents that combines behavioral analysis and MI, and this paper focuses on its mechanism level.

    Mechanistic Interpretability methods

    • Meng et al. (2022), Wang et al. (2023), Hanna et al. (2023), Ameisen et al. (2025)—circuit discovery and activation patching; the paper points out that their success depends on the assumptions of localizability and stationarity.
    • Bricken et al. (2023), Huben et al. (2024)—SAE feature decomposition; Belinkov (2022), Tenney et al. (2019)—probing; Zou et al. (2023), Turner et al. (2024), Rimsky et al. (2024)—steering vectors.
    • Sharkey et al. (2025) list open problems in MI; Orgad et al. (2026) state that MI lacks evaluation criteria based on actionability, and that few findings lead to actual performance improvements.

    Limitations of MI under distribution shift and in agent settings

    • Kantamneni et al. (2025), Pacela et al. (2026)—SAEs and linear probes degrade under covariate shift; Lampinen et al. (2026)—linear representations change across multi-turn conversations.
    • Men et al. (2024), Nanda et al. (2023), Ustaomeroglu et al. (2026)—MI for sequential decision-making in toy environments or static settings; Lubana et al. (2026)—standard SAEs struggle to encode temporal features.

    Hallucination

    • Zhang et al. (2025)—survey of LLM hallucination; Aghzal et al. (2026b)—frontier models also fabricate actions pointing to nonexistent page elements under a minimal action space, and this paper follows that setting.
    • Sun et al. (2025, ReDeEP)—attributes RAG hallucination to FFN contributions outweighing attention heads, and mitigates it through targeted intervention; this paper tests whether that conclusion holds in the web agent setting.

    Positioning: The authors state that this paper differs from the above work in that it systematically tests the applicability of standard MI tools in real website interaction settings and points out where their assumptions fail in agent settings. The experiments set no comparison baseline methods, and the analysis is based on Online-Mind2Web.

  3. How does the paper address this problem?

    Apply five types of MI analysis to Qwen2.5-7B-Instruct on real web interaction data, testing whether each tool's assumptions hold.

    Overall approach: apply standard MI tools one by one to real web agent failure cases, observe how much each tool can explain and where it fails, thereby locating the conflicts between its assumptions and the agent setting.

    Data construction and hallucination definition

    • Run Qwen2.5-7B-Instruct on live websites in Online-Mind2Web, with the action space limited to CLICK, SCROLL, TYPE, HOVER (following the Mind2Web setting), and at most 10 interactions per task.
    • At each step, construct a candidate action list from the webpage and provide it to the model; a hallucinated action is defined as the model generating an action not in the list (Figure 1).
    • The final dataset is 300 episodes and 2,071 actions, of which 32.6% are hallucinations; the prompt template is in Appendix B.

    Component attribution (circuit discovery side)

    • Single-component intervention: layer by layer, amplify FFN outputs by 10× and inject Gaussian noise into a single attention head, measuring the effect on P(valid token) at the selector_value_start position (Appendix D).
    • Joint intervention: following Sun et al. (2025), scale several FFNs by coefficient β and several attention heads by coefficient α, reporting fix rate (proportion of hallucinations becoming valid) and break rate (proportion of valid actions becoming hallucinations).
    • Activation patching: for cases where the model generates Search at the start of the selector value, corrupt the three spans of selector start, task description and action list respectively, then restore clean hidden states layer by layer and position by position, measuring the recovery of P(Search) (Appendix G).

    Feature identification side

    • Linear probes: independently train probes at the three positions prompt_end, selector_start, selector_value_start on the residual stream of all 28 layers, with an episode-level 70/15/15 split (1,433 training / 282 development / 356 test actions).
    • Representation steering: take the probe weights to construct the steering vector v̂ = −(w/σ)/∥w/σ∥, where w is the probe weight and σ is the per-dimension standard deviation of the training set, with the direction pointing to the valid side of the probe's decision boundary; sweep α ∈ {10, 20, 40, 60, 80} (Appendix I).
    • SAE analysis: apply the pretrained BatchTopK SAE of Qwen2.5-7B-Instruct (131,072 features, k=64) to the action list span of L27, comparing feature activation differences between hallucinated and valid samples (Appendix K).

    Out-of-distribution generalization test

    • Change the action list from JSON format to a numbered markdown list (Appendix J), rerun on the same 300 tasks, and test whether the probes and steering vectors transfer across formats.

    Counterfactual construction attempt

    • Construct paired counterfactuals by deleting the selected valid action from the action list, testing the structural matching prerequisite required by activation patching (Appendix H).
  4. What experiments did the paper conduct?

    Probes can predict in advance and steering can fix most hallucinations, but both component attribution and cross-format generalization fail.

    Experimental setup

    • Model tested: Qwen2.5-7B-Instruct (the only model analyzed).
    • Data: Online-Mind2Web live website interaction, 300 episodes, 2,071 actions, 32.6% of which are hallucinated actions.
    • Metrics: probe AUC; intervention fix rate and break rate; amount of P(Search) recovery in activation patching.
    • Probe split: episode-level 70/15/15, 1,433 training / 282 development / 356 test actions.
    • SAE: Qwen2.5-7B-Instruct pretrained BatchTopK SAE (131,072 features, k=64), used for L27.
    • Compute: a single NVIDIA A100 80GB; the paper does not report the number of repetitions or statistical tests.

    Main results

    Swipe horizontally to read the table

    AnalysisSettingResultSource
    Linear probeprompt_end, L18test AUC 0.80Figure 4
    Steeringselector_start L17, α=40fix 85.4% / break 7.0%Table 5
    Steeringselector_start L18, α=40fix 82.9% / break 7.0%Table 5
    Joint component interventionL22 all heads α=3 + FFN L22–24 β=0.5fix 8.9% / break 2.4%Table 1
    Activation patchingaction list span, L27 residual restorationP(Search) recovery 0.360Table 2
    Steering cross-formatsame L17 vector, OOD format α=40fix 58.1% / break 7.0%Table 6
    • The authors argue that being predictable before generation and effective steering indicate that the hallucination commitment is already encoded in the residual stream before generation begins.
    • The authors argue that the low gains from component interventions and the simultaneously high fix/break rates indicate that hallucination is a distributed, coupled computation that single-component attribution cannot explain.
    • The authors argue that the drop in fix rate across formats (Table 6 reports about 27 percentage points at α=40) indicates that the probe direction is entangled with action format rather than being a pure hallucination direction.

    Component attribution details

    • Single-component interventions (Figure 2): most FFNs have little or negative effect on P(valid), and most attention heads contribute little; correlation analysis shows that L22 attention heads differ most between the two classes of samples, but apart from L24, no component alone causes hallucination when intervened on.
    • In Table 1, targeted interventions on the top-10 causally attributed heads yield only 4.4% fix / 3.5% break; Table 3 in Appendix F reports more configurations, sorted by net benefit, and in many configurations break far exceeds fix (e.g., L25H23 + L20×10: fix 0.19% / break 100.00%).
    • The authors state that the hallucinations produced by interventions (e.g., syntactically invalid selector strings) differ in nature from the semantically coherent natural hallucinations under zero-shot conditions (Appendix E).

    Input position and counterfactuals

    • Activation patching (Table 2): the action list is the main causal contributor (L27 recovery 0.360; selector start and task description are both below 0.02), but the authors state that they cannot determine which properties of the action list drive the commitment, and that the span requiring corruption is unstable across samples.
    • After corrupting the action list, on half of the valid steps where the selected value had previously appeared in the action history, the model recovers that value from history (Figure 3: 71% unaffected in the Action history group, 21% unaffected in the Action list only group, 40% unaffected in the Page text group).
    • Counterfactual construction (Appendix H, Table 4): deleting the selected valid action produces hallucination only 38% of the time; from step 1 onward, the selector agreement rate between the two trajectories drops to about 25% and the DOM Jaccard similarity drops to 0.52, and the authors state that the premise for paired trials does not hold.

    SAE and logit lens

    • SAE (Figure 7): among the top-10 features on L27 ranked by inter-class activation difference, most of the tokens they boost are unrelated to the observed hallucination behavior; the authors state that an SAE pretrained on a text corpus struggles to explain features that arise in agent interaction.
    • Logit lens (Appendix C, Figure 8): hallucinated samples at L21–27 converge to semantically coherent labels that do not exist on the page, such as Search, Enter, Find, Explore, and the model often first produces a Chinese word at L24 before converging to an English word; the authors argue that the model commits to a strategy that is not grounded in the specific website.

    Other ablations and analyses

    • Steering full configurations (Table 5): the prompt_end and selector_value_start positions are markedly weaker, e.g., prompt_end L18 α=40 is 41.5/10.8, and selector_value_start L21 α=40 is 17.1/29.5.
    • α sweep (Table 5): selector_start L17 at α=60 has fix dropping to 48.8% and break rising to 30.2%; at α=80, fix is 0.0% and break is 96.9%.
    • Probe cross-format (Figure 6): after switching to a numbered-list format, probe AUC decreases; the main text states that steering effectiveness decreases accordingly.
    • Negative results: the pretrained SAE did not produce interpretable features; single-component interventions could not reproduce the natural hallucination pattern.
  5. What are some points that can be further explored?

    The authors propose five roadmap directions; the experiments cover only a single model, a single benchmark, and one failure mode.

    Limitations and future directions noted by the authors

    • Agent MI benchmarks (Section 5): There is currently no MI benchmark for agent applications; the authors argue that standards are needed that simultaneously specify the environment, the trajectory distribution, the ground truth for trajectory-level failures, and correct mechanistic explanations.
    • Beyond single-position/single-component attribution (Section 5): Attribution methods capable of attributing failures that span multiple time steps and environment state distributions remain scarce.
    • Counterfactual construction in stateful environments (Section 5): Existing methods (Tsirtsis et al., 2021; Triantafyllou et al., 2025) assume a known state space and comparable trajectories, do not apply to open-ended environments, and new methods are needed.
    • Feature discovery for agent distributions (Section 5): There is a need to train SAEs on agent trajectories and to develop geometric structures beyond linearity and agent-oriented feature ontologies.
    • Monitoring model beliefs across time steps (Section 5): Methods are needed that can track plan formation across layers and time steps and identify the turning point at which a model shifts from open exploration to a fixed plan.
    • Universality across agent types (Section 5): Whether circuits found in one type of agent appear in other types (e.g., embodied, code generation) remains an open question.

    Scope of experimental coverage

    • Only one model was tested: Qwen2.5-7B-Instruct (Section 4.1).
    • Only one benchmark was used, Online-Mind2Web, with the action space limited to CLICK, SCROLL, TYPE, HOVER; the only failure mode was webpage element hallucination (Section 4.1).
    • The data scale was 300 episodes and 2,071 actions; the probe test set was 356 actions (Section 4.3).
    • The SAE analysis used a third-party pretrained checkpoint and did not train an SAE on agent trajectories (Appendix K).
    • The paper does not report the number of experimental repetitions, confidence intervals, or statistical significance tests; Appendix A discusses generalization to embodied, tool-calling, and code-generation agents, but no experiments were conducted.
  6. Summarize the main content of the paper

    MI tools can detect and intervene on agent hallucination, but cannot fully explain its mechanism; the authors provide a roadmap.

    An analytical paper: it examines the explanatory power of standard MI tools on real agent failures, using the case of a web agent hallucinating nonexistent page elements.

    • Problem: Agent failures span long trajectories and multiple components; the authors state that existing successful MI explanations mostly come from narrow tasks with fixed inputs, and the mechanisms of real agent failures are largely unknown.
    • Method: Run Qwen2.5-7B-Instruct on live websites in Online-Mind2Web, collect 300 episodes (2,071 actions, 32.6% hallucinations), and apply component interventions, activation patching, linear probes, steering and pretrained SAE analysis.
    • Detection is feasible: The linear probe at the prompt_end position L18 predicts hallucination with 0.80 AUC before generating any token (Figure 4); the authors believe the hallucination commitment is encoded before generation.
    • Intervention is partially feasible: Probe-direction steering at selector_start L17 (α=40) fixes 85.4% of hallucinations and causes false positives on only 7.0% of valid actions (Table 5).
    • Explanation is incomplete: The best joint component intervention fixes only 8.9% (Table 1); the action list is the main causal input but the specific driving attributes are unclear (Table 2); after minor changes to the action list format, both probes and steering degrade (Table 6, Figure 6); the pretrained SAE does not yield interpretable features (Figure 7).
    • Authors' conclusion: These tools can detect and steer agents away from failures, but cannot explain how decisions arise; the authors argue that methods specifically designed for agent characteristics are needed, and propose six directions: benchmarks, distributed attribution, counterfactual construction, agent distribution feature discovery, belief monitoring and cross-agent universality.
Read originalarxiv.org