Same simulation, different evidence: where retrieval-style tool output causes LLM agents to lose accuracy

Core question: when an agent framework changes long tool outputs to file-retrieval delivery, at which step does the accuracy loss occur—computing wrong, not seeing it, or seeing it and still answering wrong.

• Setting and importance: LLM agents increasingly call tools with very long outputs (logs, search results, simulator reports), and frameworks must choose whether to paste the output into context, truncate it inline, or store it as a file for search or line-range reading; the authors state that file-based delivery can save context, but Sen et al. (2026) found that it lowers accuracy in most harness–model pairs and did not identify the cause.
• Shortcomings of existing methods: the authors state that existing work only observed final accuracy drops and could not distinguish whether the agent computed the value wrong, never saw the correct value, or saw it and still answered wrong; models also do not always use context containing all the required information.
• Core observation: a deterministic simulator makes every printed value traceable. Using the USGS PHREEQC geochemical simulator (reports up to 846.8k characters), the authors decompose accuracy into three stages: build (whether the simulation computed the keyed value), exposure (whether that value reached the agent), and closure (whether it answered correctly after seeing it).
• What the paper proposes: a failure-localization method on 158 execution-verified multiple-choice questions—rerun every input the agent wrote, search every tool result it received, and determine the three stages per trajectory; and based on this, test interventions (specifying the authoritative output, inlining reports that fit, rounding instructions). The authors state that the loss lies in what the agent reads rather than what it computes, and propose the failure mode of "evidence substitution."

Tool output delivery

• Sen et al. (2026)—found that file-based delivery of search results lowered accuracy in nine of ten harness–model pairs (including frontier models), and hypothesized the cause was a read–integrate–retry loop; the paper calls it the closest work, and this paper builds on it to locate the loss stage and test it with interventions.
• Rajasekaran et al. (2025), Abramovich et al. (2025)—context engineering practices that write long outputs to files for agents to open or search; Lindenbauer et al. (2025), Kang et al. (2025) handle long observations with truncation, masking, or compression.
• Yang et al. (2024) (SWE-agent)—the agent–computer interface (including file viewing and search methods) changes software engineering agent performance; Kate et al. (2025) found that long responses degrade function-calling performance.

Sufficient information and available information

• Xu et al. (2020)—proposed a theory of "available information" for computationally bounded observers, distinct from Shannon information.
• Xie et al. (2026)—states that messages retaining all decision-relevant information may still fail for bounded receivers, and decomposes the loss into externalization, absorption, and action closure; the paper states that its exposure and closure stages correspond to their absorption and closure.
• Joren et al. (2025), Chen et al. (2025)—the former states that smaller models often cannot exploit sufficient retrieved context; the latter states that search agents answer far better from gold documents than from searching on their own.

Agent failure localization

• Zhang et al. (2025), Zhang et al. (2026a)—attribute agent failures using human or model review, or counterfactual replay to label data for a learned attributor.
• Ru et al. (2024) (RAGChecker)—distinguishes retriever and generator errors against reference claims.
• Tan et al. (2024), Yang et al. (2025)—the former states that models prefer self-generated context over retrieved context, the latter states that models are unreliable at rounding to significant figures; the paper states that this makes the failure mode found in this paper plausible, and this paper shows that interface choice triggers it.

Scientific software agents

• Yue et al. (2026), Chen et al. (2024) (OpenFOAM), Mudur et al. (2025) (COMSOL), Campbell et al. (2025) (molecular dynamics), Bogin et al. (2024) (SUPER)—typically scored on final results on a single software package; the paper states that this paper re-executes agent inputs with a deterministic simulator, giving ground-truth answers for each stage without review.

Baselines and benchmarks: the no-tool baselines are Direct and chain-of-thought (CoT) prompting; questions come from Zhang et al. (2026b)'s PHREEQC-MCQ-200, and this paper retains 158 questions after re-execution verification of the answer key. The authors position this paper's distinction as: verifying the answer key, finding that cost depends on the model, and localizing the loss within a single trajectory and testing that localization with interventions.

Overall approach: use the deterministic simulator PHREEQC to make every printed value traceable, change only how the report reaches the agent (inline or directory retrieval), attribute the accuracy change to three failure stages, then intervene on the localized stage.

Question and answer key verification

• The questions are four-option multiple-choice questions written by a domain expert from Zhang et al. (2026b); each question describes a PHREEQC simulation setup and asks about one quantity in the output; distractors for numerical questions usually differ from the answer only in the last printed digit.
• The authors re-executed all 200 questions with PHREEQC 3.8.6 and a fixed phreeqc.dat: the 9 setups with the authors' original inputs were run directly, and the rest were encoded literally from the question text, never adjusting inputs toward the answer key.
• A question was retained only when the printed value matched the answer key exactly and uniquely and was unchanged under stricter convergence or step settings; 158 questions (from 19 setups) qualified, and 42 were excluded and confirmed as flawed by the question authors (Appendix A, Table 6).

Agents and output delivery conditions

• The agent has four tools: write_file, read_file (optional line range), list_file, execute_phreeqc; up to 24 steps in an isolated workspace, submitting the answer by writing a file.
• The PHREEQC report (result.out) prints each quantity at the precision used in the options; the agent can add SELECTED_OUTPUT or USER_PUNCH blocks to write selected quantities with more digits to a separate file.
• Delivery conditions: Raw100k inlines the report (truncated at head and tail to 100,000 characters); TOC returns metadata and a section index of the report, and the agent reads by line range; Grep returns only metadata, and a regex search tool is used to find evidence; Hybrid inlines when the report is no more than 100,000 characters and otherwise gives a TOC; TOC+Report and TOC+Round add three-sentence instructions on top of TOC (Appendix B gives the verbatim prompts).

Failure localization: three-stage decomposition

• For 5,773 complete logged trajectories, the authors reconstructed every input the agent executed and reran them offline with the same binary and database, totaling 11,007 simulation calls; of 5,009 comparable reruns, 4,971 reproduced the stored report, and the rest differed only in line order.
• For the 137 numerical questions, three questions are asked: build (p)—whether the agent's own simulation printed the keyed value; exposure (s)—whether that value reached the agent; closure (c)—whether it answered correctly after seeing it.
• Accuracy is exactly decomposed into p s c plus the correct-answer rate when there is no keyed value (unsupported); a Shapley decomposition assigns each condition's accuracy change relative to the same model's inline run to four factors, with 95% item-bootstrap intervals; the authors note that this decomposition assigns the observed differences and does not estimate independent causal effects.
• Matching is numerical: the number printed at any position counts as the keyed value; automatic and manual stage labels agreed on 52 of 58 manually read numerical trajectories, and results hold on the 93 questions where the keyed value appears only once (Appendix C).

Intervention design

• TOC+Report: three-sentence instructions added to the TOC tool description and to every tool result, stating that values in result.out are authoritative and printed at option precision, and that values should be taken from the report and not answered from the selected-output file.
• TOC+Round: replaced with three-sentence instructions that only require correctly rounding values with more digits than the options, without truncation, naming no file.
• Hybrid: no instructions; the report is inlined if it fits within 100,000 characters.
• Paired replay (Appendix D): freeze the checkpoint of the first successful execution on 131 numerical questions from GPT-5.2's logged inline run, and continue from the same visible history with either the inline report or the TOC index; both arms share files, system prompt, and remaining turns, and cannot run the simulator again.

Experimental setup

• Models tested: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, GPT-5.2, GPT-5.1, Gemini 3 Flash Preview, all called via API; Claude used temperature 0 with no extended thinking, GPT and Gemini used API defaults with no reasoning-effort setting.
• Benchmark: 158 execution-verified four-choice questions (137 numerical questions) from PHREEQC-MCQ-200, from 19 setups.
• Baselines: tool-free Direct and CoT prompting; the main comparison is the per-question accuracy difference of each model's TOC relative to inline (Raw100k).
• Metrics and statistics: accuracy; TOC − inline difference in percentage points, 95% item-bootstrap intervals, sign-flip permutation test with Holm correction across the five models; three-stage rates p/s/c and Shapley attribution.
• Number of runs: one headline run each for Raw100k and TOC per model; TOC rerun three times for Sonnet and the GPT models, Raw100k rerun once for the GPT models; for the 33 questions whose reports exceed 100k characters, the four TOC runs share one fixed index run; Table 8 lists all runs.
• Handling of Gemini 3 Flash: 46% of questions (71% under TOC) produced no parseable answer (Appendix E), and it was excluded from the main analysis.

Main results

According to Table 1 (158 questions, accuracy from the headline run; TOC − inline aggregated over all runs per question):

Swipe horizontally to read the table

Model	Direct	CoT	Agent-TOC	Agent-Raw100k	TOC − inline (pp)	pHolm
Claude Opus 4.6	43.0%	39.9%	87.3%	88.0%	−0.6 [−3.8, +2.5]	1.000
Claude Sonnet 4.6	38.0%	46.2%	91.8%	91.1%	−0.5 [−3.8, +3.2]	1.000
GPT-5.4	25.9%	34.2%	75.3%	70.9%	+0.2 [−5.5, +5.7]	1.000
GPT-5.2	27.2%	32.9%	57.0%	65.8%	−7.9 [−14.9, −0.9]	0.128
GPT-5.1	41.1%	36.1%	57.0%	67.7%	−11.3 [−18.0, −4.9]	0.005
Gemini 3 Flash Preview	41.1%	40.5%	27.8%	50.0%	–	–

• The authors state that under Direct, accuracy was 25.9%–43.0%, at most about 10 percentage points above the 32.9% majority-class baseline; simulator access brought a 15.8–45.6 percentage point gain for the five Claude and GPT models.
• After Holm correction, the difference for GPT-5.1 is significant (p = 0.005), and for GPT-5.2 it is not significant (p = 0.128, uncorrected 0.032); resampling by the 19 setups gives intervals of [−19.1, −2.9] (GPT-5.1) and [−16.2, +1.0] (GPT-5.2).
• The authors state that GPT-5.4's net result masks offsetting effects (see below), and that the divergence in net accuracy does not split by vendor.

Locating failures: losses in exposure and closure

• What stays the same: the proportion (p) of GPT-5.2 and GPT-5.1 producing keyed values is similar across interfaces, and build-term changes are within 2 percentage points with intervals containing zero (Table 3).
• What changes in reading: under TOC, GPT-5.2 and GPT-5.1 received no part of the report in 60.3% and 53.0% of trajectories respectively, mostly reading self-built selected-output files instead (Table 4); the authors call this evidence substitution. Sonnet skipped the report in only 10.5%.
• Attribution numbers: for GPT-5.2 under TOC, of the total change of −11.4 pp on numerical questions, closure accounts for −7.7 pp (interval excludes zero) and exposure for −2.5 pp; GPT-5.1's −11.9 pp is roughly split between exposure (−4.3) and closure (−4.4) (Tables 3, 9). Grep shows the same pattern, and the authors state the cost is not specific to TOC indexing.
• Truncation trap: when seeing rounding yields the answer and truncation yields a neighboring distractor value, GPT-5.2 chose the truncated neighbor in 26/35 TOC trajectories and GPT-5.1 in 19/27, versus Sonnet at 1/35; when the keyed value was seen at reported precision, the answer was almost always correct, whereas when seen only with more digits, accuracy was only 72.5% (GPT-5.2) and 66.3% (GPT-5.1) (Table 4). The authors note this comparison is observational.

Paired replay and GPT-5.4's net-zero illusion

• Fixed simulation still loses points: freezing GPT-5.2's first successful execution on 131 numerical questions, TOC changes accuracy by −9.4 pp [−14.5, −4.6] across three paired repetitions; on the 115 reports that fit inline it is −10.1 pp, and on the 84 questions where the keyed value exists it is −9.9 pp (Table 10, Appendix D).
• GPT-5.4's offset: GPT-5.4 likewise skips the report (41.8% of TOC trajectories) and chooses the truncated neighbor in about half of the trap cases; TOC costs it about ten percentage points in exposure and closure (−2.6 and −7.2 pp), but the build gain is +4.1 [+0.0, +8.1] pp, the net change is −4.4 [−12.4, +3.4] pp, and aggregated over all runs it is +0.2 pp, all indistinguishable from zero (Tables 3, 4).

Intervention experiments

• TOC+Report: report skip rate drops to 0.0% (GPT-5.2) and 1.6% (GPT-5.1), closure rises to 98.8% and 99.2%, no trajectory falls into the truncation trap, and build is unchanged; accuracy rises +7.9 and +6.3 pp relative to all TOC runs, and +3.5 and +7.9 pp relative to the TOC control; GPT-5.4 reaches 81.6%, +7.8 [+2.2, +13.6] pp over its TOC runs (Table 5).
• TOC+Round ineffective: skip rate is comparable to TOC, closure stays at about 82%, and 17/25 and 14/19 trap cases respectively still choose the truncated neighbor; accuracy is only +0.6 and +1.2 pp relative to all TOC runs, and TOC+Report exceeds TOC+Round by +7.3 [+0.6, +13.9] and +5.1 [−1.6, +12.0] pp (Table 5).
• Hybrid: +4.6 (GPT-5.2) and +8.0 pp (GPT-5.1) relative to the TOC control, with closure back to inline levels; relative to inline, GPT-5.2 is +1.2 pp and GPT-5.1 is −5.0 pp, and the authors state GPT-5.1's residual lies in build, exposure and unsupported terms rather than closure (Table 5).

Other ablations and analyses

• Question sets: model rankings and the TOC/Raw100k pattern are the same on the full 200-question set, the 158 questions, and the stricter 134 questions (Table 7).
• Input tokens: GPT-5.2's Hybrid is 59.7k per question, TOC 61.8k, inline 65.7k; GPT-5.1's Hybrid is 28.3k, but the authors note this is mainly because that run rarely read one optional reference file, and do not claim this as a saving of Hybrid (Table 2, Appendix B).
• Manual audit: of 24 TOC losses among 68 trajectories, 7 were caused by the retrieval channel, 2 were errors while holding the value, and 15 were build failures; the 8 Sonnet failures were all build failures (Appendix C).
• No-value answers: GPT-5.1 declined to answer in 6.1% of TOC trajectories and 1.9% of inline trajectories; under inline it tended to choose the nearest option, answering about one third correctly (Appendix C).

Limitations and future directions noted by the authors

• Models: all models are closed-source and accessed via API; evidence substitution appeared in all three GPT models and rarely in the Claude model Sonnet 4.6, for which complete trajectory logs exist; the authors did not determine which model property drives this difference (Section 9).
• Decoding and prompting: Claude agents ran at temperature 0 with no extended thinking, GPT and Gemini used API defaults with no reasoning-effort setting, and the authors do not know whether the effects persist under explicit reasoning; only one wording was tested per instruction (Section 9).
• Run coverage: due to API budget, not every model was run, repeated, or logged under every condition, and some table cells are empty; complete agent conditions were run at different times, and the authors did not model possible changes in the served models between runs; the four TOC runs shared one run on 33 long-output questions, the decomposition used one logged inline run per model, and paired replay covers only GPT-5.2 and starts from checkpoints of its inline run (Section 9).
• Nature of verification: the answer keys for retained questions were reproduced by PHREEQC under the authors' inputs or the literal encoding of the question text, not an independent scientific review of each question; the question text for three setups (24 retained questions) did not fully specify the reference input, and excluding them leaves 134 questions with the same conclusions (Section 9, Appendix A).

Experimental coverage

• The models tested are Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, GPT-5.2, GPT-5.1, and Gemini 3 Flash Preview, 6 in total, all closed-source API models; Gemini 3 Flash was excluded from the main analysis because 46% (71% under TOC) of its questions had no parseable answer (Section 2.2, 3).
• The evaluation environment is a single simulator, PHREEQC 3.8.6, and 158 verified questions (137 numerical questions, 19 setups); delivery conditions include Raw100k, TOC, Grep, Hybrid, TOC+Report, and TOC+Round (Section 2).
• Interventions and paired replay cover only some models: TOC+Report and Hybrid were run on GPT-5.2, GPT-5.1, and GPT-5.4, TOC+Round was run only on GPT-5.2 and GPT-5.1, and paired replay was run only on GPT-5.2 (Table 5, Appendix D).
• Failure localization is based on 5,773 complete logged trajectories and 11,007 rerun calls; 68 trajectories were manually audited, and automatic and manual labels agreed on 52 of 58 numerical trajectories, with the authors noting that this sample was selected by outcome and the agreement rate is indicative (Appendix C).
• The paper does not report a statistical power analysis beyond the number of repetitions for each intervention condition, nor does it report the effect of the TOC+Report instruction on other tools or domains.

An analysis paper on agent context management: where accuracy is lost and how to fix it when a framework switches long tool outputs from inline to file retrieval.

• Problem: to save context, frameworks store long tool outputs as files for retrieval; existing work has only observed a drop in final accuracy and cannot distinguish computing it wrong, not seeing it, or seeing it and still answering wrong.
• Method: using the deterministic simulator PHREEQC (reports up to 846.8k characters) and 158 execution-verified multiple-choice questions, changing only how the report is delivered; rerunning all 11,007 simulator calls, decomposing accuracy into three stages—build, exposure, closure—and then testing the localization with instruction interventions.
• Main results: TOC retrieval lowers GPT-5.1 and GPT-5.2 accuracy by 11.3 and 7.9 percentage points respectively (Table 1); the net-change intervals for Claude Opus 4.6, Sonnet 4.6 and GPT-5.4 contain zero.
• Mechanism: the frequency with which the GPT models compute the keyed value is unchanged, but under TOC GPT-5.2 in 60.3% and GPT-5.1 in 53.0% of trajectories did not receive the report, instead taking values truncated from self-built high-precision files (evidence substitution); after fixing the simulation, GPT-5.2 still loses 9.4 percentage points.
• Intervention: an instruction specifying result.out as the authoritative output drops the report skip rate below 2%, raises closure to 99%, and raises accuracy by 6.3–7.9 percentage points for the three GPT models; reports that fit inline have the same effect; an instruction only requiring correct rounding is ineffective. GPT-5.4's net-zero change masks about ten percentage points of evidence-use loss, offset by a higher build success rate.
• Authors' conclusion: output delivery should be validated model by model, and judged by where the loss occurs rather than only by final accuracy, because a near-zero net effect may mask evidence-use loss; outputs that fit inline are a safe default, and when an agent can build its own outputs, it should be told which output is authoritative.