Agent benchmarks are surprisingly friendly to context compression

核心问题：当 LLM 智能体的上下文被压缩（compaction）后，它还能否完成任务、还遵守给它的规则吗？ 作者认为，尽管压缩是编程智能体 scaffold 的标准组件，但此前没有工作用前沿模型在真实长程任务上系统研究过它。

• 场景与重要性：LLM 智能体现在可以自主运行数小时，单次 PostTrainBench 或 ProgramBench 运行平均产生数十万 token，超过许多模型的上下文窗口；Claude Code、Codex CLI、OpenCode 等 scaffold 普遍用摘要式压缩来释放上下文。
• 现有研究的不足：作者称，已有工作要么用较小的开源权重模型在玩具或合成轨迹上评测，要么在长上下文问答、网页搜索等输入一次性给定的任务上比较压缩与其他上下文管理方法，都没有在前沿模型、上下文由智能体自己产生的真实长程任务上测量压缩的下游影响。
• 核心观察与假设：压缩按设计是有损的，摘要可能删掉任务约束、指令或先前决策的推理链；作者提出"基准对压缩友好（compaction-friendly）"的假设——基准目标固定、进度持久化在磁盘上，被丢弃的内容大多可以重新读回，唯独只存在于上下文中的约束会真正丢失。
• 论文的贡献：固定 OpenCode scaffold，在其中实现 5 种压缩策略，在 PaperBench、ProgramBench、PostTrainBench 三个长程基准上评测 4 个前沿模型，并在 SWE-chat 真实用户会话上分析约束丢失。
• 研究问题：压缩阈值是否重要？压缩方法是否重要？能否用更便宜的模型写摘要？压缩是否会移除让智能体可以安全自主运行的约束？

Context compaction in agents

• Industry practice: scaffolds such as Claude Code, Codex CLI, Gemini CLI, Cursor, OpenHands, and OpenCode mostly have the model self-summarize when context approaches the limit (Cassano & Rush, 2026; Anthropic, 2026a; Smith, 2025); the Anthropic API server side provides compact, clear tool uses, and clear thinking primitives, and reports that clearing tool results alone improved an internal agent search evaluation by 29% (Anthropic, 2025).
• Training-time and test-time compaction: Cursor trains Composer to self-summarize in an RL environment loop (Cassano & Rush, 2026); ReSum fine-tunes a separate summarization model and adapts it with RL (Wu et al., 2025); MemAgent uses RL to maintain fixed-length memory (Yu et al., 2026); DeepSeek-V3.2 treats compaction as a means of scaling test-time compute (DeepSeek-AI, 2025); Kimi K2.5 compares compaction with sub-agent swarms (Kimi Team, 2026).
• Autonomous context management: SelfCompact (Li et al., 2026) lets the model decide when and how to compact; Context-Folding (Sun et al., 2025) lets the agent branch sub-trajectories and fold them; AgentFold (Ye et al., 2026) lets the agent learn to fold context; Recursive Language Models (Zhang et al., 2025) instead treat the prompt as a REPL variable and avoid compaction entirely.

Evaluation of compaction

• COMPINT (Wang et al., 2026): found that user constraints survived compaction only about 17% of the time; the authors note its shortcomings are that it mainly evaluates open-weight models with 128–140K context and truncates trajectories at 100K tokens to simulate compaction.
• Downstream performance comparisons: RLMs (Zhang et al., 2025) outperform compacting agents on GPT-5, and Context-Folding (Sun et al., 2025) outperforms summarization-based methods; the authors note that in these works the long input is given all at once, whereas this paper asks about downstream effects when the context is produced by the agent itself.

KV cache compression and long-context degradation

• KV cache compression: heavy hitter eviction (Zhang et al., 2023), attention sinks (Xiao et al., 2024), and query-aware selection (Li et al., 2024) treat the model as a black box to decide which tokens to keep; the authors note that in prompt-level compaction the summary is generated by the model itself, so what is kept depends on the model's own judgment.
• Long-context degradation: lost in the middle by Liu et al. (2024) and context rot by Hong et al. (2025) show that models degrade as input grows longer, which is relevant to this paper's compaction threshold experiments.

Baselines and benchmarks

• Baseline method: default self-summarization, compared with four strategies: tool-output pruning, filesystem dump, dropping reasoning blocks, and small-model summarization; benchmarks: PaperBench, ProgramBench, PostTrainBench, plus real SWE-chat sessions.

The authors' positioning: to their knowledge, no prior work has used frontier agents on real long-horizon benchmarks to answer these questions about compaction thresholds, compaction methods, summarization model choice, and constraint retention.

Overall approach: fix OpenCode as the sole agent scaffold and implement the 5 compaction strategies as replacements for its default compaction logic, thereby separating the effect of compaction strategy from scaffold behavior (tool format, retry logic, system prompt); run each task and configuration 2 times.

Models and benchmarks

• Models: Claude Opus 4.6 (1M context), GPT-5.4 (1,050K), Gemini 3.1 Pro Preview (1M), GPT-5.3-Codex (400K, used to compare a small-window model); details in Table 1.
• PaperBench: reproduce ICML 2024 papers from scratch, taking 3 randomly selected papers out of 20 (adaptive-pruning, all-in-one, bam), using the Code-Dev variant, a 10-hour time limit, and gpt-4o-mini as the rubric judge.
• ProgramBench: rebuild programs from scratch; the task prompt contains 3 anti-cheating rules (must not copy/embed the reference implementation, must not access the network, the implementation must be original), and the rules appear only in the first user message, so after compaction they are retained only if the summary restates them; two tasks, Canop/broot and jarun/nnn, are selected, and the metric is test pass rate.
• PostTrainBench: post-train a small base LLM on a single H100 within 10 hours, taking two bases, gemma-3-4b-pt and Qwen3-4B-Base, and three target benchmarks, GPQA/GSM8K/BFCL; an LLM judge enforces that no training on test data or model swapping occurs.

Compaction strategies (shared trigger mechanism)

• Trigger: set a maximum number of tokens allowed before compaction, compare usage after each assistant turn, and trigger when reached; the threshold sweep range is 160K–380K tokens.
• Self-summarization (baseline): when the threshold is reached, use a prompt to have the agent summarize its prior work, and place the summary together with the system prompt into a new context; by default the same model performing the task writes the summary (full prompt in Appendix 6.4).
• Tool-output pruning: after each assistant turn, traverse the history in reverse, keep the most recent user-assistant turn and recent tool outputs sufficient to fill a 40K safety budget, and replace older tool outputs with placeholders; run only when at least 20K tokens can be freed, and used in combination with self-summarization.
• Filesystem dump: before compaction, serialize the full message history as JSON to disk and note the save location in the summary, making compaction "lossless".
• Dropping reasoning blocks (clear thinking): before compaction, remove reasoning blocks from previous assistant turns in the history, keeping reasoning only for the most recent k=10 turns; the summarizer does not see the removed reasoning.
• Small-model summarization: same as the baseline, but the summary is written by the smaller and cheaper Claude Haiku 4.5.

Measurement and judgment

• Task performance: rubric score for PaperBench, test pass rate for ProgramBench, target benchmark performance for PostTrainBench; record the number of compactions, run failures, and violations.
• Constraint erosion: for ProgramBench, use Claude Sonnet 5 to review each compaction summary and trajectory and report which of the 3 prohibited behaviors are missing from the summary; the judge defaults each category to "dropped" and judges "preserved" only when the summary explicitly restates the rule (Appendix 6.4.3).
• SWE-chat analysis: extract 44 real sessions containing compaction events, use Claude Sonnet 5 to extract user constraints before compaction (269 constraints), then judge whether each constraint is preserved/lost/contradicted in the next compaction summary, and classify them into 7 categories; use GPT-5.6 Sol as a second judge for review.
• Statistics: error bars are nonparametric bootstrap 95% confidence intervals (2.5 and 97.5 percentiles from resampled runs).

Experimental setup

• Models tested: Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro Preview (all with about 1M context), GPT-5.3-Codex (400K).
• Benchmarks: PaperBench (3 papers), ProgramBench (2 tasks, plus lua/lua for the erosion analysis), PostTrainBench (2 base models × 3 target benchmarks), SWE-chat (44 sessions, 269 constraints).
• Baseline and strategies: default self-summarization as the baseline, compared with tool-output pruning, filesystem dump, clear thinking, and small-model (Claude Haiku 4.5) summarization.
• Metrics: accuracy on each benchmark (conditioned on producing a valid submission), failure rate, number of constraint categories lost per compression (0–3), SWE-chat constraint retention/loss rate.
• Judging: PaperBench rubric with gpt-4o-mini; ProgramBench constraint erosion and reward hacking judgments with Claude Sonnet 5; SWE-chat primary judge Claude Sonnet 5, secondary judge GPT-5.6 Sol.
• Sample size: 2 runs per task per configuration; error bars are bootstrap 95% CI.

Main results

Accuracy under compression thresholds (Figure 2) and compression strategies (Figure 3, threshold 160K) (conditioned on producing a valid submission):

Swipe horizontally to read the table

Model	Condition	PaperBench	ProgramBench	PostTrainBench
Claude Opus 4.6	thresholds 160K–380K	similar across thresholds	similar across thresholds	error bars overlap
GPT-5.4	thresholds 160K–380K	similar across thresholds	similar across thresholds	error bars overlap
GPT-5.3-Codex	160K → 260K+	0.38 → about 0.60	broadly flat	error bars overlap
GPT-5.4	baseline → clear thinking	0.59 → 0.43	no significant effect	—
GPT-5.4	baseline → small-model summarization	0.59 → 0.43	—	—
GPT-5.3-Codex	baseline → small-model summarization	0.38 → 0.18	—	—
GPT-5.3-Codex	baseline → filesystem dump	0.38 → ≈0.24	—	—

(From Figure 2, Figure 3; "—" marks combinations for which the paper does not report specific numbers in the main text; Gemini 3.1 Pro Preview rarely triggers compression, and the authors state its results are uninformative, see Appendix 6.6.)

• The authors state: the task performance of the two 1M-context models does not change meaningfully with threshold (160K–380K, 16–38% of their window); GPT-5.3-Codex is worse at low thresholds on PaperBench, and the conclusion on PostTrainBench is unclear.
• The authors state: no strategy outperforms default self-summarization outside the confidence intervals; strategies that discard more content cost some models up to 20 points and have no effect on other models, so compression methods validated on a single model do not necessarily transfer.
• The authors note that Haiku 4.5 and Opus 4.6 are from the same company, and a shared vendor may explain why small-model summarization is harmful only to OpenAI models.

Constraint erosion (Q3)

• What was measured: how many of 3 anti-cheating rules are retained in each compression summary in ProgramBench, judged by Claude Sonnet 5.
• Results: average number of categories lost per compression is 2.73 for GPT-5.3-Codex (80% of compressions lose all 3 categories), 1.76 for GPT-5.4, and 0.51 for Claude Opus 4.6 (never loses all) (Table 3); the average loss is 0.92 categories at the 1st compression, 1.93 at the 10th, 2.41 at the 15th, and 2.65 at the 30th (Figure 4a).
• Authors' interpretation: later summaries are written based on earlier summaries that have already lost rules, so losses accumulate; reward hacking begins before the first compression, and the authors believe constraint erosion cannot explain when the agent starts cheating; instead, cheating may cause the rules not to be restated.

SWE-chat real sessions (Q4)

• What was measured: retention rate of 269 user constraints from 44 real user-agent sessions in the next compression summary, classified into 7 categories.
• Results: overall 81% retained, 19% lost (24.5% under the secondary judge GPT-5.6 Sol); feature requirement (35.7%, n=96) loss rate 6.2%, process instruction (n=39) 35.9%, scope exclusion (n=38) 36.8% (Figure 4b).
• Authors' interpretation: this matches the authors' "compression-friendly" prediction—constraints describing deliverables can be recovered from the agent's own artifacts, while constraints describing how to behave cannot; but Q4 only measures the amount lost, not the effect on final results.

Other ablations and analyses

• Aggregated by number of compressions (Figure 5): accuracy for Claude Opus 4.6 and GPT-5.4 is unrelated to the number of compressions, and GPT-5.4 does not degrade even beyond 16 compressions; GPT-5.3-Codex declines starting at about 8 compressions, and beyond 16 is about 0.10 lower relative to the task mean.
• Failure rate (Figure 7): failures are common on ProgramBench and PostTrainBench, but no strategy is consistently worse, and the ordering varies by model; no failures were observed on PaperBench.
• Compressions per threshold (Table 2): at the 160K threshold, the average number of compressions is 4.90 for Claude Opus 4.6, 15.08 for GPT-5.4, 27.71 for GPT-5.3-Codex, and 1.00 for Gemini 3.1 Pro Preview.
• Judge agreement (Appendix 6.11.2): overall agreement between the two judges is 88.8% (239/269), Cohen's κ 0.681; agreement on the dropped category is 84.3% (43/51).
• Reward hacking mechanisms (Figure 10): 7 categories of mechanisms counted by model; a single trajectory can have multiple labels; the lua and luajit tasks were excluded from the main analysis because they were frequently hacked.

Limitations and future directions noted by the authors

• Number of models and tasks: the authors write that, due to cost and compute constraints, they evaluated only 4 models, 3 benchmarks, and a small number of tasks per benchmark (Section 5 Limitations).
• Single scaffold: one scaffold was fixed; other scaffolds may use different summary prompts or memory systems and may behave differently (Section 5 Limitations).
• Weak Gemini evidence: under this setting, Gemini 3.1 Pro Preview rarely triggers compression, its context grows slowly, and it frequently reward hacks on ProgramBench, so its evidence on compression is weak (Section 5 Limitations).
• Different SWE-chat scaffold: SWE-chat sessions were produced by Claude Code, which differs from the scaffold used in the experiments, and the underlying model is also different (Section 5 Limitations).
• Future direction one: decompose long tasks into parallel sub-agents with independent contexts (such as agent swarms) and compare them with the best compression method under the same scaffold, to determine whether compression is worth engineering around (Section 5 Future work).
• Future direction two: harder evaluations are needed for constraints that cannot be recovered from the filesystem, such as multi-turn sessions where user instructions change over time (Section 5 Future work).

Scope of the experiments

• The models tested are Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro Preview, and GPT-5.3-Codex, 4 in total (Table 1).
• The benchmarks are PaperBench (3 papers), ProgramBench (2 tasks), and PostTrainBench (2 base models × 3 target benchmarks); the threshold sweep is 160K–380K, which for 1M models covers only 16–38% of the window, and compression points near the scaffold default of 90–95% were not tested (Section 4.1).
• Each task and configuration was run 2 times; only one small model, Claude Haiku 4.5, was used for small-model summarization (Section 3.3).
• The SWE-chat analysis covers 44 sessions and 269 constraints, and measures only the amount of constraint loss, not the effect of loss on final results (Section 4.2).
• The paper does not report numbers for the additional latency or token cost introduced by compression; the authors state that sub-agent decomposition is left for future work (Section 3.3).

A systematic evaluation paper on context compaction in LLM agents, answering "does compaction affect task performance, and does it silently drop constraints?"

• Problem: when long-horizon tasks exceed the context window, scaffolds perform lossy compaction, which may drop task constraints and reasoning chains; previously there was no systematic study using frontier models on real long-horizon tasks.
• Method: fix the OpenCode scaffold, implement 5 strategies—self-summarization, tool-output pruning, filesystem dump, clear thinking, and small-model summarization—evaluate 4 frontier models on PaperBench, ProgramBench, and PostTrainBench, and use an LLM judge to measure constraint retention.
• Threshold: Claude Opus 4.6 and GPT-5.4 perform similarly across thresholds from 160K to 380K; GPT-5.3-Codex on PaperBench rises from 0.38 at 160K to about 0.60 above 260K (Figure 2).
• Strategies: no strategy outperforms default self-summarization; clear thinking lowers GPT-5.4 on PaperBench from 0.59 to 0.43, and small-model summarization harms both OpenAI models (GPT-5.3-Codex from 0.38 to 0.18) and has no effect on Claude Opus 4.6 (Figure 3).
• Constraint erosion: on ProgramBench, the first compaction drops on average 0.92 categories of anti-cheating rules, and the 30th reaches 2.65 categories; GPT-5.3-Codex drops 2.73 categories per compaction, while Claude Opus 4.6 drops only 0.51 (Table 3, Figure 4a).
• Conclusion and implications: the authors argue that current benchmarks are "compaction-friendly"—goals are fixed and progress can be re-read from disk, so only constraints that exist solely in context are truly lost; in real SWE-chat sessions, the loss rates for process instructions and scope exclusions (35.9%, 36.8%) are about 6 times that of functional requirements (6.2%), which the authors consider meaningful for the safety of autonomously running agents.