Your policy is actually a Harness: the decision geometry of the Agent interface

Core question: under KL regularization control, what a policy learns from its harness (prompts, tools, control logic), and what is retained when the harness changes.

• Problem setting: language model agents are post-trained and deployed through a harness; automated harness design modifies prompts, tools, and workflows, and scores candidate harnesses using a fixed model (Introduction). The authors argue that this scoring assumes what the model learned under the old harness transfers to the candidate harness.
• Shortcomings of existing practice: the authors state that the same interface change can improve one model's accuracy while lowering that of a fine-tuned model (Introduction, citing Mu et al., 2025); training also moves behaviors provided by the prompt or harness into the weights, so fixed-weight comparisons conflate "what the harness provides" with "how well the current checkpoint uses it."
• Core observation: under KL regularization control the optimal policy is a softmax over action scores, and the optimal policy's log odds exactly recover the harness's relative preference over each action, so a trained policy carries its harness's preferences; deployment return equals the harness's ceiling minus the policy's mismatch, training changes only the mismatch, and swapping the harness moves both terms (Abstract).
• What the paper proposes: a decision-geometry framework—relative action scores, ceiling, mismatch, and an adaptation threshold—plus a controlled validation on MiniWoB++ and a teacher-model estimation method on coding agent harnesses (contributions 1–3).
• Two motivating experiments: on MiniWoB++, reversing the observation field order leaves all optimal decisions unchanged, yet the trained model's success rate drops from 85.0% to 19.2%; a harness that adds batch clicking raises the ceiling, and after frozen-model transfer the success rate drops from 85.0% to 72.9%, rising to 88.2% after adaptation (Figure 1, Table 8).

Regularized control and decision geometry

• Levine (2018), Haarnoja et al. (2017; 2018)—control as inference and maximum-entropy RL, expressing the optimal policy via rewards, action values, and reference distributions.
• Geist et al. (2019), O'Donoghue (2022)—regularized MDP theory, providing the geometry of the Gibbs optimal policy and value gaps; Schulman et al. (2017) give an equivalent perspective between policy gradients and soft Q-learning.
• Rafailov et al. (2023; 2024)—DPO and its sequence extension connect language model policies to rewards and action values; this paper's optimal action scores merge semantic action values with a semantic reference prior, used to compare decisions across interfaces.

Behavioral identifiability and reward transfer

• Ng & Russell (2000), Cao et al. (2021), Skalse et al. (2023)—ambiguity and partial identifiability when inverse RL recovers rewards from behavior; Gleave et al. (2021)'s EPIC compares reward functions under policy-preserving transformations.
• Schlaginhaufen & Kamgarpour (2024)—study the transfer of rewards recovered by regularized inverse RL when the transition law changes.

Interfaces and agent design

• Yang et al. (2024) (SWE-agent), Sclar et al. (2024)—interface design and prompt format affect agent behavior; Mu et al. (2025) (GUI-360°) report that the same interface change improves GPT-4o action prediction accuracy while lowering that of fine-tuned Qwen.
• Hu et al. (2025) (ADAS), Zhang et al. (2025b) (AFlow), Zhang et al. (2026) (Darwin Gödel Machine), Fernando et al. (2023) (Promptbreeder), etc.—automatically search agent implementations, workflows, prompts, and tools, scoring with execution feedback from a fixed model. The paper points out that such pipelines conflate a candidate's optimal return with the current checkpoint's gap to it; this paper separates the two.

In-context distillation and joint adaptation

• Snell et al. (2022)—in-context distillation lets a model reproduce prompt-induced behavior without the prompt; Ye et al. (2026) (Harness-Zero) distills a specialized harness into the weights. The authors state that this paper's identifiability result gives the decision information such training can carry (the training harness's relative action scores), and the readout study shows that the distilled policy still changes its choices with the interface.
• Hebbar et al. (2026) (SIA), Chen et al. (2026) (Co-Harness), Kim et al. (2026) (WHALE), Wei et al. (2026) (HarnessBandit)—jointly adapt harness and weights; the authors state that these methods can use ceiling and mismatch to judge whether a candidate harness provides too little or is not being used well, and this paper's adaptation threshold gives the required mismatch reduction.

Positioning: the authors position this paper as providing a decomposition tool for harness comparison—separating each candidate's optimal return (ceiling) from the evaluated checkpoint's gap to that optimum (mismatch)—rather than proposing a new harness search or training method (Appendix A).

整体思路：把 harness 诱导的控制问题写在语义动作（执行的操作及其参数）上，证明最优策略的 log odds 恰好辨识 harness 的相对动作分数，再把部署回报恒等分解为 ceiling 减 mismatch，由此导出适配何时有益。

语义控制问题

• 接口 I = (cI, dI, eI) 把状态编码为上下文、把模型输出解码为动作并执行；语义策略 Πθ,I(a|s) 是把解码到同一动作的所有输出的概率求和（Eq. 1），因此固定权重不固定语义策略。
• 目标为正则化回报 J_I(Π) = E[∑_t γ^t r_I(s_t,a_t) − τ log(Π(a_t|s_t)/π_I^ref(a_t|s_t))]，即任务回报减去对语义参考策略的 KL 惩罚；每次对被评模型的调用（含验证、重试）都算一次决策。
• 受控研究：六个 harness（H0–H5）作用于三个 MiniWoB++ 任务族，每个诱导控制问题编译为有限图；提示词列出完整动作菜单、每动作一个 token，下一 token 分布精确给出 Πθ,I；奖励为二值任务成功，γ=0.95、τ=0.1，冻结基座作参考（第 2 节、附录 F.1）。

策略学到什么

• 最优动作分数 F_I^∗(s,a) = Q_I^∗(s,a) + τ log π_I^ref(a|s)，最优策略是其 softmax；给同一状态所有分数加常数不改变策略，故只有状态内分数差影响决策。
• Proposition 3.1：两接口最优决策等价当且仅当最优分数类相同；若检查点在训练接口上最优，则其缩放 log 概率等于训练 harness 的相对分数。行为只确定相对分数，不确定绝对值。
• 检查点的 readout 定义为居中化分数 τP⊥log Πθ,I，即它通过接口表达的动作偏好；geometry error 是 readout 与最优居中分数在固定状态库上的均方差。

迁移为何失败、代价多少

• 部署分数误差分解为三项（Eq. 5）：target shift（最优分数的变化）、training residual（原训练留下的误差）、readout drift（模型表达分数的变化）；保持最优决策不变的呈现改动可隔离 readout drift。
• Proposition 5.1：固定接口下回报差距等于沿被评策略轨迹累积的 KL 加权和，权重是策略自己的折扣占用；等价地由分数的 Bregman 散度 DΦ 度量。
• 定义 ceiling C(I) 为 harness 的最优正则化回报，mismatch M(θ,I) 为检查点未实现的回报，则 J = C − M 恒成立；Corollary 5.2 给出适配有益的充要条件 M(θ1,I2) < T = ∆C + M(θ0,I1)。

部署中的测量

• 用随机停止的 pilot rollout 从折扣占用中采样状态，每状态采 N 个动作平滑成分布，按 Eq. 11 估计 mismatch；Proposition 6.1 用居中分数误差给出 mismatch 上界 ε²d/(4τ(1−γ))。
• 编码智能体无最优解时，用更强的教师模型（GPT-5.6-luna）代替最优策略估计 ceiling 与 mismatch，输出映射到 13 个共享动作类别，以被评模型自己的类别分布作参考（第 6.3 节、附录 F.7）。

Experimental setup

• Models tested: Qwen3.5-2B (MiniWoB++ post-training and transfer, LoRA rank 16, 1,024 updates, 4 tasks per update); Qwen3.5-9B (coding harness and exact comparison in F.8, frozen); the H4 reviewer is another frozen Qwen3.5-2B.
• Benchmarks: three MiniWoB++ task families (click-checkboxes, click-checkboxes-soft, ascending-numbers), fixed test set of 576 tasks, 192 per family; Terminal-Bench Lite 39 tasks, scored by the official verifier.
• Harnesses: H0 direct execution; H1 batch clicking; H2 automatic check before submission; H3 program verification tool; H4 language model reviewer tool; H5 instruction-only change (Table 4). On the coding side, the native configurations of DeepAgents, Pi coding-agent, DeepSeek Harness, Terminus-2, and mini-SWE-agent.
• Metrics: task success rate (probability of exact completion), geometry error (MSE of centered scores), regularized return J, ceiling C, mismatch M; prediction quality via Spearman ρ and RMSE.
• Review and determination: all MiniWoB++ quantities are computed exactly by dynamic programming on the compiled graph; on the coding side GPT-5.6-luna serves as teacher in place of the optimal policy, with the objective and teacher distributions each estimated from 128 and 64 samples.
• Repeats and uncertainty: each coding harness is run 3 independent times; MiniWoB++ intervals use 2,000 paired-task bootstraps; the two adaptation branches share source weights, budget, and task sampling.

Main results

Post-training binds preferences to the training harness (Figure 3, Qwen3.5-2B, after 1,024 updates):

Training harness	Task success rate (%)	Geometry error at end of training (×10⁻²)
H0	85.0	0.63
H1	86.8	1.39
H2	95.3	0.61
H3	75.6	0.89
H4	75.2	0.90
H5	94.7	0.61

• Across the six harnesses, geometry error drops 69%–90% from its initial level, and on held-out tasks not involved in updates the drop differs from that on training tasks by less than one percentage point (Appendix F.1).
• In the 6×6 transfer matrix, under each deployment harness the lowest error is achieved by the checkpoint trained on that harness (boxed entries in Figure 3c); the authors state that post-training makes preferences specific to the training harness.

Presentation changes isolate readout drift

• What was tested: a frozen checkpoint trained on H0, with observation presentation changed while transitions, rewards, semantic reference, and optimal policy are held fixed (confirmed numerically, Table 1, Appendix F.2).
• Results: reversing observation fields changes the most likely action in 52.0% of states, drops success rate from 85.0% to 19.2%, and raises mismatch from 0.015 to 0.484; cyclic shifting of action codes drops it to 57.1%, while changing the menu to a list format instead gives 86.0%.
• Authors' interpretation: unlike the prompt-format sensitivity study of Sclar et al. (2024), the controlled problem here proves the optimal policy is unchanged, so all effects come from the model's readout.

Adaptation turns harmful revisions into beneficial ones

• What was tested: two branches split from the H0 source checkpoint, with the same budget continuing H0 training versus adapting to H1 (batch clicking), evaluated on the 576 test tasks (Tables 8, 9).
• Results: H1's ceiling gain is 0.01248; frozen transfer increases mismatch by 0.12252 and lowers return by 0.11004; after adaptation, success rate under H1 is 88.17%, exceeding the source pair (85.05) and the H0-continuation branch (83.37), with a final recovery difference of 0.00844 [0.00135, 0.01447] and a gain over continued training of 0.00840 [0.00254, 0.01411], both intervals excluding zero.
• Across swaps: of 30 ordered swaps, 22 are rejected by the frozen comparison, of which 9 with T>0 proceed to adaptation; all 7 that raise the ceiling exceed the source pair, and the 2 that lower the ceiling show no gain (Table 10). The predicted gain T−r2 has the correct sign for all eight pre-registered swaps with ranking Spearman ρ=0.95, whereas the frozen return change has only ρ=0.17; H0→H5 and H4→H3 have similar frozen losses, yet the former's gain is tied for largest and the latter's is zero.

Sampling measurement and coding harnesses

• Sampling estimates: in the configuration sweep, the predicted RMSE of ceiling minus sampled mismatch is 0.00230, close to frequency rollout (0.00240), while ceiling alone is 0.23470; on predicting recovery and ranking differences at adaptation nodes, the mismatch method has RMSE about 0.0025–0.0029, roughly ten times more accurate than frequency rollout (about 0.027) (Table 12).
• Coding harnesses: with frozen Qwen3.5-9B, Terminal-Bench Lite accuracy across the five harnesses ranges from 47.9% (DeepAgents) to 64.1% (Terminus-2); the teacher-estimated Ĉ−M̂ has Spearman ρ=1.0 with accuracy, whereas Ĉ alone has ρ=0.7 and M̂ alone ρ=−0.7 (Table 2).
• Sources of mismatch: DeepAgents' per-decision mismatch is similar to Terminus-2's, but the target model makes 14.2 decisions per trajectory (Terminus-2: 4.9), accumulating the largest mismatch; DeepSeek Harness has the largest per-decision mismatch (Table 13, Figure 6).

Other ablations and analyses

• Exact comparison (Appendix F.8): with frozen Qwen3.5-9B as its own reference, ceiling alone ranks the most successful harness last, mismatch alone ranks H0 after two worse harnesses, and only C−M gives the fully correct order.
• Measurement budget calibration (Table 11): at m=256, N=1024, the worst-node return-difference RMSE is 0.00672, at a cost of about 42,553 actor queries per task per condition.
• Three-way prediction (harmful/neutral/beneficial): 71 of 72 comparisons in the configuration sweep are correct; 6 of 7 recovery nodes agree, and all 7 ranking nodes are correct (Appendix F.6).
• Negative results: after H1→H0 adaptation the checkpoint prefers H0 by a magnitude of 0.041 but still falls below the source pair, and the authors state that the ranking reversal itself does not imply a net gain; after H2→H5 adaptation the success rate changes by −0.5 percentage points (Table 10).

Limitations and future directions noted by the authors

• Harness evaluation method: The authors say harness evaluation should report ceiling alongside frozen return, evaluate multiple checkpoints under each harness rather than a single checkpoint, and examine how presentation changes alter model selection (Section 7 Discussion).
• Measuring distillation and joint adaptation: The authors say harness distillation and harness-weight joint adaptation methods can use sampled actions to separately track ceiling and mismatch (Section 7).
• Teacher as an approximation to the optimum: The authors say the teacher is not necessarily optimal; Ĉ estimates a lower bound on the ceiling and M̂ measures distance to the teacher, so only the ranking of Ĉ−M̂ and the accuracy ranking are compared (Appendix F.7).
• Category-level comparison: The authors say the 13 action categories merge actions with different parameters and effects, and the estimates compare decisions only at the category level (Appendix F.7).
• Mechanism not identifiable: Proposition C.1 points out that the same optimal score field can be realized by a continuous family of reference priors, rewards, or even transition laws, so optimal behavior cannot uniquely determine the interface mechanism (Appendix C.3).

Scope of experiments

• The models tested are Qwen3.5-2B (all controlled MiniWoB++ experiments) and Qwen3.5-9B (coding harnesses and the F.8 exact comparison), two in total; the teacher is GPT-5.6-luna.
• The MiniWoB++ controlled study covers three task families, six harnesses, and 576 fixed test tasks; the coding study covers five harnesses and 39 Terminal-Bench Lite tasks, of which the ceiling/mismatch estimation uses only 3 probe tasks.
• The post-training budget is 1,024 updates per harness with 4 tasks per update; each adaptation branch gets 1,024 additional updates; each coding harness is run 3 times independently.
• On the coding side, the number of target trajectories fully probed is 6–9 out of 9 (depending on the harness); the paper reports no human evaluation and no task types beyond MiniWoB++.
• The paper discusses the prompt-format sensitivity study of Sclar et al. (2024), noting that this work differs in that the optimal policy of the control problem is provably unchanged (Section 4.2).

An analytical paper on agent harnesses: the authors formalize the harness (prompts, tools, control logic) as a KL-regularized control problem over semantic actions, answering "what the policy learns from the harness, and what is retained after the harness changes."

• Problem: automatic harness design uses a fixed model to score candidates; the authors argue this conflates the value the harness provides with the degree of fit to the current checkpoint, and that the same interface change can improve one model yet harm a fine-tuned model.
• Method highlights: the log odds of the optimal policy exactly identify the harness's relative action scores (Proposition 3.1); deployment return is identically equal to ceiling minus mismatch (Proposition 5.1); the necessary and sufficient condition for adaptation to be beneficial is that post-adaptation mismatch is below the threshold T = ∆C + M(θ0,I1) (Corollary 5.2); when no optimal solution exists, mismatch is estimated using sampled actions or a teacher model.
• Result one: on six harnesses in MiniWoB++, post-training reduces the geometry error of Qwen3.5-2B by 69%–90%, and under each deployment harness the checkpoint with the lowest error is the one self-trained on that harness; the authors state that preferences become harness-specific (Figure 3).
• Result two: reversing the observation field keeps all optimal decisions unchanged, yet lowers the success rate of the trained model from 85.0% to 19.2%, and the most likely action changes in 52.0% of states (Table 1).
• Result three: among nine swaps rejected by the frozen comparison with T>0, seven that raise the ceiling all exceed the source pair after adaptation; the predicted gain T−r2 ranks with Spearman ρ=0.95, while the frozen return change ranks with only ρ=0.17 (Table 10).
• Result four: on coding agents, the teacher-estimated ceiling minus mismatch reproduces the Terminal-Bench Lite accuracy ranking of five harnesses (ρ=1.0), while any single term alone ranks them incorrectly (Table 2).
• Authors' conclusion: harness evaluation should report both ceiling and frozen return, and evaluate multiple checkpoints under each harness; harness distillation and joint adaptation can use sampling measurements to track ceiling and mismatch separately (Section 7).