JevAgentHarness: Benchmarking, deploying, and training decision models that share control with LLMs

How to hand routine multiple-choice decisions in the agent harness to a faster, cheaper decision model while losing little success rate.

• Scenario: The authors state that LLM agents can already modify repository code, operate desktops and web pages, complete terminal tasks, and serve customers according to domain policies, but most harnesses call the large model at every step, making long tasks slow and expensive. The authors state that one public leaderboard evaluation costs about $40,000; most calls end in multiple-choice decisions, such as whether the customer service next step is to check the order, cancel, or ask the customer.
• Gap: Decision models like JEV do not generate text but give a probability to each listed option. In one record, JEV took 28% of the time and 2.6% of the price of one LLM call. The authors state that such models are built and tested on fixed questions, not yet examined inside agents; errors carry to subsequent steps, fixed question sets rarely include states after the agent's own mistakes, and each query adds latency.
• Hypothesis: The authors believe that decision models can help agents if selected by tests inside the agent, used only when confident, and trained on states the agent itself reaches.
• Approach: The authors build AGENT DECISION BENCH, from isolated questions to taking over a running agent; design JEVAGENT HARNESS, which fits thresholds on the agent's own trajectories for each choice type, with the LLM deciding below the threshold; and train an open 4B model AGENT DECISION-4B. The authors seek a delegation set whose success rate falls within 3 percentage points of the LLM running alone, at lower cost.

Decision Models

• JEV and open models: JEV, as well as Intern-Decision, StartLux-Decision, and OpenJev, score listed options rather than generating text (Almeida, 2026; InternLM, 2026; StartLuxLabs, 2026; razorback16, 2026). Existing systems use decision models for game actions, memory, upgrades, and judging (Ma et al., 2026; Jiang et al., 2026; Wu and Lim, 2026b; Li et al., 2026).
• DecisionFlow: Chen et al. (2025) structure LLM decisions. The authors call this paper decision modeling for agents and cite that work.
• REFLEX: Wu and Lim (2026b) hand actions to the LLM when confidence is below a global threshold. The authors state that this paper instead fits a threshold per decision point.

Decision and Tool Use Evaluation

• Fixed questions: Decision and tool use benchmarks score fixed questions (Benchmark Heaven, 2026; Ling et al., 2026; Patil et al., 2025; Ross et al., 2025).
• Full trajectories: Agent environments score full trajectories (Yao et al., 2025; Trivedi et al., 2024; Lu et al., 2025). The authors state that AGENT DECISION BENCH adds step-level and task-level tests.
• Same score, different decisions: Zhang et al. (2026) discuss that models with the same score may still make different decisions. The paper only describes this.

Routing, Deferral, and Training

• Routers and cascades: Selecting a model for a single call (Chen et al., 2024; Ong et al., 2025; Aggarwal et al., 2024). LLMRouter compares learned routers (Feng et al., 2026a). The authors state that routing, cascades, small function-calling models, and fast-slow agents are mostly tested on fixed questions or single choices.

• Learning to defer: Handing to an expert when uncertain (Geifman and El-Yaniv, 2017; Mozannar and Sontag, 2020). The authors state that decision models instead score listed options and defer when uncertain.

• Agent reinforcement learning: Training targets are LLMs (Jin et al., 2025; Wang et al., 2025d). The authors state that this paper trains decision models.

• Baselines and benchmarks: Task-level control is the LLM alone, main experiment Gemini 3.5 Flash, plus Flash-Lite. Decision model controls include Intern-Decision-4B, StartLux sizes, OpenJev, OpenAI Decisions, and untrained Qwen3-4B. Shortlist controls include BM25 keyword shortlist. Data includes BFCL v4, When2Call, DECISION GYM and its Shift and Hard, τ2-bench, SWE-bench Verified, Terminal-Bench 2.0, and ALFWorld.

The authors place the distinction in: decision models only score options listed by the harness, returning to the LLM when uncertain; thresholds are fitted per decision point on agent trajectories; evaluation goes to step and task level; training states come from trajectories the agent itself visited.

The overall approach is to write the harness's routine choices as multiple-choice questions, adopt them only when the decision model confidence exceeds a trajectory-fitted threshold, and have AGENT DECISION-4B learn JEV's answers on states the agent visits.

Decision points, modes, and thresholds

• Eight decision points: Three determine actions: tool, parameters, and next action when no tool. Five control loops: whether to write memory, whether to reflect after failure, whether to stop, whether to route steps to a cheaper LLM, and which notes to read back. Options are listed by the harness.
• Five modes: REPLACE always decides by the decision model; CASCADE first answers, and only adopts if confidence reaches the threshold, otherwise the LLM decides; ASSIST narrows candidates then the LLM chooses; GUARD approves or returns the LLM's proposal; TRIGGER decides whether optional steps run. The authors state all 40 combinations of points and modes are implemented (Appendix A). A delegation is a decision point plus mode and threshold, e.g., cascade for tool selection.
• Output: The model reads state text, a question, and options, and returns probabilities for each option. The answer is the most likely option; confidence is that option's probability. JEV directly returns that probability; open models score all options in one forward pass.
• Handoff condition: The authors give that, under the listed assumptions, handoff is worthwhile when

c(s) ≥ τ^⋆ = max(0, q_L − Δ C/V) where q_L is the LLM's accuracy on that choice, ΔC is the price saved per call, and V is the cost of one wrong choice. The meaning is: when savings cover the extra risk, the decision model can be slightly less accurate than the LLM.

How thresholds are fitted

• Silent recording: First run the agent on separate tasks, 150 tasks per agent LLM. The decision model answers each question but does not execute, i.e., shadow mode.
• Estimators: Estimate q_L per decision point, the price per call, and V, the drop in task success rate from one wrong choice, valued at $1 per solved question. Trajectories also give accuracy at each confidence level, so confidence need not equal true accuracy (Appendix A).
• Values: The threshold is the lowest confidence above which acceptance is worthwhile. If no qualifying interval exists, the LLM still decides. Refit when the LLM, decision model, or task rules change.
• Fitted results: The authors report that with JEV and Flash, the tool threshold is 1.00 and the parameter threshold is 0.99, so only near-certain answers are used.

AGENT DECISION BENCH

• L1: Convert BFCL v4's 1,114 tool selections and 1,949 parameter items, plus When2Call's 928, into multiple-choice questions. No agent is present; wrong answers do not affect the next step.
• L2: Answer on 5,314 identical states from 200 new DECISION GYM tasks, scoring only without execution. The reference is an acceptable next-step answer key.
• L3: The decision model makes every tool selection for a running agent; in ALFWorld, every action. No threshold; the LLM writes parameters, and selections are executed. Measures task success rate.
• DECISION GYM: Simulates a small shop's customer service. In the evaluation setting, each question provides the same 68 tools plus finish, shuffled: 7 core tools, 1 to ask the customer, 31 near-duplicates, 29 irrelevant tools. Tasks come from scripted plans, so every still-solvable state, including after detours, can derive an answer key. Answer keys are hidden from all models. In strict mode, data rewrites not in the plan end the episode. Training, fitting, and testing use disjoint random seeds. Later-added Shift changes tool names, wording, and request types; Hard has 10 or 12 customers per question and adds new rules for cancellation or shipping.

JEVAGENT HARNESS

• Question format: Each choice uses JEV's input format. Tool calls are split into one tool selection, plus one question per bounded parameter or parameter copyable from context. Free text is written only by the LLM. Any model accepting the format can be integrated (Appendix C).
• Below threshold: The LLM decides as usual. Uncertain answers incur one extra decision model call but do not hand the task to a possibly wrong choice.
• Adapters: Ask questions through existing harness hooks, including mini-SWE-agent's strong/weak LLM routing, pi's guards and triggers, and Codex CLI and Claude Code command hooks. If the adapter or decision model fails, the harness runs as before (Appendix Table 9).

Training and what counts as passing

• Data: Record 15,694 decision states on 587 training tasks. The scripted policy repeats errors seen in development; 30.9% of states occur after an error. When teacher JEV ranks the correct answer first, the target mixes the one-hot of that answer with teacher probabilities, with mixing coefficient α of 0.3; otherwise the target is just one-hot.
• SFT: Cross-entropy plus a squared error term, with square term weight μ of 0.5. Base is Qwen3-4B, LoRA rank 32, 2 epochs, learning rate 10 to the -4.
• GRPO: The SFT model makes decisions on training tasks; a fixed agent LLM does the rest. G is 8 trajectories per question. Reward is success minus a penalty for LLM calls, minus irreversible error calls; λ is 0.5, N0 is 10, η is 0.2. Agent LLM is Flash-Lite, keeping iteration 15. Appendix D gives the objective.
• Pass criteria: Stage 1 tests each delegation separately, keeping those that are cheaper and non-inferior. Non-inferiority means that on the same batch of tasks, a one-sided 5% test shows the success rate loss is less than 3 percentage points, and 5 for τ2-bench. The 11 registered tests use Holm correction to keep the probability of any false pass at 5%. Stage 2 tests passing delegations together on new tasks; its plan is registered before reading any Stage 1 results.

Experimental Setup

• Agent LLM: The main experiment uses Gemini 3.5 Flash (called Flash in the paper), native tool calling, temperature 0. The cheaper Gemini 3.5 Flash-Lite receives routed steps and simulates τ2-bench users, and is also the second agent LLM on DECISION GYM. Additional experiments also include DeepSeek-V4-Pro and V4-Flash, Gemini 3.8 Flash, GPT-6-Luna, and GPT-6.1-Sol.
• Decision models: The registered experiments primarily use JEV, with three delegations also using Intern-Decision-4B. Comparisons also include StartLux-0.8B to 27B, OpenJev, OpenJev-27B, OpenAI Decisions, untrained Qwen3-4B, and the authors' trained AGENT DECISION-4B and AGENT DECISION-0.8B. Appendix Table 32 has another round of servable public decision models.
• Data scale: L1 is 1,114 tool choices from BFCL v4, 1,949 parameter items, and 928 from When2Call. L2 is 5,314 states from 200 tasks. Stage 2 uses 500 new DECISION GYM tasks. Shifted task-level is 200 questions; Hard is 10 or 12 customers. Sample sizes for τ2, SWE-bench Verified, Terminal-Bench 2.0, and ALFWorld are in the respective tables.
• Controls: LLM alone on the same tasks; simple control without decision model; fixed threshold 0.5; BM25 keyword shortlist; random triggering at the same call rate.
• Metrics and criteria: Task success rate, cost, time, number of LLM calls, and isolated and step-level accuracy, ECE. DECISION GYM labels come from answer keys, not human. Non-inferiority margin is 3 percentage points, 5 for τ2-bench. Unless repetitions are stated in the table, each arm runs once per question, temperature 0.
• Selection rules: Stage 1 has 11 registered tests. Stage 2 combinations are selected by pre-specified rules from Stage 1 results: for each decision point, take the delegation with the largest savings that passes, otherwise leave to the LLM. Four Holm-corrected tests check whether the combination is still non-inferior and cheaper, and whether the self-trained model can replace JEV.

Main Results

The authors state that isolated question scores mis-rank decision models for agents. The table below is Figure 4b. Appendix G.5 states: L1 is 1,114 BFCL tool choices; L2 and L3 are shifted DECISION GYM, L2 is 1,250 recorded states from 100 questions, L3 is 200 questions with the model making every tool choice. The spreads in the figure are 5.1, 19.7, 57.5 percentage points. AGENT DECISION-0.8B is omitted (87.1 / 95.6 / 82.0).

Swipe horizontally to read the table

Decision model	L1 (%)	L2 (%)	L3 (%)
StartLux-9B	92.2	90.9	69.5
AGENT DECISION-4B	89.3	94.9	91.5
JEV	89.9	91.8	82.0
StartLux-27B	90.8	94.7	79.9
OpenJev-27B	90.2	92.4	69.5
OpenAI Decisions	90.6	86.1	66.0
StartLux-4B	91.3	86.5	49.0
Intern-Decision-4B	88.2	75.9	34.0

• Authors' interpretation: The authors state that isolated scores mis-rank. StartLux-9B, with the highest BFCL tool selection, completes only 69.5% of tasks; AGENT DECISION-4B, lower on BFCL, completes 91.5%, and JEV 82.0%. Kendall correlation between step-level accuracy and task success is 0.84.
• Controls: Appendix G.5 states that Flash alone solves 100.0% of these 200 shifted tasks. StartLux-27B's L3 is the first 164 questions.
• must-ask: The authors state the gap is in states where the customer must be asked. AGENT DECISION-4B answers 67.4% correctly, JEV 58.3% (§7.1). The 91.5% in Figure 4b is the same as the task success rate in Table 3 for the row with public data; the paper does not separately state the training phase in the figure caption.

Which Delegations Pass the Registered Tests

• Cascade: With Flash as the agent LLM, JEV made 27.2% of tool choices and 20.8% of parameter choices, cost down 14.2%, no result changed across 500 questions, and passed the registered tests (§7.2, Table 24). On Flash-Lite, fitted thresholds leave all tool choices to the LLM, and the cascade saves nothing.
• No threshold: Letting JEV make every tool choice reduces success rate by 19.0 percentage points. Fixed threshold 0.5 reduces by 21.5 percentage points (Table 2). The authors state that fitted thresholds preserve success rate.
• No transfer: After refitting on τ2-bench, the cheapest passing delegation still drops 8.1 percentage points, and the authors state they cannot show it transfers (Table 24).

Post-Deployment Cost, Time, and Stage 2

• Near-perfect 500 new tasks: Flash alone has 99.4% success (Table 29). JEV's selected combination has 97.2% success, a difference of −2.2 percentage points, cost saving 40.7%, failing non-inferiority. Switching to AGENT DECISION-4B V1 gives 100.0% success, difference +0.6 percentage points, cost saving 81.1%, and the test holds. On common states, V1's accuracy difference relative to Intern-Decision is +24.0 percentage points, relative to JEV +7.7 percentage points, and the test holds.
• Own GPU: In Table 2, the 4B with public data on own GPU saves 86.3% cost and 60.3% time. The authors state that AGENT DECISION-4B makes agents 86.3% cheaper and up to 2.5x faster. The shared GPU row saves 81.1% cost, with time saving −49.5%, i.e., slower. Appendix C.1 states that with about four-way sharing, queueing accounts for 66.1% of round time.
• Harder settings: The authors state that Table 1 rows are added after Stage 1. On ALFWorld, the 4B trained on its states raises success from 90.7% to 95.0% (n=140), cost saving 89.0%, time from 37.0 seconds to 18.2 seconds. Coding agents are routed by JEV between strong and weak LLMs, cost down 7.0–10.5%, success varies with harness; SWE-agent from 63.0% to 57.0% (n=100). On Hard tasks, delegation maintains or improves success but saves only 4.6–30.5% cost and is slower.

Can the 4B Model Learn These Decisions

• SFT is the main gain: On shifted states, step-level accuracy rises from 52.9% to 98.4%, isolated accuracy only rises 7 percentage points (Table 3). SFT uses 2.2 GPU hours. Qwen3-4B's success on shifted tasks goes from 13.5% to 96.0%; StartLux-4B with the same recipe goes from 49.0% to 98.0%.
• GRPO: Adding 4.5 GPU hours and $26.53 of LLM calls on top of SFT, the authors state no measurable gain. Without SFT, after GRPO shifted task success is 11.5%, before is 13.5% (Appendix D.4).
• Failure modes: The authors report that all 34 JEV failure tasks start with an irreversible wrong tool choice, usually canceling when it should ask. Models trained on agent states do not have such errors (Appendix G.7). On Hard, 10 customers, when the model has not learned the rule, AGENT DECISION-4B makes most cancel or ship decisions but is correct 81.6%, task success 3.0%, LLM alone 88.0%. After retraining on that rule, held-out accuracy goes from 76.5% to 95.3% (Appendix D.6).

Other Ablations and Analyses

• Fixed threshold 0.5: success −21.5 percentage points, cost saving 84.5%; shadow mode fitted threshold: +0.0 percentage points, saving 14.2% (Table 2).
• Three-tool shortlist: −7.8 percentage points, cost saving 53.7%; keyword shortlist task success 18.8%, JEV shortlist 92.0% (§7.2).
• Stop guard: success difference −0.6 percentage points, cost 6.2% higher, registered test non-inferior (Table 24).
• Sending Flash-Lite's steps to Flash: cost 36.2% higher, success difference +0.6 percentage points, test not held (Table 24).
• Continued training on public tool data: isolated accuracy from 78.8% to 85.1%, shifted task success from 96.0% to 91.5% (Table 3).
• Refitted tool cascade on GPT-6-Luna: success from 45.0% to 83.0% (n=100), cost 8.4x the reference (Table 1).

Limitations and future directions noted by the authors

• Language and agent LLM: Tasks and prompts are all in English; most runs use Gemini, with a few using DeepSeek and OpenAI. The authors state that with other languages or agent LLMs, the types of choices worth delegating may differ (Limitations).
• Test type: The main test is non-inferiority, indicating that the success rate drop is less than a fixed margin, i.e., 3 percentage points, and 5 for τ2-bench. The authors state that demonstrating equal success rates requires equivalence testing (Limitations).
• Binding to a specific agent: Thresholds and the trained decision model are fitted to that agent's own runs. The authors state that new agents, new tools, or new task families require refitting; to get better accuracy, training on that agent's states is needed, as in ALFWorld (Limitations).
• Free-text actions: The coding agent's actions are free text; the decision model can only choose which LLM writes the step. The authors state savings are 7.0–10.5%, and handing read-only file steps to the fast model gives no reliable savings because the decision model can barely distinguish these steps (Limitations, Appendix C.3).
• Price and latency: The authors state that reported prices and latencies were recorded at the time and will vary with vendor, hardware, and how the decision model is served (Limitations, Appendix F).
• Missing comparisons and drift: The authors state that no comparison was made with learned routers like RouteLLM on the same tasks, and GRPO showed no measurable gain over SFT in this run. In deployment, the proportion of answers above the threshold can be tracked to detect task drift and the need to refit thresholds (Limitations).

Experimental coverage

• Agent LLMs: Experiments ran Gemini 3.5 Flash, Flash-Lite, DeepSeek-V4-Pro, DeepSeek-V4-Flash, Gemini 3.8 Flash, GPT-6-Luna, and GPT-6.1-Sol (§6, Table 1, Table 28).
• Decision models and hierarchy: Main comparisons include JEV, AGENT DECISION-4B and its V1, 0.8B recipes, Intern-Decision-4B, StartLux in five sizes, OpenJev, and OpenAI Decisions. Evaluation is in L1 isolated questions, L2 recorded states, and L3 taking over a running agent (§3, Figure 4).
• Tasks and harnesses: DECISION GYM includes normal, Shift, and Hard; public benchmarks include BFCL v4, When2Call, τ2-bench, SWE-bench Verified, Terminal-Bench 2.0, and ALFWorld. Adapters connect to mini-SWE-agent, SWE-agent, and pi (§3, Appendix C).
• Delegation grid: Eight decision points, five modes; Appendix A states all 40 combinations are implemented. Stage 1 has 11 registered tests, Stage 2 has 4, plus 6 secondary tests (Table 24).
• Quantities specified in the protocol: Calibration is 150 questions per agent LLM; training is 587 questions, 15,694 states; GRPO uses 8 trajectories per question, keeping iteration 15. Unless otherwise stated in tables, each arm is one question per run, temperature 0. The paper states no comparison with RouteLLM on the same tasks.

This work moves the multiple-choice decisions in the agent loop from the large LLM called at every step to a small decision model, and provides evaluation, delegation rules, and training recipes.

• Problem: The authors state that long tasks are slow and expensive because the harness calls the large model at every step, while most calls end in enumerable choices, such as the next tool, parameters, or whether to stop. Scores on fixed questions do not indicate whether the model is usable in a running agent.
• Method: AGENT DECISION BENCH has three levels: isolated multiple-choice questions, recorded trajectories that are only scored, and task-level where choices are actually executed. JEVAGENT HARNESS fits confidence thresholds on the agent's own trajectories for each decision point; below the threshold, the LLM still decides. AGENT DECISION-4B is based on Qwen3-4B, trained with SFT on JEV answers on states visited by the agent, then GRPO in the loop.
• Evaluation findings: On shifted tasks, when the model makes every tool choice, the task success rate of StartLux-9B, which has the highest BFCL tool selection, is 69.5%, AGENT DECISION-4B is 91.5%, and JEV is 82.0% (Figure 4b). Without fitted thresholds, JEV taking over all tool choices reduces success rate by 19.0 percentage points.
• Deployment numbers: The fitted tool and parameter cascade saves 14.2% cost on Flash's 500 questions with unchanged results. On 500 new DECISION GYM tasks, the JEV combination drops 2.2 percentage points and fails non-inferiority; switching to V1 gives 100.0% success rate, saves 81.1% cost, and passes. Adding public data and own GPU, Table 2 cost saving is 86.3%, and the authors claim up to 2.5x faster. On ALFWorld, locally trained 4B saves 89.0% cost.
• Authors' conclusion: The authors believe the results support the hypothesis that decision models help agents when tested inside the agent, used only when confident, and trained on states the agent reaches. Coding agents' actions are free text, so the decision model can only choose which LLM writes the step, yielding smaller savings.