The paper proposes the Intent-Eval benchmark, revealing that models confuse rejected intents in multi-turn dialogue

In multi-turn tasks, when the user mentions a change and then refuses, models often still execute according to the mentioned content rather than the still-active requirement.

• Scenario: In multi-turn, the user clarifies or changes their mind, leaving content in the context that should not affect the answer. The authors believe that even when information is complete, splitting into multiple turns is harder than a single message (Laban et al., 2026).
• Existing gaps: Requirement change benchmarks (Tack et al., 2026; Wu et al., 2026) do not compare clarification, acceptance, and refusal on the same task, nor do they examine whether the impact deepens or persists with dialogue.
• Core observation: The authors state that models treat mentioned content as still-active requirements. In Figure 1, Retained refuses the proposal, Revised accepts the same proposal, and only the latter changes the goal.
• This paper proposes: Intent-Eval includes 414 source tasks, four domains, and four multi-turn conditions, with single-turn controls. Intent-OPSD supervises a student with complete single-turn tasks matched to decisions, where the student only sees the full dialogue.
• Author-reported drops: Eight-model average: multi-turn Original is 36.30 percentage points lower than single-turn, Neutral is another 4.43 lower; Retained and Revised are 8.06 and 5.55 lower than Original.

The authors divide related work into multi-turn dialogue capability, evolving user intent, and on-policy distillation.

Multi-turn dialogue capability

• Early multi-turn benchmarks: MT-Bench-101, MT-Eval, MultiChallenge (Bai et al., 2024; Kwan et al., 2024; Deshpande et al., 2025) evaluate instruction retention, contextual reasoning, and response refinement.
• LiC: Laban et al. (2026) report performance drops when tasks unfold over multiple turns. The 414 source tasks in this paper are adapted from LiC.

Evolving user intent

• Intent Mismatch: Liu et al. (2026) reconstruct instructions from dialogue context to study intent alignment. The paper calls this a harder, more fundamental problem.
• Requirement changes: SEQUOR (Canaverde et al., 2026) and Evolving User Intent (Tack et al., 2026) study requirement changes. The introduction states that Tack et al. (2026) and Wu et al. (2026) do not compare clarification, acceptance, and rejection on the same task.
• Specifications and commitments: SpecPath (Wu et al., 2026) compares interaction histories and final specifications that may drop canceled requirements. Trip+ (Chen et al., 2026a) evaluates whether valid commitments are preserved under travel requirement changes.

On-policy distillation

• Teacher distribution: Agarwal et al. (2024) use the teacher distribution to supervise the student's own generated prefixes.
• Same-model teacher: Self-Distilled Reasoner (Zhao et al., 2026) lets a same-model teacher see reference solutions. Ye et al. (2026) let the teacher see information the student does not have.
• Transfer to multi-turn: Found in Conversation (Chen et al., 2026b) transfers single-turn capabilities to dialogue. MAIGO (Zheng et al., 2026) uses cleaned histories for self-distillation. The paper states that Intent-OPSD instead constructs still-valid single-turn tasks based on user decisions.

Baselines and data

• Comparisons: The baselines for RQ4 are Base and matched SFT on the same base model, with the evaluation environment being Intent-Eval. Domain sources are BFCL, HumanEval, LiveCodeBench, Spider, GSM8K; training also uses MBPP.

The authors state that prior work focuses on evolving intent in specific forms or domains, while Intent-Eval jointly examines clarification, rejection, and revision, with matched single-turn comparisons and longer conversations.

The authors use Intent-Eval to pin down the "still-active requirement," then use Intent-OPSD to align the student to this requirement on the full conversation. The teacher and student are initialized from the same model, and the teacher is not used at inference time.

Four multi-turn conditions

• Active intent: The authors call the still-active requirement active intent. Each source task changes only one requirement, and the reference answers for the original and revised tasks are denoted y(0), y(1).
• Four conditions: Original inserts no event; Neutral inserts two clarifications; Retained proposes a change and then refuses; Revised accepts the same proposal. The events are two user messages inserted after that requirement appears.
• Scoring: Only the final reply is scored. The first three conditions score against y(0), Revised against y(1). The assistant reply depends on the history Hk up to the current user message.

Construction, single-turn controls, and longer conversations

• Four-step construction: Modify tool parameters, entry names, query conditions or output fields, and numeric premises by domain; then require unambiguity, changing only one requirement, and verifiability. Select one revision, construct y(1), render after human review. All 414 pass automated checks. Appendix A has details and full examples.
• Single-turn controls: SOrig directly states the original task, SDir directly states the revised task. SRet and SRev append the same proposal after the original task in the same user message, then refuse or accept. The first three multi-turn conditions match SOrig, Revised matches SDir.
• Longer conversations: The number of additional blocks k goes from 0 to 4, each block with two user messages. Cumulative protocol adds k clarification blocks on Original, or k refusal blocks with different proposals, with the original task remaining active. Sustained protocol adds k Neutral blocks after the decision, with the active task unchanged. Appendix C gives the construction.

Self-distillation for the decision condition

• Who sees what: The student sees the multi-turn history Hv, the frozen teacher sees the full single-turn task matching the decision. The choice is c_v=S_Dir (if the condition is Revised, otherwise SOrig). If refused, the teacher stays on the original task; if accepted, it switches to the revised task.
• Source filtering: The teacher must finish normally on both SOrig and SDir and pass their respective validators for the source to enter training. Reference answers and validation results are used only for filtering and are not provided to either model.
• Loss: The student samples the final answer on-policy. On the same student-generated prefix, the Jensen–Shannon divergence between the student's and frozen teacher's next-token distributions is computed over the full vocabulary, then averaged over answer tokens. Intermediate replies remain in the history and are not included in the loss. Wrong answers are also included in the loss.
• Equal-weight updates: The four conditions are equally weighted, i.e., L=1/4(L_Orig+L_Neu+L_Ret+L_Rev). Only the student is updated; the updated student continues sampling. At inference, the student only sees the conversation and does not need the teacher or single-turn prompts.

Training implementation and evaluation

• Shared hyperparameters: Intent-OPSD and matched SFT both use LoRA (rank 16, alpha 32, dropout 0), AdamW, learning rate 1×10−5, 1 epoch, seed 42 (Table 7).
• SFT: Reuses the same batch of sources and conversation histories, with the target being the teacher answer verified at admission time, doing hard-target learning with equal weights across the four conditions.
• Budget: Code, Database, Math training sequences are at most 32768, with intermediate replies and final answers each up to 4096. Actions is 8192, 128, and 512. Each update uses 4 sources (Actions, Database) or 8 sources (Code, Math).
• Validators: Actions compares call structure and parameters; Code checks entry names and runs tests, and defining both names does not count as passing; Database compares execution results; Math extracts and normalizes the final number. Appendix B.2 gives the full prompt for the user simulator: it restates information fragments turn by turn, with input being the existing conversation and fragment list, and output being the next user message.

Experimental Setup

• Models: Ten, namely Qwen3.6-27B, Qwen3-8B, Qwen3-4B-Instruct-2507, Qwen2.5-7B-Instruct, Llama-3.1-8B, GPT-5.6-Luna, DeepSeek-V4-Flash-0731, Claude-Sonnet-5, Gemini-3.7-Flash, Gemma-4-26B-A4B. Appendix B.1 specifies that Llama-3.1-8B uses Unsloth's Meta-Llama-3.1-8B-Instruct, and Gemma is Gemma-4-26B-A4B-it. Table 1 reports eight models, Table 8 adds two earlier Qwen models.
• Data: 414 source tasks: Actions 104, Code 100, Database 107, Math 103. Each source has 4 multi-turn conditions plus 4 single-turn controls, 3312 per model (Figure 2: 832, 800, 856, 824). Multi-turn averages 7.14 turns, range 3–14.
• Generation: Local models temperature 0, top-p 1, seed 0; single-turn output limit 1000, multi-turn 4096. API uses default sampling, same limits including reasoning tokens. Multi-turn user simulator fixed as Qwen3.6-27B, temperature 0.3, output limit 192.
• Metrics: Accuracy of domain verifiers, only final response evaluated. The paper does not use LLM judges. RQ2 additionally reports the proportion of new errors containing invalidated content.
• RQ4: Students are Qwen3-8B, Gemma-4-26B-A4B, Qwen2.5-7B-Instruct, Llama-3.1-8B. Controls are Base and matched SFT. Each domain has 500 training candidates, 50 validation sources, non-overlapping with evaluation sources.
• Depth and diagnostics: Cumulative protocol uses 372 sources with four distinct proposals; persistent protocol uses 414 sources; k from 0 to 4. RQ2 pairs outputs where Original is correct and the other two conditions are wrong across ten models.

Main Results

The paper does not provide per-model cross-domain averages; the table below only reproduces the equal-weight summary of the eight models from Table 1, excluding Table 8. The second column for Single is Direct Rev., for Multi is Neutral; they are not the same condition.

Swipe horizontally to read the table

Setting	Original	Second column	Revised	Retained
Single	91.73	90.85	85.11	84.21
Multi	55.43	51.00	49.88	47.37

• Three types of drops: The authors decompose the multi-turn gap into disclosure, dialogue overhead, and decision confusion, giving 36.30, 4.43, 5.55, and 8.06 percentage points. The authors state that the two task versions are close under direct statement, so the difficulty of revision tasks cannot explain the post-decision drop; in single-turn, relative to direct statement, Revised and Retained drop 5.74 and 7.52 percentage points.
• Code: The authors state that GPT-5.6 and DeepSeek-V4 achieve at least 85% on all multi-turn Code conditions, and suggest this may relate to progress in conversational programming, with other structured tasks not showing the same performance. In Table 1, Claude-Sonnet-5's Code Neutral is 58.00%.
• Not every cell is uniform: The authors state that the additional drop for Neutral appears across all eight models. The domain cells in Table 1 are not all uniform; for example, Qwen3.6-27B's Database Multi Neutral is 43.93%, while Multi Original is 42.99%. The lowest multi-turn cell in that table is Llama-3.1-8B's Code Retained at 10.00%.

Invalidated Content (RQ2)

• What was measured: Final responses where Original is correct but Retained or Revised is wrong. Invalidated content is the refused proposal in Retained and the replaced original requirement in Revised. Inspection rules are in Appendix E.2.
• Results: Across ten models, new Retained errors 446/748 contain invalidated content (59.63%), new Revised errors 201/649 (30.97%). Figure 4 by domain: Math 67.70% and 47.94%, Code 69.60% and 10.08%, Database 56.67% and 23.24%, Actions 42.31% and 36.88%.
• Author interpretation: The authors state this is consistent with mentioned-as-in-effect confusion, more common after refusal. They believe Math is high on both sides, possibly related to step-by-step calculations and not updating intermediate results. Appendix E.2 gives examples of wrong answers using refused or replaced values.

Dialogue Depth (RQ3)

• Cumulative: On the four models in Figure 5, the authors state the first block has the largest average single-step drop, with accuracy generally continuing to decline afterward. Final depth relative to Original: Neutral drops 8.13 percentage points, Retained drops 20.77 percentage points.
• Persistent: After the decision, adding four Neutral blocks, relative to their k=0, Retained drops 2.72 percentage points, Revised drops 3.50 percentage points. The authors state that even after the decision is clear, continued clarification can still reduce accuracy.
• Other six models: Appendix E.3 and Figure 7. The authors state that averaging over six models and four paths, the first block has the largest drop; repeated Retained average loss is greater than Neutral. At final depth, post-decision clarification still keeps the mean below the corresponding baseline, with differences across models and domains. The endpoint values in the figure are not listed individually in the text.

Decision-Condition Distillation (RQ4)

Table 2's Mean and Overall Avg. are equal-weight across four models and four conditions. Units are %.

Swipe horizontally to read the table

Method	Original	Neutral	Revised	Retained	Overall
Base	46.10	42.71	40.96	31.46	40.31
SFT	55.57	51.83	43.03	39.13	47.39
Intent-OPSD	59.42	55.55	50.93	38.57	51.12

• Author interpretation: The authors state that relative to Base, SFT's average gain is 10.81 and 3.73 percentage points; all 16 model–domain pairs are above Base, 13 above SFT. Relative to SFT, the Revised gain is more pronounced, and Retained mean is close. The authors believe the method may be useful when intent tracking is the bottleneck, and state Actions is 21.46 percentage points above Base, with Code gain smaller.
• Cell-level exceptions: Not every cell is above Base. Qwen3-8B's Math Retained: Intent-OPSD 23.30%, Base 37.86%, SFT 28.16% (Table 2).
• Depth: k=1 to 4, four-path summary relative to Base gains are +9.61, +11.74, +11.67, +10.64 percentage points, relative to SFT +2.91, +3.86, +2.49, +2.16 (Table 2). The authors state that at each test depth, the summary gain relative to both baselines remains positive.

Other Ablations and Analyses

• Table 8's Qwen3-4B-Instruct-2507 and Qwen2.5-7B-Instruct have Multi Original lower than Single Original in every domain, and Multi Retained even lower; they are not included in Table 1's eight-model summary.
• User simulator manual check: 500 dialogues (125 per domain), 2345 user turns; with fragmented messages 2284/2317 (98.6%) judged to convey task-relevant information, contextually appropriate 2345/2345 (100%) (Appendix B.3).
• Table 6: Teacher included only if both single-turn versions are correct. Qwen3-8B has Actions 435, Code 316, Database 201, Math 451. Database candidates after checking remain 381, then admission is done.
• Table 9's Accumulation: Retained path, at k=1 Intent-OPSD is 38.84%, SFT 40.32%, Base 32.19%; on that path k=1–4, Intent-OPSD is below SFT.
• Training dialogues are generated online according to LiC's fragmentation method, and history changes as the student updates (Appendix D.1). The paper does not report multiple repetitions of evaluation.

Limitations and future directions noted by the authors

• Single requirement: Appendix F states that the current benchmark focuses on decisions about one selected requirement; extending to multiple interacting requirements, as well as more diverse user behaviors, remains future work.
• Rejected changes: Appendix F states that average Retained accuracy is close to that of SFT, and handling rejected changes remains difficult and is a direction for future methods.
• Natural conversation: Appendix F states that training uses the complete active task specification given by the benchmark; applying it to natural conversation requires reconstructing active requirements from messages and decisions, along with consistency checks and human review.
• Longer and more open: Section 6 lists longer, more open interactions, as well as richer forms of intent evolution, as future directions.

Experimental coverage

• Models and scale: The evaluated models are the ten in the experimental setup; the eight-model summary in Table 1 does not include the two Qwen models in Table 8. 3312 items per model, 414 source tasks, with domains Actions, Code, Database, Math.
• Conditions: Four multi-turn conditions plus four single-turn controls. Cumulative protocol 372 sources, persistent protocol 414 sources, k from 0 to 4. Figure 5 reports four models, Figure 7 reports six more.
• Evaluation: Final replies are scored by domain validators. The paper does not report LLM judging, statistical significance tests, or multiple repetitions; local generation seed is 0.
• Training comparison: RQ4 compares Base, matched SFT, and Intent-OPSD on four models. The teacher is a frozen copy of the same base model and is not used at inference time.
• User side: Multi-turn user messages are generated by a fixed Qwen3.6-27B simulator. Appendix B.3 reports human inspection of 500 dialogues and 2345 user turns.

Intent-Eval measures whether, in multi-turn dialogue, models execute only requirements that are still valid. Users can clarify, reject, or accept changes; content that is mentioned and then invalidated should not enter the answer.

• Approach: 414 source tasks are turned into Original, Neutral, Retained, Revised, with single-turn comparisons to separate task difficulty from multi-turn interference. Intent-OPSD lets a frozen teacher see complete single-turn tasks matched to decisions, and the student aligns on full conversations using token-level JS divergence, dropping the teacher at inference.
• Condition gaps: Averaged over eight models, multi-turn Original is 36.30 percentage points lower than single-turn, Neutral is another 4.43 lower; Retained and Revised are 8.06 and 5.55 lower than multi-turn Original. The domain cells in Table 1 are not all in the same direction.
• Errors and depth: Among new errors across ten models, Retained has 59.63% (446/748) containing invalid content, Revised has 30.97% (201/649). For the four models in Figure 5, at final depth, Neutral and Retained drop 8.13 and 20.77 percentage points relative to Original.
• Distillation: Across four models, Intent-OPSD overall accuracy is 51.12%, 10.81 and 3.73 percentage points higher than Base and SFT. The authors state that all 16 groups are above Base and 13 above SFT, but the Retained average is still close to SFT; Table 2 also has cells below Base.
• Authors' conclusion: The authors argue that one must distinguish what was said from what is still valid. They state that the method mitigates this problem without relying on the teacher at inference, and list longer, more open-ended intent evolution as future work.