Agentic on-policy distillation implicitly reshapes LLMs' ReAct patterns
Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns
The paper studied how on-policy distillation (OPD) reshapes the ReAct interaction patterns of language model agents across three domains—math, ALFWorld and τ²-bench—covering reasoning, actions and observation acquisition. Results showed that general reasoning supervision reduced the tool-call ratio on math tasks from 70.4% to 14.4% while raising accuracy from 16.7% to 38.0%; code interpreter supervision improved household action and retail query performance. ALFWorld supervision, however, dropped τ²-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violated user information dependencies. The authors proposed a selective OPD that uses only source-domain training trajectories, retaining the top 95% of positions based on changes in teacher preference before and after domain-specific RL, raising τ² accuracy from 11.5% to 29.9% at the cost of impaired source-domain learning.