When subagents work in parallel: opportunities and pitfalls of dynamic concurrency in long-horizon coding tasks

Frontier coding agents dynamically spawn and coordinate sub-agents during execution, but their effectiveness, efficiency and failure characteristics on long-horizon programming tasks remain unclear.

• Setting: The authors consider long-horizon software development to involve extended code exploration, implementation, testing and verification, and a single agent has substantial limits in context, reasoning and execution. The authors view multi-agent decomposition as a path to expanding execution capacity, parallel exploration and reducing end-to-end latency.
• Limits of static concurrency: Earlier systems such as TRAE and OpenHands rely on predefined workflows, with the number, roles, division of labor and synchronization points of sub-agents specified in advance. The authors argue that this rigidity makes it hard to adapt to decomposition, dependencies and parallel opportunities that only emerge as execution unfolds.
• Evaluation gap: The authors state that existing work mostly studies single agents, or multi-agent systems with predefined protocols, and that evaluations mostly target bounded tasks whose expected output is a small patch, for example only tens of lines. Dynamic concurrency targets repository-level generation and software release; the authors state that the reference implementation or gold-change is 0.8–54.4 thousand lines across difficulty groups.
• This work: The authors compare Codex multiagent Version2, Claude Code Dynamic Workflow and Kimi Code AgentSwarm with concurrency on and off. The authors state this is the first systematic study of native dynamic concurrency, covering 354 tasks and 2124 executions, and analyze failure modes and conditions for benefit.

Section 2 of the paper divides related work into predefined collaboration and empirical studies of coding agents; the experimental comparison is the same agent's sequential mode.

Predefined collaboration

• Roles and workflows: ChatDev (Qian et al., 2024) and MetaGPT (Hong et al., 2024) organize collaboration with specialized roles and development processes. The introduction also lists TRAE and OpenHands as earlier coding agents that rely on predefined workflows.
• Task graphs and discussion: CodeR (Chen et al., 2024) coordinates issue resolution with a predefined task graph. Multiagent Debate (Du et al., 2024) and SWE-Debate (Li et al., 2026) use repeated discussion rounds, and Mixture-of-Agents (Wang et al., 2025) uses hierarchical aggregation.
• The authors' distinction: The authors consider static paradigms to orchestrate subagents according to a workflow fixed in advance; dynamic concurrency, by contrast, has the main agent decide during execution whether to spawn subagents, what to delegate, and how to converge. The paper makes only this kind of contrast with the above systems and gives no numerical comparison with them.

Failure and trajectory analysis

• Interaction and cost: A set of studies cited in the paper analyzes how coordination architectures and debate settings affect accuracy, cost and latency. The main text gives no specific numbers from these studies.
• Failure modes: TRAJEVAL represents analysis of coding agent errors and success/failure trajectories. MAST and AgentFail examine misalignment between agents, information loss and failure propagation. TrajAudit and CodeTracer represent automatic trajectory diagnosis; the paper also cites failure attribution and feedback-based recovery work alongside these.
• The authors' positioning: The authors state that existing evaluations mostly target single agents or predefined collaboration and mostly test bounded tasks; this study instead examines dynamic concurrency in frontier coding agents and evaluates effectiveness, efficiency and failure modes together on long-horizon tasks.

Experimental baselines and benchmarks

• Baseline method: Sequential mode with dynamic concurrency turned off, under the same backbone model, prompts, environment and time budget. The authors state that this mode does not create subagents.
• Benchmarks: RepoZero, ProgramBench, NL2Repo, LoopsBench, and the bounded control SWE-bench Verified.

The authors turn native dynamic concurrency on and off across three frontier coding agents, compare concurrent and sequential trajectories on the same tasks, and then induce failure modes and conditions for benefit from the trajectories.

Agents and switches

• Pairing: The July 2026 configuration is Codex with GPT-5.4, Claude Code with Claude Opus 5, and Kimi Code with Kimi K3. The authors consider that post-training reinforcement learning may adapt models to the scaffold's tool interfaces and that swapping models may break compatibility, so they do not change the backbone model. All three use native scaffolds and are requested at the highest reasoning effort.
• Three mechanisms: In Claude Code, the main agent generates a workflow script that arranges parallel stages, explicit dependencies and convergence, and Dynamic Workflow is launched with ultracode; sequential mode disables the Workflow, Task and Agent tools. Kimi Code uses AgentSwarm to issue a batch of subagents at once from a shared template, and the main agent waits and then receives ordered reports; the switch is swarm_mode=true or false. Codex supports direct and nested delegation, and the main agent can continue executing and exchange messages; concurrency turns on multi_agent and multi_agent_v2, and sequential mode disables both.
• Meaning of off: The authors state that with dynamic concurrency off, no subagents are created. Both modes share the backbone model, prompts, environment and time budget. The methods section gives no unified time limit applicable to all tasks.

Tasks and sampling

• Benchmarks: RepoZero contains Py2JS (rebuilding Python packages in JavaScript) and C2Rust (rebuilding C interfaces in Rust); ProgramBench rebuilds command-line programs from a reference executable and documentation; NL2Repo builds a repository from a natural-language specification and an empty workspace; LoopsBench is interdependent continuous development; SWE-bench Verified resolves a single issue in an existing repository and serves as the bounded control.
• Sampling: Long-horizon benchmarks are stratified-randomly sampled at 25% by original difficulty group, and SWE-bench Verified at 20%. The cost example the authors give is that one full run of LoopsBench can cost $11,354.
• Scale: The Table 1 sample is Py2JS 100, C2Rust 50, NL2Repo 26, ProgramBench 50, LoopsBench 28, SWE-bench Verified 100, totaling 354 tasks and 2124 execution units. Recorded tokens correspond to $20,780.02; the main text separately writes 11.2 billion recorded tokens. The authors state that reference implementations or gold changes for the different difficulty groups of the long-horizon benchmarks are 0.8–54.4 thousand lines, and map the 26.8K lines in Figure 3 to LoopsBench.

Process measures

• Delegation: Each subagent context counts once per execution. Intersecting time windows count as overlap. The creation count can exceed the peak number existing simultaneously because of multiple rounds of delegation.
• Overlap ratio: Trajectories are normalized by the first and last timestamps of the main agent or any subagent. The ratio is the proportion of time during which at least two subagents are active simultaneously, excluding the main agent; if there is no such overlap it is recorded as 0.
• Determination: Official Task pass is the proportion of tasks fully resolved, and Test pass is the mean of the per-task proportion of tests passed, given by the official testers. Figure 4 additionally defines success on ProgramBench as test pass greater than 0, because neither mode fully resolved the sample tasks; other benchmarks still use full resolution.

Annotation and benefit screening

• Codebook: Two annotators first openly code 50 trajectories to form three levels: main class, subclass, and specific pattern. Functional failure means ownership, information exchange, isolation, observability or acceptance was violated; performance failure means coordination was ineffective under the existing budget.
• Formal annotation: Of three annotators, each trajectory is independently annotated by two, with multiple selections allowed. Disagreements are first discussed, and unresolved ones are adjudicated by a senior author. The object is all 1062 concurrent trajectories, compared against the sequential trajectories, artifacts and official tests for the same agent and task.
• Benefit pairs: Inclusion requires that concurrent Test pass be higher, or that concurrency fully resolve a task the sequential mode did not resolve, or that completion time be reduced by at least 60 seconds and at least 20% without lowering Test pass.

Experimental setup

• Tested configurations: Codex with GPT-5.4, Claude Code with Claude Opus 5, Kimi Code with Kimi K3. The configurations are from July 2026, native scaffolds, highest reasoning effort. Results are reported by agent, with each agent paired with only one backbone model.
• Tasks: 354. SWE-bench Verified 100 (easy 39, medium 52, hard 9, total 500); Py2JS 100 (33/32/35, total 400); C2Rust 50 (15/18/17, total 200); NL2Repo 26 (6/12/8, total 104); ProgramBench 50 (7/36/7, total 200); LoopsBench 28 (4/11/13, total 112). Table 1.
• Modes: Each task and each agent is run once in concurrent and once in sequential mode, for 2124 execution units in total. Both modes share the model, prompts, environment and time budget; sequential mode creates no subagents.
• Metrics: Task pass, Test pass, Shell calls (main agent plus all subagents combined), Time (minutes), Tokens (millions).
• Determination: Official testers for each benchmark. The paper uses no LLM judge and reports no number of repetitions for the main comparison. Recorded tokens correspond to $20,780.02; the main text writes 11.2 billion recorded tokens.
• Tests: After pooling short-horizon and long-horizon, exact McNemar tests with Holm correction are performed for the three agents and compared with α=0.05. Delegation and difficulty use two-sided Spearman permutation tests, with Holm correction across 30 comparisons.

Main results

According to Table 2, Task pass (%), with concurrent/sequential in each cell.

Swipe horizontally to read the table

Benchmark	Codex	Claude Code	Kimi Code
SWE-bench Verified	71.0/71.0	59.0/83.0	59.0/77.0
RepoZero	40.7/40.0	26.7/42.7	38.7/37.3
NL2Repo	57.7/34.6	65.4/57.7	46.2/57.7
ProgramBench	0.0/0.0	0.0/0.0	0.0/0.0
LoopsBench	25.0/21.4	21.4/7.1	25.0/25.0

• The authors' interpretation: The authors state that dynamic concurrency does not reliably improve end-to-end success rates; the resolution-rate effect in the introduction ranges from a decrease of 11.9 percentage points to an increase of 2.3 percentage points. On LoopsBench, the Task pass increase for Codex and Claude Code is at most 14.3 percentage points, and Kimi Code is unchanged; on that benchmark Test pass rises for all three, by at most 6.0 percentage points. SWE-bench Verified shows no improvement, and Claude Code decreases by 24.0 percentage points. Other long-horizon benchmarks are inconsistent, as seen in the table for Claude Code on RepoZero and Kimi Code on NL2Repo.
• Statistical tests: After pooling the two task types, the authors report that the overall decreases for Claude Code and Kimi Code are statistically significant at α=0.05, with adjusted p of 5.08348×10−7 and 0.0132176 respectively; the increase for Codex is not significant, p=0.229481.
• Relation to scale and internal offset: After equal-weight averaging across the three agents, the Test pass difference of concurrent minus sequential, ordered by implementation scale, is RepoZero −8.29, NL2Repo −0.04, ProgramBench +1.24, LoopsBench +4.31 percentage points. The authors consider that larger expected implementation scale is associated with a larger average Test pass advantage. Within Codex on RepoZero, Py2JS Task/Test pass decreases by 2.0/1.7 percentage points and C2Rust increases by 6.0/2.4 percentage points, resolving 2 fewer Py2JS and 3 more C2Rust, a net increase of 1 across 150 tasks.

Resource consumption and time

According to Table 2, Tokens (millions), with concurrent/sequential in each cell.

Swipe horizontally to read the table

Benchmark	Codex	Claude Code	Kimi Code
SWE-bench Verified	3.37/1.34	2.14/1.02	1.01/0.98
RepoZero	2.47/0.89	4.03/2.42	1.33/0.82
NL2Repo	8.00/2.87	8.55/6.53	2.70/2.21
ProgramBench	27.05/9.19	6.02/6.18	11.64/6.05
LoopsBench	51.65/11.45	21.73/15.52	41.49/11.29

• Calls and tokens: The authors state that average shell calls rise for every agent and every benchmark, and tokens reach 1.41–3.31 times sequential mode. In the table above, the Claude Code ProgramBench cell is not higher in concurrent mode. For example, Claude Code's Shell calls on LoopsBench are 791.6 versus 107.4. Amortized by resolved tasks, all three rise; the Codex figures the paper gives are 24.12 million concurrent and 7.69 million sequential.
• Time: The authors state that average end-to-end time rises overall, in 14 of the 15 agent–benchmark combinations. In Table 2, Kimi Code on ProgramBench is 39.9 minutes versus 40.1 minutes; the authors do not name which cell is the exception. Claude Code on SWE-bench Verified is 27.8 minutes versus 17.5 minutes while resolving fewer tasks.
• Minority speedups: Of the 366 cases resolved by both modes, 102 (27.9%) are faster in concurrent mode, of which 46 (12.6%) are at least 20% faster. The authors consider that concurrency can still speed things up when coordination is done well.

Delegation scale and temporal overlap

• Variation with difficulty: After equal-weighting all concurrent executions, the average creation count goes from 6.69 for easy to 7.53 for medium to 15.84 for hard, and the average peak from 3.66 to 4.61 to 6.70. On LoopsBench, the average creation count across the three agents goes from 16.25 for easy to 68.18 for hard, and the peak from 6.00 to 21.77. LoopsBench accounts for 14.6% of hard tasks but 62.9% of the subagents created on hard tasks. The authors state that Claude Code on RepoZero does not necessarily create or maintain more subagents for harder tasks.
• Agent differences: Codex's average peak across task groups ranges from 1.21 to 3.00. On hard LoopsBench, Claude Code averages 133.69 creations and a peak of 25.46; Kimi Code averages 67.00 creations and a peak of 37.46. After inspecting the implementations, the authors state that Codex can leave part of the implementation with the main agent, Claude Code can create new groups in later stages, and Kimi Code can issue large batches at once.
• Temporal structure: In executions that created subagents, the median trajectory position of the first spawn is Codex 4.0%, Claude Code 22.1%, Kimi Code 26.0%; the median first overlap is 4.7%, 22.5%, 26.9%. The median overlap ratio across all tasks is 23.9%, 52.7%, 15.6%. The median endpoint of the last subagent is 83.3%, 100.0%, 93.6%. Of the 354 executions, the proportion that created subagents but had no overlap among them is Codex 24.3%, Claude Code 1.7%, Kimi Code 2.8%. The authors state that Codex spawns early but overlaps briefly, Claude Code spawns later but overlaps more persistently, and Kimi Code has the lowest median overlap ratio.

Failure taxonomy and benefit cases

• Taxonomy scale: All 1062 concurrent trajectories were manually annotated, Cohen's κ=0.77. Four classes, 13 subclasses and 28 specific patterns appear in 650 trajectories, for 804 instances in total; one trajectory may contain multiple patterns. Table 3: task orchestration 229 (28.5%), execution governance 233 (29.0%), global context 75 (9.3%), shared state and merging 267 (33.2%).
• High-frequency patterns: Load imbalance 166 (20.6%), of which Fan-out Budget Exhaustion is 154 (19.2%) and accounts for 39.8% of Claude Code pattern instances. Errors in start, convergence or result timing 166 (20.6%), of which premature termination is 128 (15.9%). Concurrent writes 213 (26.5%), accounting for 62.9% of Codex pattern instances; deliverable overwriting 120 (14.9%). In the seqtk example of Figure 2, Kimi Code issues a batch of 20 subagents, the task budget is 40 minutes, and 92/440 tests pass; the authors use it to illustrate that waiting for the whole batch can crowd out integration.
• Conditions for benefit: 131 trajectory pairs satisfy higher Test pass, unique full resolution, or at least 60 seconds and 20% faster without lowering Test pass. The authors summarize three cases: complementary work, such as Kimi Code's NL2Repo justext, where both modes pass all tests and wall-clock time goes from 734.116 seconds to 467.997 seconds; alternative selection, such as Codex's ProgramBench Halite, with concurrent 292/391 and sequential 0/391; independent verification, such as Codex's LoopsBench security experiment task and NL2Repo trimming, where concurrent passes all checks or all tests and sequential mode does not. The authors state that benefits appear when the main agent uses parallel results for integration, selection or correction.

Other ablations and analyses

• ProgramBench delegation: Claude Code's average creation count from easy to hard is 10.71, 22.08, 31.29, and the peak is 8.00, 11.56, 13.86 (Section 5.1).
• Difficulty association: Of the 30 Spearman comparisons, the authors report as significant Claude Code's creation count on ProgramBench (ρ=0.470, adjusted p=0.015) and Codex's creation count on SWE-bench Verified (ρ=0.316, adjusted p=0.042). The authors state that task structure and coordination mechanisms also shape delegation.
• Unique successes (Figure 4): Claude Code has 4 tasks on LoopsBench resolved only in concurrent mode (sample 28), with no unique successes in sequential mode; on SWE-bench Verified it has 5 resolved only in concurrent mode (sample 100). Codex has 9 unique to each mode on that benchmark, with the same aggregate value. The figure counts success on ProgramBench as test pass greater than 0.
• Test pass (Table 2, concurrent/sequential): ProgramBench is Codex 58.3/53.5, Claude Code 0.5/1.8, Kimi Code 1.7/1.5; LoopsBench is Codex 58.1/52.1, Claude Code 48.7/45.8, Kimi Code 66.6/62.5.

Limitations and future directions noted by the authors

• External validity (Section 8.2): The authors write that the findings are limited to the agents and benchmarks tested. Their response in the paper is to select frontier coding agents and their native models, and to conduct an evaluation costing more than 20K USD.
• Randomness and repetition (Section 8.2): The authors write that execution is stochastic and that the cost of long-horizon evaluation limited large-scale repeated runs. The authors consider it unlikely that run-to-run randomness would substantially change the overall trends, based on the fact that in this experiment Codex's sequential Task pass and Test pass on LoopsBench are close to that benchmark's original source's report for Codex with GPT-5.4, and the original source states that overall trends are stable under repeated attempts.
• Manual annotation (Section 8.2): The authors write that manual analysis may introduce subjective judgment. Their approach in the paper is a 50-item pilot codebook, independent annotation by two of three people, recording of rationales, discussion of disagreements, and adjudication of unresolved disagreements by a senior author.
• Open questions (Section 8.1): The authors list four categories of follow-up questions: task decomposition, ownership allocation and load balancing; detecting stalled or failing sub-agents, reclaiming unfinished work, and deciding when to collect results or terminate; passing requirements, interface contracts and reusable progress between isolated contexts; and isolation, synchronization and conflict detection for shared files and environments.

Experimental coverage

• Tested are Codex, Claude Code and Kimi Code, with GPT-5.4, Claude Opus 5 and Kimi K3 respectively; configurations were selected in July 2026, using the highest reasoning effort (Section 3.2).
• Benchmarks are RepoZero, ProgramBench, NL2Repo, LoopsBench and SWE-bench Verified; a stratified sample of 354 tasks and 2124 executions (Table 1, Section 3.3).
• Effectiveness is judged by each benchmark's official tester. The paper reports no LLM judge, and reports no number of repetitions per execution unit for the main comparison (Section 3.4, Section 8.2).
• Failure analysis covers all 1062 concurrency trajectories, each annotated by two people, Cohen's κ=0.77; benefit analysis uses 131 pairs of trajectories (Section 6.1, Section 7).
• The experimental control is sequential mode with dynamic concurrency turned off. The paper discusses static systems such as ChatDev, MetaGPT and CodeR in Section 2, and reports no experimental comparison with these systems.

Three frontier coding agents are used to test a kind of parallelism decided only during execution: the main agent itself judges whether to spawn subagents, what to delegate, and how to converge. The questions to answer are whether success rates improve on long-horizon programming tasks, how much resources increase, and under what coordination conditions failures and benefits each appear.

• Protocol: Codex with GPT-5.4, Claude Code with Claude Opus 5, Kimi Code with Kimi K3. The 354 tasks come from RepoZero, ProgramBench, NL2Repo, LoopsBench and SWE-bench Verified, with each task run once in each of the two modes, for 2124 executions in total; sequential mode creates no subagents.
• Success rates: The authors state that the resolution-rate effect ranges from a decrease of 11.9 percentage points to an increase of 2.3 percentage points. On LoopsBench the Task pass increase is at most 14.3 percentage points; on SWE-bench Verified Claude Code decreases by 24.0 percentage points. After pooling short-horizon and long-horizon, the authors report that the decreases for Claude Code and Kimi Code are statistically significant at α=0.05, and the increase for Codex is not significant.
• Resources: The authors state that average tokens reach 1.41–3.31 times sequential mode. Codex records 24.12 million tokens per resolved task, versus 7.69 million in sequential mode. In 14 of the 15 agent–benchmark combinations average time is higher; of the 366 cases resolved by both modes, 12.6% are at least 20% faster.
• Failures: Harder long-horizon tasks create more subagents, but the three agents differ in spawn timing and overlap duration. Among the 804 failure instances, shared state and merging accounts for 33.2%, execution governance 29.0%, task orchestration 28.5%, and global context 9.3%.
• The authors' conclusion: Benefits appear when there is complementary division of labor, selection among alternative implementations, or subagents independently verifying and correcting the main agent. The authors recommend deciding per task whether to turn on concurrency, and within the execution budget prioritizing verifiable convergence and delivery.