Skip to content
Original
Hugging Face Daily Papers· arXiv:2610.19128·Indexed by this site · Originally published

CLASHarness: Evaluating Multi-Agent Conflicts in Computer Operation Tasks with a Controllable and Verifiable Generation Pipeline

CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

CLASHarness evaluated 15 models over 600 rounds, and GPT-6 Astra's High Pass Rate was 28.7%.

Questions
Multi-agent collaboration in computer use often encounters stale reports, contradictory instructions, and contention for shared resources, but existing benchmarks mostly assume consistent information. Manual tasks are hard to scale, and unconstrained generation causes narrative drift, answer leakage, and unsatisfiable scoring.
Methods
CLASHarness splits scenario generation into a macro outline and an eight-step micro-pipeline: first submitting decisive facts, then rendering visual evidence, and compiling code checks and semantic checks from reference trajectories already executed in the sandbox. Each 10-round pack shares an evolving workspace, which the authors state ensures task solvability.
Experiments and results
15 models are evaluated on 600 turns. GPT-6 Astra's High Pass Rate is 28.7%. The authors say repairing state is harder than only investigating, and closing actions fail more than 58% of the time. Gemini 3.5 Flash's Pass Rate drops from 13.5% in Low to 3.0% in High.
Limitations and future work
The authors note: semantic checks rely on LLM review without human studies; evaluation uses logs and workspace trajectories rather than live dialogue; scenarios only include conflict turns without a no-conflict baseline. Experiments cover 600 turns across 15 models, plus step budget, image ablation, and cross-generator comparisons.
Experimental setupGPT-6 Astra、Claude Opus 5 et al. · CLASHarness · Pass Rate、Mean Score et al.
Evaluated models
GPT-6 AstraClaude Opus 5Claude Opus 4.8Claude Sonnet 5GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaGemini 3.7 FlashGemini 3.6 FlashGemini 3.1 ProGemini 3.5 FlashKimi K3DeepSeek V4 ProGLM-5.2 FP8DeepSeek V4 FlashGemini 3.8 Flash
Benchmarks
CLASHarness
Metrics
Pass RateMean ScoreCo-Failure Lift
AI summaryThe paper proposes CLASHarness, a controllable framework for generating verifiable multi-agent conflicts in computer operation tasks, and provides a benchmark of 600 multi-turn tasks. The framework combines hierarchical scenario planning with an eight-step verification micro-pipeline, pre-planting decisive visual facts before rendering, and compiles code and semantic scorers from executed reference trajectories, making each task solvable by construction. The authors use it to evaluate 15 frontier models and report three types of bottlenecks: when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information while stronger models check visual evidence to verify ground truth; resolving conflicts is harder than detecting them, and proactively fixing system states significantly reduces task success rates; coordination protocols are a universal bottleneck, with even frontier models failing over 58% of the time on closing actions, often forgetting to release shared locks or hand off tasks.

The paper proposes CLASHarness, a controllable framework for generating verifiable multi-agent conflicts in computer operation tasks, and provides a benchmark of 600 multi-turn tasks. The framework combines hierarchical scenario planning with an eight-step verification micro-pipeline, pre-planting decisive visual facts before rendering, and compiles code and semantic scorers from executed reference trajectories, making each task solvable by construction. The authors use it to evaluate 15 frontier models and report three types of bottlenecks: when sub-agent outputs conflict with the environment, weaker models blindly follow misleading information while stronger models check visual evidence to verify ground truth; resolving conflicts is harder than detecting them, and proactively fixing system states significantly reduces task success rates; coordination protocols are a universal bottleneck, with even frontier models failing over 58% of the time on closing actions, often forgetting to release shared locks or hand off tasks.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing computer use benchmarks mostly assume cooperative consistency; CLASHarness generates verifiable multi-agent conflict tasks.

    Multi-agent computer use requires verifying the environment and rewriting system state under contradictory information, but lacks scalable and verifiable evaluation.

    • Scenario: The authors say VLMs are being deployed into multi-agent computer use pipelines. Peer or sub-agents often report stale state, issue contradictory instructions, or contend for shared permissions and files; agents must check others' outputs and verify ground truth in the running environment.
    • Existing gaps: The authors say existing computer use benchmarks mostly assume consistent, cooperative single-agent environments, and multi-agent evaluations are mostly limited to text-only dialogue. Manual authoring is hard to scale; unconstrained LLM generation brings narrative drift, answer leakage, ungrounded visual assets, and scoring criteria that make tasks unsolvable.
    • Design principles: The authors propose hierarchical staged synthesis, committing multimodal ground truth before rendering, and compiling hybrid checkers from executed reference trajectories, making each task solvable by construction.
    • This paper proposes: The controllable generation framework CLASHarness, plus a benchmark of 20 operational domains and 600 executable conflict turns, with diagnostics on 15 frontier models.
    • Task setup: A conflict turn contains an initial workspace, an instruction that does not reveal which claim is true, a typed action space, a decisive fact, and a mixed scoring suite. The agent is reminded that contradictions exist, must investigate multimodal evidence, and execute handoffs, config patches, or rollbacks. The paper does not use white-box, black-box, or no-box.

  2. What related research exists?

    The author places this paper on evaluations of factual and temporal contradictions in tools, files, vision, and peer dialogue.

    The author groups related work into three categories per Section 2, and places this paper on computer use evaluation under explicit factual and temporal contradictions, rather than consistent collaborative environments or text-only knowledge conflicts.

    Computer use and multi-agent benchmarks

    • Tool-augmented benchmarks: The paper lists SWE-bench (Jimenez et al., 2024), OSWorld (Xie et al., 2024b), WebArena (Zhou et al., 2024), GAIA (Mialon et al., 2024), AgentBench (Liu et al., 2024), etc., for software engineering, OS execution, and web navigation.
    • Collaboration and failure: Du et al. (2023), CAMEL (Li et al., 2023), AutoGen (Wu et al., 2023) study collaborative problem solving; Cemri et al. (2026), CooperBench (Khatua et al., 2026), Pappu et al. (2026), Silo-bench (Zhang et al., 2026), etc., discuss coordination failures.
    • Authors' distinction: The authors argue these benchmarks mostly assume internally consistent, cooperative environments; CLASHarness adds factual and temporal contradictions across tools, files, visual evidence, and peer dialogue.

    Knowledge conflicts and multimodal grounding

    • Text-only conflicts: ConflictBank (Su et al., 2024), Xie et al. (2024a), Wang et al. (2023; 2025), etc., study contradictions between model memory and retrieved text. The author places such work in text-only QA.
    • Interface navigation: Mind2Web (Deng et al., 2023), VisualWebArena (Koh et al., 2024) evaluate GUI navigation. The author says there are no conflict signals in them.
    • Authors' distinction: CLASHarness pre-commits visual ground truth on dashboards, terminal outputs, and system diagrams, and adds stale evidence and expired authorizations.

    Test-time compute and reasoning dynamics

    • Formal tasks: The author relays Guo et al. (2025), s1 (Muennighoff et al., 2025), Snell et al. (2025): increasing test-time reasoning compute improves performance on math and formal tasks.
    • Action loops: Chen et al. (2025), Cuadron et al. (2025) discuss overthinking and repeated actions in open-ended agent environments.
    • Use in this paper: The author says CLASHarness is used to observe how reasoning effort affects multi-turn execution under operational conflicts.

    Baselines and benchmark The paper does not list an external method as a unified baseline. Comparisons occur on CLASHarness's 600 turns; difficulty is split into Easy, Mid, Hard by reference trajectory length. Generator comparisons use the same 20 seeds, synthesized by Gemini 3.6 Flash and Claude Sonnet 5 respectively.

    The author argues this paper evaluates factual and temporal contradictions across tools, files, visual evidence, and peer dialogue, rather than consistent collaborative environments or static single-modal QA.

  3. How does the paper address this problem?

    CLASHarness generates conflict rounds hierarchically, first submitting visual facts, then compiling scorers from reference trajectories.

    CLASHarness splits the generation of multi-agent conflict tasks into macro planning, turn-by-turn verified execution, and post-generation exclusivity checks, avoiding a single prompt that simultaneously writes the narrative, workspace, and scorer.

    Problem setup

    • Turn definition: A conflict turn consists of an initial workspace W0, an instruction I, an action space T, a decisive fact v★, and a mixed scoring suite Φ. W0 contains OS state, file tree, asynchronous communication logs, and rendered visual evidence.
    • Agent-visible information: At inference time, only W0, I, and T are given, with a reminder that contradictions exist. I frames the peer's competing claims but does not indicate which is factually grounded.
    • Completion: The agent must find v★ and execute state changes that write to the audit log Elog, such as protocol handoffs, config patches, or rollbacks. Scoring looks at both the final workspace and Elog.
    • Package structure: Turns form 10-turn packages sharing an evolving workspace, generated from structured scenario seeds.

    Three-tier generation pipeline

    • Tier 1 (Stages 1–3): First build a consistent multi-agent world. Stage 1 writes the global timeline and shared resources, Stage 2 defines roles, channels, and permissions, Stage 3 initializes repositories, configs, logs, and interfaces. The paper writes this tier as 8 LLM calls.
    • Tier 2 (Stages 4–5): Each package has R=10 rounds. Every 3–4 rounds (Ps in [3, 4]), one call expands a stage milestone. Each round then goes through eight steps: task instruction, peer dialogue, file updates, visual assets, committing v★ with leak scanning, step-by-step plan, sandbox execution of a reference trajectory, and compiling checks from the trajectory without further model calls.
    • State continuation: Workspace modifications from round r carry into round r+1. Failures are confined to a single substep, with bounded retries using diagnostic feedback, not whole-package discards.
    • Tier 3 (Stages 6–8): Stage 6 checks the instruction, reference trajectory, and scoring; Stage 7 embeds v★ via HTML/Canvas specs; Stage 8 renders screenshots in a headless browser and checks that v★ does not appear in text shortcuts for vision-required tasks. Output is scenario.json.

    Controllability, plugins, and self-correction

    • Seeds: Conflict types, interaction protocols, tool families, and evidence modalities are specified by declarative seeds. Appendix §B.4 describes user-modifiable parameters and synthesis of new domains.
    • Plugins: Domain mechanics are split into 23 plugins, 16 conflict mechanisms and 7 challenge plugins (Table C1). Valid actions are commands whose declared effects intersect non-trivially with plugin effects; new conflicts only require registering effects, not changing the scoring engine.
    • Two-level correction: Format, syntax, and path errors are fixed directly by deterministic code with 0 model calls. Logical inconsistencies only retry the affected substep.
    • Call count: The paper writes the total calls as 8 + ceil(R/Ps) + K times R + A, with K around 7–8 and A the number of assets; a 10-round package averages 95 calls (Appendix §B.2).

    Scoring and visual exclusivity

    • Check sources: Exact code checks verify identifiers, and semantic criteria only cover file modifications within specified scope. Checks are compiled from executed reference trajectories, which the authors say removes unreachable scoring.
    • Visual facts: v★ is committed at substep 5.5, before rendering. Visual assets are provided as separate images. Of 600 turns, 346 mark opening visual assets as required, with text leak scanning.
    • Reference budget: Reference trajectories average 10.28 steps, with an informed step budget b=20; length ranges from 7–18 steps. Appendix Table B1 lists 14 categories of generation defects M1–M14 and their fixes.

  4. What experiments did the paper conduct?

    GPT-6 Astra's High Pass Rate is 28.7%; the authors state that fixing the state makes conflicts harder to detect.

    Experimental setup

    • Models tested: The 15 models in Table 1 are Claude Opus 5, Claude Opus 4.8, Claude Sonnet 5, GPT-6 Astra, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, Gemini 3.7 Flash, Gemini 3.6 Flash, Gemini 3.1 Pro, Gemini 3.5 Flash, Kimi K3, DeepSeek V4 Pro, GLM-5.2 FP8, DeepSeek V4 Flash. The step experiment additionally includes Gemini 3.8 Flash.
    • Budget and inputs: Ten 20-step API models have Low/Medium/High; GPT-6 Astra and four open-source models have High only. GPT-6 Astra uses an extended budget, averaging 26.1 steps, and the appendix also writes a 40-step budget. The three models marked with † use OCR text of visual assets. Most runs are r01; Gemini 3.7 Flash uses r02.
    • Data: 20 seeds, 3 variants per seed, 60 packs, 600 rounds. There are 3,760 checks, of which 3,284 are required; 7,970 visual assets, and 346 rounds require looking at images.
    • Difficulty: By reference trajectory length, Easy is at most 9 steps, Mid is 10–11 steps, and Hard is at least 12 steps. The reference trajectory averages 10.28 steps, with a median of 10.
    • Metrics: Pass Rate is the proportion of tasks satisfying all required checks; Mean Score is the average proportion of satisfied checks per round. Table 1 gives mean ± bootstrap standard error, with 10,000 seed-clustered resamples.
    • Review: Of the 3,284 checks, 1,147 (34.9%) are code checks and 2,137 (65.1%) are semantic checks, with 3-sample majority voting and 97.5% agreement across 19,419 judgments. The main text §4 does not name the default reviewer; Appendix F.5 states the corpus-standard reviewer is Gemini 3.6 Flash, with re-review by Claude Sonnet 5 and Claude Opus 5. F3–F6 use only code checks.

    Main results

    The table below shows High, All, and Hard from Table 1. Pass Rate and Mean Score are in %, with ± standard error. GPT-6 Astra is not on a 20-step budget.

    Swipe horizontally to read the table

    ModelAll Pass RateAll Mean ScoreHard Pass Rate
    GPT-6 Astra (extended budget)28.7±1.966.8±1.218.6±3.3
    Claude Opus 525.2±1.864.2±1.213.6±2.9
    GPT-5.6 Sol24.5±1.763.3±1.213.6±2.9
    Kimi K321.8±1.755.9±1.412.1±2.7
    Gemini 3.6 Flash18.5±1.658.6±1.33.6±1.6
    GLM-5.2 FP8 (OCR)14.7±1.448.7±1.35.7±2.0
    DeepSeek V4 Flash (OCR)8.5±1.137.5±1.34.3±1.7
    Gemini 3.5 Flash3.0±0.725.9±1.12.1±1.2

    Omitted High rows are in Table 1: Claude Opus 4.8, Claude Sonnet 5, Gemini 3.7 Flash, GPT-5.6 Terra, GPT-5.6 Luna, DeepSeek V4 Pro (OCR), Gemini 3.1 Pro.

    • Authors' interpretation: The authors state that a low pass rate does not mean all individual checks fail; frontier models' individual checks fall in the 55%–67% range they give. GPT-6 Astra hits the execution cap 1.0% of the time, but often stops early without releasing shared locks or completing sub-agent handoffs.
    • Reasoning effort: The authors state that gains from Low to High become smaller or worse. Claude Opus 5 increases by 1.7 percentage points, GPT-5.6 Luna by 0.3 percentage points. Gemini 3.6 Flash and Gemini 3.7 Flash peak at Medium, at 20.2% and 21.5%, respectively.
    • Negative examples: Gemini 3.5 Flash's Low Pass Rate is 13.5%, dropping to the value in the table at High. The authors state its internal reasoning loop consumes tool actions within the step budget.

    Failure modes and closing protocols

    • What was tested: §5.2 divides the 3,284 checks into visual grounding F3, content accuracy (F1 messages, F2 files, F4 evidence citations), and protocol compliance (F5 closing actions, F6 time windows).
    • Results: The authors state that strong models are 31–65 percentage points higher than weak baselines, and F4 still has a 15.2 percentage point gap among top models. F5 failure rates are 58%–72%, and F6 83%–97%; the denominators in Figure D1 are 515 and 30, respectively. GPT-6 Astra and Claude Opus 5 fail closing actions more than 58% of the time.
    • Authors' interpretation: The authors state that coordination protocols are a general bottleneck. After Bonferroni correction, conflict-category residuals have p no smaller than 0.094, which the authors take as no domain specialization.

    Resolving conflicts vs. investigating only

    • What was tested: Table 3 compares Resolve, which must change state, with Investigate, which only investigates, on High, 15 models, and 9,000 rounds. 12 seeds are the treatment group, and gating is inactive for 8 seeds, serving as controls.
    • Results: The treatment group drops from 28.1% to 11.0% (-17.1 percentage points), the control from 19.3% to 20.4% (+1.1 percentage points), giving a DiD of -18.2 percentage points, 95% CI [-29.1, -7.2], exact p=.052. The dose response is investigate-only 28.1%, advance protocol 12.3%, complete protocol 4.4% (Table D5).
    • Authors' interpretation: The Mean Score gap is only -7.9 percentage points (p=.252). The authors state that agents easily detect inconsistencies, but revoking, withdrawing permissions, or handing over locks lowers task success. The authors state that p=.052 is at the boundary of statistical significance for this design, and the DiD is conservative: gated rounds include declared plugin effects 41.7% of the time, versus 25.7% for ungated.

    Budget, vision, and generators

    • Step budget: In Table F2, as b increases from 20 to 40, Claude Opus 5 goes from 25.2% to 27.0%, with a cap-hit rate of 0.5% at b=20. Gemini 3.5 Flash goes from 3.0% to 13.0%, with 89.2% of rounds hitting the cap at b=20. Gemini 3.8 Flash goes from 12.8% (20 steps) to 29.2% (40 steps), and 30.7% at 60 steps; §6 writes p=.0098 after this.
    • Removing images: In Table F2, Claude Opus 5 drops from 25.2% to 4.2% across all 600 rounds. On the 254 rounds without mandatory image checks, it drops from 24.8% to 2.8%. The authors state that blind agents retain only 11.1%–13.1% of their pass rate with vision, with the maximum p on the order of 2.6×10 to the -12 (the text writes p<10^-11, and after correction the maximum p is 2.6×10^-12).
    • Switching generators: The same 20 seeds are regenerated by Claude Sonnet 5. The reference trajectory mean is 12.48 steps versus 10.28 steps, with Pearson r=0.186 (Table F1). The ranking of seven models has Spearman ρ=0.96; the authors state there is no self-preference.

    Other ablations and analyses

    • On easy tasks, 11/15 models have a co-failure lift above 1.0; Table 2's Hard lift is 0.99–1.06, with GPT-6 Astra at 1.01. Appendix D.3 also writes GPT-6 Astra as 0.92, inconsistent with Table 2.
    • Reasoning tokens as a share of output: Claude Opus 5 is 8.8%, Gemini 3.5 Flash is 90.5%; the correlation with 20-step API model success is ρ=-0.697 (§5.5).
    • Cost per solved problem: GPT-5.6 Luna is $0.72, Gemini 3.5 Flash is $39.54; the authors state this is about 55 times. The ten 20-step API models at High total $11,737 and 87,493 calls.
    • Code-check and semantic-check rankings have ρ=+0.90; visual grounding and semantic conflict resolution have ρ=+0.02 (§4).
    • Flipping 479 split semantic judgments changes Pass Rate by at most 2.33 percentage points and moves ranks by at most 1 position (§C.3). Of the 45 pairs among the ten 20-step models, 21 differ after Bonferroni correction; GPT-6 Astra's +3.5 percentage points over Claude Opus 5 does not pass the corrected threshold (§D.2).
    • 14/15 models' pass rates decline as reference trajectories lengthen; the exception is DeepSeek V4 Flash, with Mid at 11.2% and Easy at 8.5% (§C.2).

  5. What are some points that can be further explored?

    The authors note that semantic review has no human studies, evaluation is based on logs rather than live dialogue, and there is no no-conflict baseline.

    Limitations and future directions noted by the authors

    • Semantic review: Semantic checks rely on LLM review, with no human study. The authors state that 3 votes and multi-reviewer cross-validation are used as mitigation (§7).
    • Non-real-time interaction: Conflict resolution is evaluated through logs and workspace trajectories, not real-time dialogue. The authors state this ensures reproducibility and corresponds to ticket and review workflows (§7).
    • No conflict-free baseline: Scenarios focus on conflict rounds and do not include a conflict-free baseline. The authors state that ablation gating is used to isolate conflict handling (§7).

    Experimental coverage

    • Main results are 15 models, 600 rounds each; ten 20-step API models have Low, Medium, High, while GPT-6 Astra and four open-source models report only High (Table 1, Table G1).
    • The benchmark includes 20 seeds, 6 conflict types, 23 plugins, 3,284 required checks, and 7,970 visual assets; 346 rounds require visual evidence (§4, Table C1).
    • The step-budget experiment includes the six models in Table F2 (b=20, 30, 40); Gemini 3.8 Flash additionally has b=60 (§F.3). The image-removal experiment uses Claude Opus 5, Claude Sonnet 5, and Gemini 3.6 Flash.
    • The cross-generator comparison is seven models, 600 rounds each, with generators Gemini 3.6 Flash and Claude Sonnet 5 (§6, Table F2). Appendix F.5 gives 9,000 High rounds from 14 20-step models to three reviewers for replay.
    • The default semantic reviewer is written as Gemini 3.6 Flash in Appendix F.5; F3, F4, F5, F6 use deterministic code checks (§4). The paper states there is no human study (§7).

  6. Summarize the main content of the paper

    CLASHarness constructs 600 solvable conflict rounds, and GPT-6 Astra's High Pass Rate is 28.7%.

    CLASHarness is a generation framework and benchmark for multi-agent conflicts in computer use, used to check whether agents can complete scoreable operations amid contradictory reports, shared locks, and visual evidence.

    • Problem: Peer or sub-agents may give stale or contradictory information and compete for files and permissions. The authors argue that manual tasks are hard to scale, and unconstrained generation can leak answers or produce unsatisfiable scoring.
    • Method: Generation is split into a shared world, staged turn-by-turn synthesis, and post-generation audit. Decisive facts are committed before rendering; code checks and semantic checks are compiled from reference trajectories executed in a sandbox. The benchmark is 20 domains, 600 turns, with workspaces persisting across turns.
    • Main results: In the High setting of Table 1, GPT-6 Astra has a Pass Rate of 28.7% and Mean Score of 66.8%; in the same table, Gemini 3.5 Flash has a Pass Rate of 3.0%. This model uses an extended step budget and should not be directly compared side-by-side with 20-step models.
    • Gating: In Table 3, 15 models have a pass rate of 11.0% when state must be changed, 28.1% when only investigating, with a DiD of -18.2 percentage points (exact p=.052). The authors say finding contradictions is easy, but repairing state is harder.
    • Two other bottlenecks: The authors say weak models blindly trust misleading text, while strong models look at visual evidence; even frontier models fail at closing actions more than 58% of the time, with common omissions being releasing shared locks and handing off to sub-agents. Gemini 3.5 Flash's Pass Rate drops from 13.5% in Low to 3.0% in High.
    • Authors' conclusion: The authors say the three bottlenecks are visual verification, state repair, and closing coordination. They also say this pluggable pipeline can serve as a diagnostic bench and as a data engine for post-training next-generation VLMs.
Read originalhuggingface.co