Skip to content

SafeRoute-MAS: Coverage-Aware and Risk-Calibrated Message Routing for Multi-Agent Scientific Reasoning

SafeRoute-MAS: Coverage-Aware and Risk-Calibrated Message Routing for Multi-Agent Scientific Reasoning

arXiv· arXiv:2611.13454· Zhiming Xue, Jianan Liu, Weiran Yan, Aicheng Xie, Jing YangCollege of Engineering, Northeastern University; Olin Business School, Washington University in St. LouisIndexed by this site · 6 days ago· Originally publishedDefenses

Quick read

Other

Based on the paper PDF (AI-generated); refer to the original

The authors propose the SafeRoute-MAS multi-agent message routing method, achieving +1.80 percentage points in accuracy and removing 12.76% of tokens on 5,328 cached committees, but endpoint review did not confirm the gains.

Questions
In multi-agent scientific reasoning, more messages are not necessarily better: misleading messages may carry wrong answers, and redundant messages consume context. The authors ask: can a router decide, using only information visible before aggregation, whether to keep all messages or delete one, without harming accuracy?
Methods
SafeRoute-MAS trains a three-class message risk model on TF–IDF text features plus committee context features, scores by utility and passes it through a calibrated selective gate, executing Prune-One only when the frozen threshold is exceeded, otherwise Keep-All; it also proposes a numerical qualification procedure for the target backend, reverting to Keep-All if any gate is not satisfied.
Experiments and results
On 5,328 cached committees, SafeRoute-MAS accuracy +1.80 percentage points, 12.76% tokens deleted, risk UCL 3.61%; however, in 4,580 controlled aggregations on the gpt-oss-120b public endpoint, the sample estimate was −0.09 points and the design-weighted estimate +0.94 points (CI includes 0), and the selector advantage was not established, so the qualification review process rejected that endpoint.
Limitations and future work
The authors note: the endpoint experiment cannot separate backend drift from sampling variation, and temperature 0 calls have no seed control; the qualification procedure was formalized after seeing the endpoint results, the application is retrospective, and prospective validation is future work. Gains are concentrated in LAB-Bench and incomplete committees, and the non-LAB gain CI includes 0.
Experimental setupgpt-oss-120b (openai/gpt-oss-120b) · SciAgentTrace v2.1.0、JEEBench et al. · accuracy (∆ pp)、coverage et al.
Evaluated models
gpt-oss-120b (openai/gpt-oss-120b)
Benchmarks
SciAgentTrace v2.1.0JEEBenchLAB-BenchMaScQAOmni-MATH-2SciBench
Metrics
accuracy (∆ pp)coveragetoken reductionhelpful-prune risk UCLrecoveries/corruptions
AI summarySafeRoute-MAS is a coverage-aware, risk-calibrated message routing method for multi-agent scientific reasoning that decides whether to keep all expert messages or delete one. It improves by 1.80 percentage points and prunes 12.76% of message tokens across 5,328 committees; on gpt-oss-120b, a weighted equal-token comparison did not establish an advantage, and only 2/11 of the preset margins passed the support/risk rule. The authors propose a numerical eligibility determination with a Keep-All fallback, which is a retrospective application.

SafeRoute-MAS is a coverage-aware, risk-calibrated message routing method for multi-agent scientific reasoning that decides whether to keep all expert messages or delete one. It improves by 1.80 percentage points and prunes 12.76% of message tokens across 5,328 committees; on gpt-oss-120b, a weighted equal-token comparison did not establish an advantage, and only 2/11 of the preset margins passed the support/risk rule. The authors propose a numerical eligibility determination with a Keep-All fallback, which is a retrospective application.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Multi-agent message routing: deciding which message to delete using only pre-integration information, and whether this can save tokens without harming accuracy.

    Core question: in a multi-agent committee, can a router decide which message to ignore without using the gold answer, preserving accuracy while saving context tokens.

    • Setting and motivation: LLM systems increasingly use committees of expert agents to solve hard problems, and debate and round-table consensus can provide complementary evidence (citing [1], [2]). But the authors argue that more communication is not always beneficial: a plausible but misleading message can turn a correct answer into a wrong one, and redundant messages consume scarce context.
    • Shortcomings of existing methods: communication pruning can reduce cost (citing [3]), but the authors note that a deployed router cannot use the gold correctness of proposals, nor should it sacrifice minority evidence merely because it disagrees with the majority. AgentPrune and MessageSieve require actions beyond the cache and online system re-execution.
    • Core resource: SciAgentTrace [4] provides cached full-committee results and leave-one-out (LOO) results under the DHD controlled replay protocol [5], so that every "delete message i" counterfactual action has an exact label without re-running the system.
    • The paper proposes: SafeRoute-MAS—a coverage-aware, risk-calibrated message routing procedure that uses only pre-integration visible information to choose Keep-All or Prune-One; plus a numerical qualification procedure for the target backend, reverting to Keep-All if any gate is not satisfied.
    • Threat model: this is not an attack/defense paper and has no adversarial threat model; its risk definition is deployment risk—the router mistakenly deleting a "helpful" message when it cannot see the gold answer (the answer goes from correct to wrong after deletion), measured directly on the selected action as Pr(e=+1 | action), with a tolerance line of 95% UCL ≤ 5%.
  2. What related research exists?

    Related work falls into four groups: multi-agent debate, system-level routing, communication pruning, and calibration and selective classification.

    Multi-agent debate and consensus

    • Du et al. [1] (ICML 2024)—multi-agent debate improves factuality and reasoning; Chen et al.'s ReConcile [2] (ACL 2024)—round-table discussion improves reasoning through multi-LLM consensus. The paper cites them to show that committees can provide complementary evidence.

    System-level routing

    • MasRouter [6] (ACL 2025)—selects collaboration mode, roles, and models at the system level. The paper notes the difference: SafeRoute selects one already-realized message within a fixed committee rather than configuring the system.

    Communication pruning

    • Cut the crap [3] (ICLR 2025)—an economical communication pipeline (pruning topology); Optima [7] (Findings of ACL 2025)—training efficient dialogue; MessageSieve [8] (EMNLP 2026)—runtime message pruning; Zhang et al. [9] (EMNLP 2026)—note that random pruning may already preserve accuracy.
    • The paper notes that AgentPrune and MessageSieve require actions beyond the released cache and online system re-execution; SafeRoute instead learns a single-message-level action under empirical risk control, without using gold correctness, and evaluates with corresponding cached replay.

    Calibration and selective prediction

    • Guo et al. [10] (ICML 2017, neural network calibration) and Geifman & El-Yaniv [11] (NeurIPS 2017, selective classification)—provide motivation for abstention; the paper states that its UCL is not a guarantee about future distributions.

    Baselines and benchmarks: the offline baselines are Keep-All, random prune-one (100 seeds), lowest self-confidence, lowest completeness, longest message, and minority-answer pruning (all thresholded by validation coverage); proposal-correctness and oracle LOO are diagnostic rows. The data use the gpt-oss-120b LOO slice of SciAgentTrace v2.1.0, covering five benchmarks: JEEBench, LAB-Bench, MaScQA, Omni-MATH-2, and SciBench.

    The authors position this paper as leakage-safe, coverage-aware keep/prune on real counterfactual actions, with explicit calibration and abstention, and emphasize transparent retrospective diagnostics rather than a deployment claim.

  3. How does the paper address this problem?

    A three-class message risk model plus calibrated selective gating: only when above the threshold is one message deleted, otherwise all are kept; there is also a target backend qualification review process.

    Overall approach: SafeRoute-MAS models the deletability of each expert message as a three-class problem (deletion effect harmful/neutral/beneficial), predicts it using features visible before integration, and executes Prune-One only when the utility score passes a threshold, otherwise Keep-All; it is paired with a numerical target-backend qualification procedure that decides whether deployment is allowed.

    Problem formalization

    • For question q, the cached full result of committee Cq = {m1, ..., mK} is yqall ∈ {0,1}, and the cached result of deleting message i is yq,−i. The signed effect is e_{qi} = y_q^{all} - y_{q,-i} \in \{-1, 0, +1\typerightarrow}, where −1 is harmful (deletion instead restores the answer), 0 is neutral, and +1 is beneficial (deletion causes the answer to become wrong). These results serve only as labels and evaluation fields and do not enter the model.
    • The candidate deletion utility is Ui = ph(i) − λ p+(i) + γ si, where si is the visible character context share, λ = 2, and the main-accuracy operating point is γ = 0; the character share requires no tokenizer at decision time, and all reported savings use the official token. The context-weighted sensitivity analysis uses γ = 0.25.
    • For i* = arg max Ui, deletion occurs only if its score exceeds threshold τ and it passes the frozen K/diversity abstention rule; otherwise all are kept. The core risk is Pr(eqi = +1 | action), measured directly on the selected action.

    Message encoding and committee context

    • The encoding concatenates role name/description, reasoning lens, proposed answer, reasoning direction, key ideas, key assumptions, items to check, and self-reported confidence; text is represented with TF–IDF word 1–2 grams.
    • The six structured fields each have an independent observation bit, and missing fields are not treated as semantically empty text; numerical features include parse status, confidence parsed from visible language, character/word counts, and field completeness. The gold answer, proposal correctness, reviewer rationales, integrator output, and effect labels do not enter the model.
    • Context features include normalized answer agreement, unique/majority status, reasoning-lens frequency, relative confidence and length, context share, and committee completeness/entropy summaries; the main model includes benchmark identity features, with a no-benchmark ablation and LOBO evaluation.

    Calibrated selective gating

    • A class-balanced averaged logistic SGD predicts the three effect classes; message rows are weighted 1/K so that each committee has equal training weight. A sigmoid calibration is fit on a validation subset with disjoint questions, and (τ, m) is selected on the remaining validation questions over a frozen grid of thresholds and diversity margins.
    • Validation requires at least 5% action coverage and at most 3% observed beneficial prunes; the test criterion is a 95% question-cluster bootstrap UCL ≤ 5%, with at least 100 actions covering 50 acted-on questions.
    • The diversity rule always abstains when K = 2; if the selected message is the sole member of its normalized-answer cluster or reasoning-lens cluster, it must exceed τ + m. The authors state that this rule did not improve test accuracy and was retained only because it was frozen before test access.
    • The implementation uses scikit-learn: 25,000-dimensional TF–IDF 1–2 grams (min_df=3, max_df=.995), one-hot categories, and averaged class-balanced logistic SGD (L2 α = 3×10⁻⁵, 2,000 iterations). Validation is split 1:2 for calibration and gating development; 201 utility quantiles and m ∈ {0, .05, .10, .15} are searched, selecting τ = −.04558 and m = .05; the seed is fixed at 20261031.

    Target-backend qualification (Procedure 1)

    • On label-blind target calls, pruning occurs only if all five conditions hold: (i) the design-weighted full-workload accuracy 95% lower bound ≥ −1 pp; (ii) acted-on input reduction ≥ 10%; (iii) the weighted selector-minus-matched-control lower bound is non-negative; (iv) the weighted expected corruption UCL ≤ 5%; (v) each pre-specified benchmark, K, and completeness margin has at least 59 acted-on question clusters with UCL ≤ 5%.
    • This is a global intersection–union decision: any applicable failure or insufficient evidence reverts to Keep-All. The authors explicitly state that this rule was formalized after observing the endpoint results, and this paper applies it only retrospectively.
  4. What experiments did the paper conduct?

    Cache test +1.80 pp, 12.76% tokens deleted; endpoint recheck CI includes 0, qualification review rejects that endpoint.

    Experimental setup

    • Data: the gpt-oss-120b LOO slice of SciAgentTrace v2.1.0 (commit c36465d): 93,282 actions, 6,663 questions, and 26,652 committee states, covering JEEBench, LAB-Bench, MaScQA, Omni-MATH-2, and SciBench. Effect labels are 5,431 harmful, 84,306 neutral, and 3,545 beneficial, with K ∈ {2,3,4,5}.
    • Split: a benchmark-stratified 60/20/20 split by source-question hash and exact deduplication; the test set has 1,332 questions and 5,328 committees.
    • Endpoint experiment: the Groq public openai/gpt-oss-120b endpoint, with 5 calls per full/pruned prompt (temperature 0, low reasoning, 4,096 token limit), totaling 4,580 integrations, covering a stratified sample of 458 committees (158 validator actions + 300 learned-tail actions, 60 per benchmark).
    • Baselines: Keep-All, random prune-one (100 seeds), lowest self-confidence, completeness, longest, minority answer; proposal-correctness and oracle LOO are diagnostic rows using gold or LOO labels (not deployable).
    • Metrics: accuracy, recoveries/corruptions, coverage, official HARMONY_GPT_OSS tokens; risk is conditional beneficial prunes per action, with a 95% question-cluster bootstrap UCL; the cached main analysis uses 5,000 resamples for accuracy and 10,000 for risk/token.
    • Leakage control: automated tests assert that yall is constant, that there are K LOO targets, that signed effects are consistent, that splits are disjoint, and that there are no result fields; the input-validation priority protocol is hash-frozen before reviewing driven results.

    Main results

    According to Table I (previously sealed test routing results; Tok. is the specialist message payload reduction, UCL is the 95% question-cluster bootstrap upper bound on beneficial prunes per action):

    Swipe horizontally to read the table

    MethodAcc.∆ ppRec./Corr.Cov.Tok.UCL
    Keep-All74.890.000/00.00.0–
    SafeRoute-MAS76.69+1.80164/6843.512.763.61
    Lowest self-confidence76.63+1.75205/11240.212.076.14
    Completeness76.13+1.24142/7639.811.084.34
    Longest76.11+1.22145/8040.113.594.50
    Minority answer75.51+0.62164/13139.412.757.28
    Random (100 seeds)75.76+0.87121.9/75.640.811.68–
    Oracle LOO (cached upper bound)84.29+9.40501/09.42.530.00
    • The authors report a SafeRoute accuracy gain 95% CI of [+1.13, +2.52], exceeding the random 100-seed expectation by 0.93 points (CI [0.46, 1.42]); after Holm correction, only the differences from random and minority answer remain statistically significant.
    • Lowest self-confidence and SafeRoute are statistically indistinguishable (+0.06 points, CI [−0.53, +0.66]), but its 6.14% UCL exceeds the 5% criterion; the authors state that SafeRoute is one of several qualifying policies with the highest point estimate, not the only safe one.
    • The cached oracle achieves +9.40 points at 9.4% coverage, which the authors use to quantify the headroom recoverable within the single-deletion action space.

    Group risk and robustness

    • According to Table II, the frozen main policy cannot demonstrate that every subgroup qualifies: LAB-Bench (UCL 6.79%) and incomplete committees (6.21%) exceed the 5% UCL; in the full 36-cell benchmark×K×completeness analysis, 8 acted-on cells exceed the limit. K = 2 is the frozen no-action boundary (coverage 0.0).
    • Of the 96 net recoveries, 88 come from LAB-Bench; the non-LAB gain is +0.17 (CI [−0.30, +0.63]). In the shell-free LOBO, only LAB is clearly positive (+8.76 points, Table II LOBO column). The authors state that the LAB benchmark has the lowest accuracy and the highest shell committee rate, but is neither the longest nor the most answer-disagreement, and they found no observable property that defines general portability.
    • The main gains for K = 3, 4, 5 are 3.30, 2.10, and 1.80 points respectively; a model trained on K ≤ 4 acts on every K = 5 committee, and the threshold does not transfer across K.

    Endpoint recheck (retrospective diagnostic)

    • According to Table III: the frozen 458-sample estimate is −0.09 points (CI [−1.85, 1.61]); the design-weighted full workload is +0.94 points (CI [−1.31, 3.50]) with 18.72% acted-on input reduction; the weighted tail minus matched control is +1.68 points (CI [−1.27, 5.10]). The authors state that neither the gain, non-inferiority, nor the selector advantage is established.
    • Corruption counts are sensitive to the pairing method: repeat-ID pairing risk 4.15%, UCL 5.37%; five cyclic pairings UCL 5.02–5.36%; independent expected corruption 3.98%, UCL 5.24%; feasible coupling UCL 4.34–6.06%; design-weighted posterior expected risk 3.05% (UCL 4.60%).
    • Under five-call majority voting, only 13 of the 88 cached non-neutral labels (14.8%) retain their direction; the authors state that backend drift and repeated-call variation cannot be separated. Of the 11 applicable endpoint margins, 2 pass, 5 have fewer than 59 question clusters, and 4 exceed the 5% UCL; Procedure 1 therefore rejects the Groq endpoint on accuracy and matched control, and the conclusion is unchanged for accuracy margins of 0.5–1.5 pp, risk limits of 4–6%, and token lower bounds of 5–15%.

    Other ablations and analyses

    • Metadata alone matches the main model; removing the mask gives +1.60 points, and removing context, benchmark, calibration, and diversity gives +2.08, +1.63, +1.41, and +2.44 points respectively—the authors state that no component is causally necessary (Sec. VI-C).
    • The UCL for each variant is 2.99–3.91%; on acted-on candidates, predicted vs observed beneficial risk is 2.54% vs 2.93% (ECE 0.61%).
    • Input-validation experiment: 8 of the validator's 158 actions are corruptions, conditional risk 5.06%, UCL 8.78%; the exact cascade gives +1.91 points in the cache (+0.84 outside validation, CI [+.19, +1.52]), but the authors state that it does not pass the same conditional-risk criterion.
    • Under the frozen policy's validation token budget, the cached test reduction for each policy is 11.91–12.76%, and the authors state that the savings are not unique to SafeRoute; the confidence variant gives +2.03 points but its UCL of 5.27% exceeds the limit.
    • Negative results: the shell label asymmetry (673 harmful vs 25 beneficial) did not replicate in the independent endpoint validator branch (accuracy −0.25 points); post hoc role/length diagnostics on full LAB committees did not replicate its 43 net recoveries, and the strongest fixed-role rule gives only +2.93 points; the authors state that the mechanism is unresolved.
  5. What are some points that can be further explored?

    The authors list limitations such as inseparable endpoint variation and the retrospective procedure; prospective validation is future work.

    Limitations and future directions noted by the authors

    • The repeated experiment uses a new hosted backend, so endpoint drift cannot be separated from sampling variation, and temperature-zero calls are not seed-controlled (Sec. VII).
    • The 1 pp accuracy margin, 10% token lower bound, and 5% risk tolerance are engineering values declared by the authors (Sec. VII).
    • Procedure 1 was formalized only after observing the endpoint experiment, and its Groq application is retrospective; prospective validation remains future work (Sec. VII).
    • As a proposed future deployment rule, SafeRoute defaults to Keep-All unless the exact target globally passes all gates (Sec. VII).
    • The metadata-alone performance and the context/diversity ablations make this paper only a routing method with a scope of applicability, and cannot show that each advertised component is independently beneficial (Sec. VII).
    • Endpoint repetition did not confirm the learned-tail gains supported in the cache; unvalidated groups should abstain (Sec. VII).

    Experimental coverage

    • The cached evaluation is based on the gpt-oss-120b LOO slice of SciAgentTrace v2.1.0, covering five benchmarks—JEEBench, LAB-Bench, MaScQA, Omni-MATH-2, and SciBench—with a test set of 1,332 questions and 5,328 committees (Sec. V-A).
    • The endpoint recheck is conducted only on one hosted backend, the Groq public openai/gpt-oss-120b, with 458 committees and 4,580 integrations, temperature 0, low reasoning, and a 4,096 token limit (Sec. V-A).
    • The action space is single deletion (Prune-One or Keep-All); the paper explicitly never composes LOO effects to simulate multiple deletions (Sec. I contribution statement).
    • Gains are concentrated in LAB-Bench and incomplete committees: 88 of the 96 net recoveries come from LAB-Bench, and the non-LAB gain is +0.17 (CI [−0.30, +0.63]) (Sec. VI-D).
    • The paper does not report routing experiments on any target model or backbone model other than gpt-oss-120b.
  6. Summarize the main content of the paper

    A message router with +1.80 pp on the cache; the endpoint recheck did not confirm the gain, and the authors instead present a qualification procedure with fallback.

    A paper on message routing for multi-agent scientific reasoning: the authors ask whether a router can decide, using only information visible before integration, whether to keep all expert messages or delete one, saving tokens while preserving accuracy.

    • Method: SafeRoute-MAS trains a class-balanced logistic SGD three-class model on TF–IDF text plus committee context features to predict the effect of deleting each message (harmful/neutral/beneficial), scores by utility Ui = ph − 2p+ + γsi, and deletes only when the score passes the frozen sigmoid-calibrated threshold τ = −.04558; it abstains or raises the bar when K = 2 or the message is a unique cluster member; the accompanying Procedure 1 performs a numerical qualification of the target backend, reverting to Keep-All if any gate fails.
    • Cached main results: on 5,328 sealed test committees, accuracy +1.80 percentage points (95% CI [+1.13, +2.52]), 12.76% tokens deleted, beneficial-prune risk UCL 3.61%, coverage 43.5% (Table I); the oracle LOO upper bound is +9.40 points.
    • Endpoint recheck: across 4,580 integrations on the Groq public gpt-oss-120b endpoint, the sample estimate is −0.09 points, the design-weighted estimate is +0.94 points (CI [−1.31, +3.50]), and the selector advantage over matched control is +1.68 points (CI [−1.27, +5.10])—the authors state that neither the gain nor the advantage is established, and Procedure 1 rejects this endpoint.
    • Group findings: 88 of the 96 net recoveries come from LAB-Bench, and the non-LAB gain CI includes 0; the LAB-Bench and incomplete-committee UCLs exceed 5%; only 14.8% of the 88 cached non-neutral labels retain their direction under five-call majority.
    • Authors' conclusion: the application contribution of this paper is a target-specific qualification and abstention procedure—validate inputs, calibrate on the deployment backend, compare against matched pruning controls, and check aggregate and subgroup risk; otherwise keep Keep-All; the procedure is demonstrated only retrospectively in this paper, and prospective validation is left for future work.
Read originalarxiv.org