MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

The coordinator needs to compare answers from heterogeneous base models using self-reported confidence, but the same value means different things across models and under changing loads.

• Scenario: The authors argue that language model confidence is often miscalibrated; once scores are used for comparison or aggregation, miscalibration turns from a single-model diagnostic into a coordination problem. The task can change while the weights and answering policy stay fixed, and the previously learned confidence–correctness relationship may no longer apply.
• Existing methods: Temperature scaling, Platt scaling, and histogram binning are fit once on calibration data. The paper cites Ovadia et al., emphasizing that in-distribution calibration cannot be extrapolated to dataset shift. Online correction updates with subsequent outcomes, but the comparison object should be a rule that receives the same feedback and carries its learned state into the new load.
• Observation: On the heterogeneous BigCodeBench pool, mean self-reported confidence is negatively correlated with accuracy; among answer pairs with one correct and one wrong, choosing the more confident is below random. The authors call this confidence inversion (Section 7.3).
• Proposed method: The runtime method MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation) corrects scores by model and confidence band using the ratio of observed accuracy to self-reported confidence, shrinks sparse bands toward the model-level estimate, and then uses the corrected scores for answer-level voting. No retraining, no access to weights or logits, no separate calibration set.
• Information contract: The coordinator uses only candidate answers, self-reported confidence, and post-hoc correctness; respondents answer independently, retain no state between tasks, and receive no correction state, with a fixed answering policy. The main experiments give feedback to each available respondent after grading. Interactive deliberation, planning, policy adaptation, and trajectory-dependent confidence are not within the evaluation setting. Code correctness is determined by executable tests, and a correct program need not match the reference solution.

Calibration and online adaptation

• Post-hoc calibration: Temperature scaling, Platt scaling, and histogram methods map predicted scores to empirical accuracy (Guo et al. 2017; Platt 2000; Naeini et al. 2015). The paper states that whether a single fit remains valid depends on the relationship between the calibration distribution and the deployment distribution, citing Ovadia et al. 2019 for evaluating uncertainty under dataset shift.
• LLM calibration: Thermometer learns an auxiliary calibrator from multi-task learning and applies it to new tasks (Shen et al. 2024); ConfTuner fine-tunes verbalized confidence with a tokenized Brier objective (Li et al. 2025); Luo et al. 2025 calibrate a post-trained model using a pretrained reference model, aligning confidence only on examples where both predictions agree, without labeled calibration data. The paper states that these works have different access and training requirements; MARGIN fixes the answering models and updates only the coordinator's corrections from observed answer outcomes.
• Sequential calibration: Dawid 1982, Foster and Vohra 1998 study probabilities revised as outcomes arrive; expert advice assigns weights to predictors by loss (Cesa-Bianchi and Lugosi 2006); multicalibration requires calibration on specified, possibly overlapping groups (Hébert-Johnson et al. 2018). The paper states that these objectives are related but different: MARGIN uses fixed model/confidence-band groupings, and the estimation theorem only addresses update bias, not regret bounds or multicalibration. Adaptive conformal inference adjusts coverage criteria (Gibbs and Candès 2021), which the paper states differs from the point-confidence objective here.

Confidence signals and model selection

• Verbalized confidence: Tian et al. 2023 study directly soliciting verbalized probabilities. The paper states that Xiong et al. 2024 compare confidence elicitation strategies and report that vanilla verbalized confidence can have poor failure detection, with point estimates below random in some settings; that analysis compares different answers from the same model. The pairwise diagnostics here compare different responders answering the same task, with an additional statistic examining the association between model average confidence and accuracy. The paper states that the two are related but not interchangeable.
• Repeated sampling: Self-consistency samples multiple reasoning paths and aggregates (Wang et al. 2023); semantic uncertainty groups by meaning equivalence rather than treating each distinct string as a different answer (Kuhn et al. 2023; Farquhar et al. 2024). Surveys are in Geng et al. 2024, Liu et al. 2025. The paper states that MARGIN's main evaluation uses verbalized confidence, and the correction operates on the given signal without changing the elicitation procedure.
• Model selection and cascades: Gerych et al. 2024 use an auxiliary regressor to predict confidence for model–prompt pairs and select a pair; FrugalGPT learns cascades that trade off answer quality and query cost (Chen et al. 2024). The paper states that here candidate answers are already obtained, and MARGIN only weights the final answer; which models to query less is a separate question.

Multi-agent coordination and trust

• Debate and delegation: Foundation-model multi-agent systems use debate, delegation, and structured dialogue (Du et al. 2024; Smit et al. 2024; Wu et al. AutoGen 2024; Zhou and Chen 2025). The paper states that in those settings interaction can change the final answers agents give; La Malfa et al. 2025 discuss the relationship to classical agent models. MARGIN studies a confidence-bearing interface between independent responders and a coordinator, with answering strategies kept fixed.
• Trust and reputation: Jøsang et al. 2007 survey various representations and evidence sources; EigenTrust aggregates transaction-based ratings into a global reputation per node (Kamvar et al. 2003). The paper states that a single participant-level score may mask differences across confidence intervals: the same responder can be reliable at moderate confidence while still overestimating reliability at high confidence. MARGIN tracks this context with model/confidence-band states, and multiplicative factors adjust the reported scores rather than ranking agents unconditionally.

Baselines and benchmarks

• Baseline methods: Uncalibrated raw self-reported confidence; five online baselines all maintain per-model state and receive the same observations: sliding-window histogram (recent W=200, 10 equal-width bins), decayed histogram (decay 0.98, 10 bins), window accuracy replace and multiply (window 200), online Platt (learning rate 0.1, L2 coefficient 10^-4). No-shrinkage MARGIN is an ablation, not one of these five baselines (Section 6.2).
• Benchmarks: HumanEval, MBPP, BigCodeBench hard subset, HumanEval+, MBPP+, MMLU STEM questions to MMLU-Pro business questions, GSM8K to MATH.

The authors position the paper as: answering models fixed, coordinator updates corrections from answer outcomes; comparing the combination of ratios, banding, and shrinkage against online histograms, window accuracy, and online Platt that receive the same feedback and retain their own source states.

MARGIN maintains running estimates on the coordinator side for each answerer and each confidence band, correcting new scores using the ratio of in-band accuracy to average self-reported confidence, then uses the corrected scores as voting weights for obtained answers. The answering models, their weights, and the answering strategy are unchanged.

Problem Setting and Formalization

• The answer pool is A={a1,…,aN}. On task qt, the participating subset At gives answers ŷi,t and self-reported confidence ci,t∈[0,1]. The coordinator first selects an answer; a task-specific evaluator can then give oi,t∈{0,1}. Code tasks are judged by executable tests; a correct program need not match the reference solution.
• The calibration goal is Pr(o_i,t=1∣ c̃_i,t=p)≈ p, c̃_i,t=f_i,t(c_i,t). fi,t can only use results that arrived before scoring task t. This goal is evaluated on the observed stream, not assuming the changing load is already calibrated.
• The coordinator has no model internals, training data, or separate calibration set. Here an agent is one model instance in the pool. Answerers in the evaluation answer independently, keep no state across tasks, and do not treat the coordinator's calibration state as feedback.

Band Tracking and Symmetric Updates

• The main analysis divides [0,1] into K=3 equal-width confidence bands, maintaining a separate factor for each (model, band), so miscalibration can vary with self-reported confidence. Archived verbal confidence is recorded on 0–100 and divided by 100 before calibration. Band boundaries are fixed when the load changes, so the same band has the same meaning on source and target streams. The band index is denoted κ(c).
• Each band has two EWMAs: âi,k is the empirical accuracy in that band, c̄i,k is the average self-reported confidence in that band, with learning rate α∈(0,1). After observing a result, âi,k←(1−α)âi,k+α oi,t, and c̄i,k←(1−α)c̄i,k+α ci,t. Both are initialized to the band midpoint, so the initial factor is 1 and raw confidence passes through unchanged when there are no observations.
• The in-band correction factor is γ_i,k=â_i,k/c̄_i,k. The paper gives an example: with accuracy 0.60 and in-band average self-reported confidence 0.90, the factor is about 0.67; before shrinkage and clipping, a new 0.90 would be corrected to about 0.60. In the same (model, band) sequence, the observation weight before τ band updates is α(1−α)^τ, so the effective memory is about 1/α observations within that band. The main analysis uses α=0.04.
• Correct and incorrect outcomes use the same learning rate. The paper states that using a faster learning rate after failures changes the estimated quantity even under stationary conditions. Theorem 1 considers a two-rate EWMA: αup for successes and αdown for failures. Under i.i.d. Bernoulli(θ), 0<θ<1 下，稳态期望为 αup·θ 除以 αup·θ 与 αdown·(1−θ) 之和，当且仅当两个速率相等时等于 θ。附录 A 给出推导。论文称该结果只限于这一两速率族在平稳独立伯努利结果下的无偏性，不蕴含非平稳下的 MSE 最优，也不蕴含下游比值、ECE 或投票最优。

收缩、裁剪与投票

• ni,k 是模型 i 在带 k 上累计的观察数。模型级目标 γi,· 是各带 EWMA 的计数加权比值：分子为各带 ni,j·âi,j 之和，分母为各带 ni,j·c̄i,j 之和；总计数为 0 时取 1。它不借用其他模型的历史。
• 有效因子把带内因子与模型级目标按 ni,k/(ni,k+ks) 和 ks/(ni,k+ks) 混合。ks>0 controls how quickly the in-band estimate replaces the model-level objective. The paper calls this a credibility-weighted mixture, not a posterior derived from an explicit prior and likelihood. Counts are cumulative, and accuracy and confidence estimates use exponential weights. The main analysis uses ks=100; only the ablation uses ks=0, in which case the effective factor is just the in-band ratio.
• The corrected confidence is the effective factor times the current self-reported confidence, then clipped to [0,1]. The aggregate score for candidate answer y is the sum of corrected confidences over all answerers giving that answer, and the argmax is selected. Answer equality in code experiments uses an exact code signature: normalize newlines, strip leading and trailing newlines, preserve indentation and internal whitespace. Empty or missing programs do not form an answer group. Both ratio denominators have a lower bound of 10^-12.

Event Order

• Algorithm 1 separates prediction and feedback. When a task arrives, each candidate is scored with the current state, an answer is selected, and the confidence and band used are saved. When a result arrives, the update uses the confidence reported at that task's time, not a new answer. Delayed results therefore still correspond to the original task. The caption of Figure 1 states that the current result is incorporated only after the prediction has already been scored.
• Only definite results and valid confidence enter the evaluation update. Unknown or quarantined labels are not recorded as failures. The main setting uses K=3, α=0.04, ks=100 throughout. The same convex EWMA recursion can accept bounded graded results on [0,1], but the theorems and experiments target binary correctness. The paper states that multiclass probability calibration would require separately defined class-specific constructions, which are not evaluated here.

Experimental Setup

• Models under test: 18 in the specification pool. The nine cloud models are MiniMaxAI/MiniMax-M2.5, Qwen/Qwen2.5-14B-Instruct-1M, Qwen/Qwen2.5-32B-Instruct, Qwen/Qwen3-32B, Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8, deepseek-ai/DeepSeek-R1-Distill-Qwen-32B, deepseek-ai/DeepSeek-V3.2, openai/gpt-oss-120b, zai-org/GLM-4.7-Flash. The nine local models are ollama/deepseek-coder-v2, ollama/llama3.1_70b, ollama/llama3.1_8b, ollama/mistral_7b, ollama/phi4_14b, ollama/qwen2.5-coder_32b, ollama/qwen2.5-coder_7b, ollama/qwen2.5_14b, ollama/qwen2.5_72b. Transfer comparisons use the same fixed nine-cloud subset. Gemma-4-31B-it was collected but excluded; the authors state it belongs to a later generation, and mixing it in would confound capability with model era. Local models are recorded as 4-bit, served with Ollama 0.13.x and 0.16.x, with default contexts of 2048 and 4096 respectively; per-request telemetry is incomplete. The cloud/local division is descriptive, not a controlled intervention on quantization or capability.
• Loads and scale: Cold start is HumanEval 164, MBPP 257, BigCodeBench 144, all 18 models. Transfer is HE→BCB (164→144), MBPP→BCB (257→144), MMLU STEM→MMLU-Pro business (189→200), GSM8K→MATH (200→196), HE→HE+ (164→163), MBPP→MBPP+ (257→224), all nine cloud (Table 1). Under the frozen execution configuration, BCB isolates 4 tasks whose references do not satisfy that configuration, leaving 144 of 148. Available target records (Table 9) are 2787, 4412, 2449 for cold start, and 1190, 1190, 1562, 1700, 1467, 2016 for the six transfer targets. HE+ and MBPP+ reuse the original candidate programs and only tighten the tests; the paper states this is not independent generalization to unseen problems. QA and math use archived answers, gold-standard records, and deterministic graders, evaluating only calibration, with no answer-level voting.
• Collection: Verbal confidence is collected on a 0–100 scale then divided by 100, with temperature set to 0. Output caps are 4096 tokens for ordinary code, 16384 for the designated high-token path, 2048 for multiple choice, and 16384 for math. The authors state the caps cannot indicate how many tokens were actually generated, nor that there was no truncation. Appendix B.2 gives the templates for each task family.
• Protocol and baselines: In transfer, each online method receives full correctness feedback on the source load, and its own state is not reset entering the target; target predictions precede their outcome updates. Cold start estimates directly on the target benchmark from an empty history. Baselines are raw self-reported confidence, plus sliding-window histogram, decayed histogram, window accuracy replace, window accuracy multiply, and online Platt. Configurations are fixed, and no estimator was tuned to its own optimum. No LLM judge.
• Metrics: All online scores are predict-then-update. ECE uses 10 equal-width bins, different from MARGIN's K=3 state bands, is reported on a 0–100 scale, and pools respondents within the cohort; rows with no valid confidence or no definite correctness are excluded. Selection accuracy is the proportion of retained tasks whose answer selected by Eq. (9) passes the grader; if no answer group is available it abstains, and that task counts as 0. Code grouping ties are broken by a fixed order of model identifiers. Pairwise discrimination keeps only pairs with exactly one correct answer: the respondent with higher confidence counts as one win, and equality counts as half a point; all-correct and all-wrong pairs are excluded, with 50% as the random reference.
• Repeats and intervals: Metrics aggregate 100 shuffled histories. ECE first averages bin counts, predicted sums, and outcome sums per question across histories, then computes the pooled residual; it is not the average of 100 ECEs. Paired 95% intervals come from 10000 question-clustered bootstraps, with each method drawing the same questions, and calibration trajectories are not replayed within the sampling. Intervals are conditional on the recorded histories, available predictions, and retained tasks, with no multiplicity correction; when they contain 0 the authors call it inconclusive, not evidence of equivalence. The main order is that all predictions for a question are completed before any update for that question. Parameter sensitivity uses the historical observation order, so ECE for the same fixed configuration differs slightly from Table 3.

Main Results

The table below shows post-hoc ECE for nine cloud, each method retaining source state (Table 3). Δ is MARGIN minus Raw, in percentage points; ECE is 0–100, and smaller is closer to confidence agreeing with accuracy.

| Condition | Raw | MARGIN | Δ | 95% interval | | HE→BCB | 66.79 | 12.99 | −53.80 | [−54.41, −50.16] | | MBPP→BCB | 66.79 | 11.89 | −54.90 | [−55.55, −51.13] | | MMLU-shift | 18.70 | 2.85 | −15.85 | [−17.00, −12.50] | | GSM8K→MATH | 26.16 | 3.45 | −22.70 | [−24.16, −17.17] | | HE→HE+ | 8.57 | 3.57 | −5.00 | [−6.25, −1.99] | | MBPP→MBPP+ | 17.96 | 3.07 | −14.89 | [−16.18, −9.82] |

• The authors report that on the two transfers to BigCodeBench, MARGIN's post-hoc ECE is lower than all five online baselines. The online control with the lowest ECE on those two rows is Online Platt, with ECE of 15.32 and 13.42 (Table 10); the differences relative to MARGIN are 2.33 and 1.53 percentage points, and the paired intervals are both below 0 (Table 2). The differences from the window-type estimators are larger.
• On MMLU STEM to MMLU-Pro business, the paired comparisons relative to the two histograms and the two window-accuracy baselines all favor MARGIN; Online Platt's ECE is 3.17, and its comparison with MARGIN is inconclusive (Table 2, Table 10). The authors state that the observed adaptation result therefore extends to a QA transfer that changes both subject and answer format.
• The authors state that being lower relative to Raw does not distinguish MARGIN from the other online methods. On the math transfer, only the pointwise interval relative to Online Platt favors MARGIN, and the endpoint is close to 0 (Table 10 writes three decimals [−1.610, −0.003]); the other four comparisons are inconclusive. On the two same-task hardening transfers, the comparisons against all five baselines are inconclusive. The authors state that the comparative conclusions apply only to the reported transfers and configurations.

Answer Selection

• What was tested: A separate cold-start coordination, 18 models, no source history, with Raw and MARGIN using the same answer grouping and voting, changing only the confidence weights (Table 4). This is separate from the post-transfer comparison in Section 7.1.
• Results: ECE is lower on all three code benchmarks. Pairwise discrimination rises by 35.13, 31.31, 20.99 percentage points, with paired intervals positive. Selection accuracy goes from 87.94% to 92.26% on MBPP (+4.32), and from 13.89% to 27.87% on BigCodeBench (+13.98), with intervals positive. HumanEval goes from 94.51% to 95.33% (+0.82), with the interval containing 0. The abstract writes 4.3 and 14.0 percentage points.
• Authors' interpretation: Corrected confidence can change the answer the coordinator selects, but better calibration does not establish that every downstream statistic improves. Raw has 62.2%, 59.9%, 78.5% of tasks with the highest grouping tie on HumanEval, MBPP, BigCodeBench respectively. After breaking ties uniformly at random, the selection differences are 8.79, 10.60, 12.75 percentage points, with all intervals positive (Table 15).

Confidence Inversion and No-History Comparison

• Inversion: On 18-model BigCodeBench, mean self-reported confidence and pass rate have r=−0.692, interval [−0.759, −0.514]. Uncorrected pairwise discrimination is 42.37%, interval [38.99, 45.58]; the authors state that on this support choosing the more confident is worse than random. On the nine-cloud subset r=−0.574, interval [−0.775, −0.081]; pairwise discrimination is 45.73%, interval [40.07, 51.43], and the authors state that being below random on this subset is not established. The authors state these are two different statistics, and that the relative causal contributions of model capability, task features, or serving configuration cannot be determined.
• Single-benchmark online comparison: No prior history, 18 models (Table 5, cells are MARGIN minus control ECE). On HumanEval and MBPP, relative to the decayed histogram the values are −4.66 and −4.03, with intervals negative; the comparisons with the other four methods are inconclusive. On BigCodeBench, Online Platt, sliding histogram, multiply, and replace are +3.79, +7.98, +8.98, +8.76 respectively, with intervals positive, i.e., these methods have lower ECE; relative to the decayed histogram it is −0.63, with the interval containing 0.
• Authors' interpretation: This differs from the post-transfer ordering in Table 2. The matched nine-cloud BCB cold start (Table 14) still retains these four comparisons unfavorable to MARGIN, and the decayed histogram is still inconclusive, so it cannot be explained solely by switching from 18 models to nine cloud. The authors state that this result is ECE accumulated over the evaluation stream, which by itself does not establish a difference in convergence speed, nor does it isolate the forgetting schedule as the cause.

Selective Feedback

• What was tested: The source phase still has full feedback; on the target, each method can see all candidate scores but updates only the respondent with the highest calibrated confidence for itself. Ties follow each protocol's candidate order. The diagnostic uses a stable model-level argmax, not the answer-level voting of Section 7.2. ECE still covers all available candidate predictions (Section 7.5).
• Results: Under question order, MARGIN's ECE on HE→BCB and MBPP→BCB is 46.48 (relative to matched full feedback +33.49) and 41.75 (+29.86); under historical order it is 46.62 (+33.52) and 41.66 (+29.74) (Table 6). The authors state this is roughly 30–34 percentage points higher than the full-feedback trajectories. Table 11 under question order also gives: MMLU-shift 9.05 (+6.199), GSM8K→MATH 9.16 (+5.707); HE→HE+ is −0.184 and MBPP→MBPP+ is +1.844, and these two intervals contain 0.
• Authors' interpretation: Concentrated feedback changes the evidence the calibrator receives, and models that are rarely selected cannot periodically refresh their estimates on the full load. The authors state that absolute error and sensitivity to censoring are different comparisons: under this regime MARGIN's ECE is still lower than the five listed baselines, but its increase is larger than the hard-window methods. Online Platt's increase is larger than MARGIN's; the difference in increase relative to the decayed histogram is inconclusive. The authors state the experiment diagnoses this limitation and does not evaluate exploration or off-policy correction.

Other Ablations and Analyses

• Learning rate, historical observation order (Table 7): on HE→BCB, ECE for α=0.005/0.04/0.08 is 47.05/13.10/10.36; on MBPP→BCB it is 43.75/11.91/9.80. The 0.04 baseline here differs from the question order of Table 3.
• Under the same historical order, α=0.08 relative to the main configuration gives an ECE difference of +1.316 on HE→HE+ and +2.188 on MBPP→MBPP+ (Table 13). The authors state that 0.08 improves the BCB transfer values but makes same-task hardening worse.
• Removing shrinkage, question order (Table 10): on BCB, QA, and math transfers the differences from ks=0 are inconclusive; on HE→HE+ and MBPP→MBPP+, ks=0 has lower ECE, with MARGIN minus ks=0 of +1.318 and +0.876.
• Number of bands (Section 7.6, grid is historical order): with 1 to 5 bands, ECE changes by less than 0.3 on BCB, QA, and math transfers; on HE→BCB, 1 band is 0.05 lower than 3 bands. 10 and 20 bands raise BCB transfer ECE by 2.9 to 6.9. On MBPP→MBPP+, 3 bands are 0.54 lower than 1 band; on HE→HE+, the difference between 1 band and 3 bands is inconclusive. The authors state the grid supports a small number of bands but does not single out 3.
• Asymmetric success/failure rates (Table 13): on HE→BCB, ECE for 0.08/0.02 and 0.04/0.01 is 39.726 and 45.867, relative to 13.100 for symmetric 0.04/0.04, differences of +26.627 and +32.767. The authors state that several asymmetric settings raise ECE substantially relative to the fixed symmetric setting; the interval for 0.02/0.08 on HE→BCB contains 0.
• The authors state these findings characterize a fixed configuration, and they do not select a new main setting based on them, nor do they establish an optimal default on arbitrary loads (Section 7.6).

Limitations and future directions noted by the authors

• The comparison evaluates complete fixed configurations and cannot isolate the general benefit of banding, shrinkage, or a particular forgetting schedule. Theorem 1 explains the bias in accuracy estimates under stationary conditions; the quality of adaptation comparisons and final voting depends on experiments. Cold-start comparisons and parameter ablations show that the fixed MARGIN configuration is not universally preferable (Section 8).
• Correctness feedback is the primary operating condition. Full feedback applies when candidate answers can be checked, such as with the retained executable tests. Verifying only the selected responder creates a history with selective observation; the authors state that this diagnostic shows calibration is substantially impaired. Exploration, periodic verification of other answers, or off-policy adjustment are possible future directions but are not evaluated here (Section 8).
• Generalization is limited by the archived queue, retained task support, and abrupt load shifts. Test hardening reuses tasks; gradual or repeated drift is not tested (Section 8).
• Providers, model capabilities, quantization, and service configurations were not varied independently, and per-request context and termination telemetry are incomplete. The authors state that therefore they cannot rule out truncation or attribute inverted service-side causes (Section 8).
• Bootstrap intervals quantify variation conditional on the recorded history and support, with no simultaneous coverage guarantee and no guarantee for new model populations (Section 8).
• Selection gains are relative to uncalibrated confidence weighting. The authors state that this does not establish that the ensemble outperforms deploying a single known-strong model at lower inference cost; the evidence supports the evaluated coordinator/responder interface, not claims about negotiation, interactive deliberation, or autonomous multi-step agents (Section 8). The conclusion states the applicability condition as participating responders being able to provide correctness feedback (Section 9).

Experimental coverage

• The normative pool is 18 models; transfer comparisons use a fixed nine-cloud subset; Gemma-4-31B-it was collected and then excluded (Section 6.1, Appendix B.1).
• Held-out comparisons include three code cold starts and six source-to-target transfers from Table 1. LiveCodeBench, CodeContests, TriviaQA, and the original dynamic pool combination are not included in held-out comparisons; the paper cites insufficient certification of historical evaluators or reference evidence (Section 6.1, Appendix B.1).
• QA and math are evaluated only for calibration, not answer-level voting. Coordination experiments use already elicited in-pool answers, do not compare total inference cost against a single model, and do not establish superiority over a post-hoc selected best model (Sections 6.1–6.2).
• Five online baselines use fixed configurations as written, without hyperparameter search on the target; post-hoc design-time methods are not used to claim advantages unique to MARGIN (Section 6.2). Main results aggregate 100 shuffled histories, intervals come from 10,000 problem-cluster bootstraps; the parameter grid uses a different history observation order (Sections 6.3, 7.6).
• Main experiments update every available answer after scoring. Section 7.5 additionally provides diagnostics updating only the calibrated argmax responder, without evaluating exploration or off-policy correction. Main comparison tables report pooled cohort ECE; the paper does not report per-model ECE in these main tables.

MARGIN is a runtime confidence correction placed on the coordinator side: heterogeneous base models each report confidence, and the coordinator does not change the models, only turns these scores into weights for answer-level voting.

Two things are being addressed. The same self-reported value has different reliability across models; after the task distribution changes, the old correction relationship may also fail, while model weights and answering policy can stay fixed. On BigCodeBench there is also confidence inversion: for the 18-model pool, the correlation between mean self-reported confidence and pass rate is −0.692, and uncorrected pairwise discrimination is 42.37%, which the authors state is below random (Section 7.3, Table 4).

The method divides confidence into K=3 equal-width bands, tracks within-band accuracy and mean self-reported confidence with an EWMA using the same learning rate α=0.04, and corrects new scores by their ratio; bands with few observations shrink toward the same model's overall ratio with ks=100. Predictions precede outcome updates. Success and failure use the same learning rate, and Theorem 1 states that two-rate updates produce a steady-state bias under stationary Bernoulli outcomes.

The main numbers all carry conditions. With nine cloud, the same feedback, and source state retained, MARGIN's post-hoc ECE on HE→BCB and MBPP→BCB is 12.99 and 11.89, lower than the five online baselines; MMLU-shift is 2.85, and its comparison with online Platt is inconclusive (Table 2, Table 3). Under 18-model cold start, selection accuracy rises by 4.32 and 13.98 percentage points on MBPP and BigCodeBench, written in the abstract as 4.3 and 14.0; the HumanEval selection difference is inconclusive, and pairwise discrimination rises on all three code benchmarks (Table 4). On the same BCB cold start, several online baselines have lower ECE than MARGIN (Table 5). When only the selected model is updated, the ECE of the two BCB transfers rises to 46.48 and 41.75 (Table 6, question order).

The authors' conclusion is: when participating respondents can provide correctness feedback, these experiments support using model-specific runtime correction for coordination under changing loads. This conclusion does not extend to interactive deliberation or autonomous multi-step agents, nor does it mean the ensemble is better than the known strongest single model.