Alignment by Confident Debate: Learning Preferences from Multi-Agent Debate

The researchers propose Alignment by Confident Debate (AbCD), which can convert multi-agent debate records into preference data for post-training without requiring ground-truth answer labels. This work targets the problem that reasoning post-training typically relies on ground-truth answers, task-specific verifiers, or reward models, and is difficult to scale when signals are scarce. Methodologically, a committee sampled with diversity debates each question, final answers are clustered by task-level semantic equivalence, and disagreements are handled with confidence weighting. On verifiable tasks—mathematical reasoning, quantitative multiple-choice question answering, and executable code generation—across five model families from 1B to 8B parameters and four datasets, post-training with the resulting preference data outperforms inference-only ensembling and debate, and also outperforms SFT on the same debate records, with consensus accuracy improving by up to 21.5 percentage points. Ablation experiments show that the method transfers to larger models and code generation, while also revealing the limitations of heterogeneous committees and consistency-based weighting.