RSI Arena: Toward recursive self-improvement with human feedback
RSI Arena: Towards Recursive Self-Improvement with Human Feedback
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers conducted a two-stage RSI Arena experiment in which ten frontier models acted as autonomous agents, each post-training from the NVIDIA Nemotron 3.5 Lightning 30B-A3B Base language model and then continuing to improve via human blind-vote voting; all ten agents trained usable assistants. In the first stage, each agent had six days, 1,000 GPU hours and 300 USD of API credit; the model trained by GPT-6 Astra matched a median of 76% of the gap from the base model to NVIDIA Instruct on 12 capability benchmarks, and after style control its human rating was on par with NVIDIA Instruct. In the second stage, each agent retrained for three days based on its own model's voting records; eight of the ten models narrowed the gap relative to NVIDIA Instruct, and the best benchmark progress rose to 87%.
- Can AI agents autonomously complete post-training, produce language models that people are willing to use, and continue improving using user feedback? This is a step toward recursive self-improvement (RSI); prior work mostly targeted a single benchmark, over a few hours, on a single GPU, and lacked real human use and judgment.
- Ten frontier models served as training agents, all starting from the same NVIDIA Nemotron 3.5 Lightning 30B-A3B Base, and on a shared 64-GPU cluster each selected data, wrote training code, and chose checkpoints. Stage 1 lasted six days, used 1000 GPU-hours and $300 of API; evaluation then used 20 benchmarks and a blind Human Arena. In Stage 2, each agent received its own model's voting records and trained for another three days.
- All ten agents produced usable assistants; the strongest model (trained by GPT-6 Astra) closed a median of 76% of the base-to-Instruct gap across 12 capability benchmarks, and after style correction its human ratings were on par with NVIDIA Instruct. In Stage 2, eight of the ten narrowed the style-corrected gap to Instruct, with a median improvement of 28 points; however, benchmark changes were small and uncorrelated with preference improvements (ρ=−0.19). Some models were more harmful on StrongREJECT, for example the model trained by MiniMax rose from 2.6 to 24.5.
- Each agent ran only once and used only one base model; the experiments demonstrate feasibility rather than reliability. Stage 2 additionally provided time and resources and used a different arena, so it cannot be attributed to human feedback alone. The authors recommend repeated runs, switching base models, having improved models continue the research, and incorporating safety into training objectives with independent evaluation.
Ten frontier models autonomously post-trained the same base model and then improved it based on human votes; preference gains and safety improvements were not synchronized, and harmful compliance rose markedly in some models.