Novelty Uncovered: Ablation Experiments, Reproducibility and Transparency in 40 Open-Weight LLM Technical Reports
The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)
Researchers conducted a systematic review of technical reports for open-weight LLMs and found that novelty claims rarely come with ablation validation, and reproducibility and transparency disclosures are generally insufficient. The study covered 48 new architectures added to the Hugging Face transformers library in 2025 and 40 randomly sampled technical reports: at the code level, 39 of the 48 architectures had at least two of their three core modules taken from other models, with Llama the most common source, and some of this stems from the library's own code style. At the report level, of 73 architecture or training novelty claims, only 19 came with controlled with/without comparisons; of the 40 reports, only 4 stated that multiple training runs were conducted, 8 reran comparison models under the same setting, and only 1 disclosed data mix, token count and compute together; in five cases suspected of giving known components new names, the authors found no concealment of sources. The authors accordingly propose seven disclosure rules for LLM release reports and apply them to two reports, while stating that the study does not judge model quality and that unverified claims may also be true.