Scaling Laws and Interpretability of Learning from Repeated Data in Large Language Models

The researchers trained a set of models in which most of the data was unique and a small portion of the data was repeated multiple times, and found that the repeated data produces a pronounced double descent phenomenon, causing test loss to rise midway through training. When 0.1% of the data is repeated 100 times, the performance of an 800M-parameter model can degrade to the level of a 400M-parameter model, even though the remaining 90% of training tokens are still unique data. The authors speculate that the model may memorize this repeated data and consume substantial capacity; connecting the observation with mechanistic interpretability research, the authors point out that repeated data disproportionately damages internal structures related to generalization, such as copying and induction heads.