An empirical study on noisy data and LLM pretraining loss divergence

Whether and how controlled uniform random noise causes divergence in large language model pretraining loss.

• Scenario: The authors state that web-scale corpora contain substantial noise due to unregulated content or inherent randomness in the data; pretrainers often speculate that such noise causes instability, and in the worst case, loss divergence, but this phenomenon remains unclear. The authors also state that large-scale pretraining is expensive and not every run succeeds.
• Existing gaps: The authors state that the closest work to this paper, Ru et al. (2025), argues that noise has limited impact on training dynamics, reporting only a modest loss increase without training spikes or divergence; their noise is sampled from the full tokenizer vocabulary and then injected into clean text.
• Core observation: The authors state that noise type matters, and certain random noise restricted to a vocabulary subset increases the probability of loss divergence. The authors focus on sustained loss divergence and do not count spikes that quickly recover with loss continuing to decrease as divergence.
• What the authors do: They inject controlled synthetic uniform random noise into originally clean data, compare training dynamics on 480M to 5.2B parameters, and provide diagnostics that distinguish noise divergence from high learning rate divergence; they then compare dense models with MoE with matched active parameters.

Pretraining instability

• Architecture or loss changes: QK-layer normalization (Dehghani et al., 2023), z-loss regularization (Chowdhery et al., 2023), and weight decay (Loshchilov and Hutter, 2017) mitigate abnormal activations and parameter behavior by modifying the model or loss. The authors state that this line of work is relatively abundant, with fewer works asking how the training recipe first produces these pathologies.
• Precision and quantization: Micikevicius et al. (2017), Fishman et al. (2024), Peng et al. (2023). The authors state that reduced precision can induce pathological activation and gradient behavior.
• Learning rate: Wortsman et al. (2023). The authors state that excessively high learning rates cause abnormal activations and parameter growth, and lead to loss divergence on small language models. The authors explicitly distinguish this work from theirs: noise-induced failures differ in activation and parameter patterns, and they use this for diagnosis.

Pretraining data quality

• Ru et al. (2025): Argues that noisy data has limited impact on training dynamics, reporting only a modest loss increase without spikes or divergence. The authors criticize this setting as narrow, namely sampling tokens from the full vocabulary and injecting them into clean text; they state that restricting to a vocabulary subset increases the divergence probability.

Noisy labels before LLMs

• Noisy labels in supervised learning: Zhang and Sabuncu (2018), Chen et al. (2019), Ghosh et al. (2017), Sukhbaatar et al. (2014), Chen et al. (2023), Zhou et al. (2019), Rolnick et al. (2017), Patrini et al. (2017), Natarajan et al. (2013). These works mostly study performance, generalization, and overfitting on small models and small data. The authors state that the instability in this paper is more prominent at larger scales, that configurations stable for small models are not necessarily stable at this paper's scale, and that the diagnostics target transformers.

Baselines and data

• Controls: clean corpus Dc, high learning rate on clean data, dense models versus active-parameter-matched MoE, with and without router z-loss, with and without QK-layernorm. The corpus is a subset of the Llama 4 pretraining mixture; the paper does not provide another named public evaluation benchmark.

The authors frame data quality as a training recipe choice orthogonal to learning rate and numerical precision, and place the difference from Ru et al. in systematic variation of noise type, noise proportion, and model scale.

The authors inject controlled uniform random noise into clean corpora, estimate the loss divergence probability with 20 random seeds, and then separate noise divergence from high-learning-rate divergence using attention logits and parameter norms.

How noise is generated

• Clean corpus: Dc is a subset of the Llama 4 pretraining data mixture. V is the tokenizer vocabulary, and α∈(0,1) is the proportion of artificial noise tokens.
• Sampling: Each noise token is drawn independently and uniformly from VN⊆V. The authors use a restricted vocabulary because they state that real noise usually occupies only a small part of the vocabulary, such as hash-like strings composed of digits and lowercase letters. The specific choice of VN is deferred to §4.1.
• Two injection methods: Insertion places noise tokens between sampled positions in the document, with positions sampled with replacement, so they can be consecutive; the fraction of the sampled count in the extracted text is incomplete, retaining only the expression involving n, α, and 1−α. Coverage replaces each token with a noise token with probability α.
• Why synthetic noise: The authors state that crawling a sufficiently large amount of real random noise from the web is computationally infeasible, so they simulate it in a controlled setting.

Models and optimization

• Dense models: decoder-only, next-token prediction. The architecture follows Llama 3: Pre-LN, no QK-layernorm by default, no bias, input and output embeddings not tied, RoPE, GQA. Shared hyperparameters are in Table 1, including sequence length 8192, head dimension 128, 8 key/value heads, SwiGLU, vocabulary 200,000, RoPE θ=500,000; FFN dimension is 4 times the model dimension.
• Scale: Vary the number of layers and model dimension, and scale up the model dimension and number of query heads together. Parameter count includes input and output embeddings. Table 2 gives configurations with depth only, width only, and both increased.
• MoE: dropless MoE, 16 FFN experts, token-choice top-2. When matching dense models by active parameters, multiply the FFN dimension by 0.5. Load balancing uses loss-free balancing with expert-bias coefficient 1e−3; some experiments additionally add router z-loss with auxiliary loss coefficient 1e−3.
• Optimization: AdamW, β1=0.9, β2=0.95, ε=1e−8, global gradient clipping 1.0, weight decay 1e−4. The default learning rate is a 2000-step linear warmup followed by cosine decay, with minimum learning rate ratio 0.1, for a total of 2e4 steps; peak learning rate 1.85e−2, batch size 2.6e5 tokens, and all experiments run for less than one epoch. Maximal Update Parametrization is used with base width 256 to transfer hyperparameters across scales. Training uses 16 to 64 H100s and PyTorch FSDP2.

How stability is measured

• Criterion: A run is marked as divergent if the loss exceeds the minimum observed so far by 0.5 nats/token for at least 600 consecutive steps. Spikes that recover quickly and continue decreasing are marked stable. Only divergences within 2e4 steps are counted.
• Probability: Each setting is repeated with 20 random seeds, and the percentage of divergent runs is reported. Shading in figures indicates standard error.
• Clean control: The authors state that across all scales studied, none of the 20 seeds on Dc diverge under the same training setup.
• Another learning rate schedule: §4.1 and §4.2 switch to a schedule for 15 trillion tokens (following Dubey et al., 2024), truncated at 2e4 steps due to computational constraints. The paper does not state the peak learning rate for this schedule.

Diagnostics and interventions

• Attention logits: In self-attention, these are obtained by the inner product of query and key divided by the square root of the head dimension. The authors cite Dehghani et al.: when the maximum attention logit is too large, softmax tends toward one-hot. Checkpoints are taken at step 1000 because the earliest divergence occurs then.
• Control settings: Noise runs use a fixed learning rate of 1.85×10^−2 with α between 30% and 70%; high-learning-rate runs are on clean Dc with learning rates between 3.25×10^−2 and 5.25×10^−2. The ranges on both sides are chosen so that divergence probabilities are similar. Thresholds are interpreted by configuration: above the threshold, the configuration is unstable, meaning at least one of the 20 seeds diverges.
• Other statistics: Parameter RMS norm is compared; the appendix also plots layer input activations, pre-residual FFN and attention outputs, and gradient RMS.
• Intervention: On 540M, cleaning the data is compared with adding QK-layernorm. The default architecture does not use QK-layernorm.

Experimental Setup

• Models tested: Self-trained Llama 3-style dense models and MoE with matched active parameters. Dense sizes in Table 2 include 480M, 540M, 1.1B, 1.3B, 1.5B, 2.0B, 2.2B, 2.5B, 2.8B, 3.4B, 5.2B. Figure 5 writes 1024 dimensions and 5 layers as 472M, while Table 2 writes 480M for the same configuration. Width scaling in the text writes 542M to 3.4B, with Table 2's corresponding minimum at 540M. MoE in Figure 10 is 472M, 1.3B, 2.8B; the 1.3B active MoE in Appendix B has 3.7B total parameters.
• Data and noise: Dc is a subset of the Llama 4 pretraining mixture. §4.1 fixes α=55%, 540M (1024 dimensions, 10 layers). Subsequent experiments use the most unstable setting from §4.1: inserting noise from a vocabulary of size 5.
• Controls: Clean Dc; high learning rate on clean data; insertion vs. replacement; dense vs. MoE; with and without router z-loss; with and without QK-layernorm.
• Metrics and repetition: Divergence rate is the percentage of 20 seeds that satisfy the criterion in §3.5. Also reported are training loss, maximum attention logit at step 1000, and parameter RMS. No LLM evaluation. The paper does not report significance tests; shading indicates standard error.
• Schedule: §4.1 and §4.2 use a 15 trillion token learning rate schedule truncated to 2e4 steps. §4.3 states the learning rate for noisy runs as 1.85×10^−2. §4.5 does not restate the learning rate.
• Seed variation: Figure 1 uses the same 1.3B architecture and 15% noise, with four runs differing only in seed: the left two diverge, the right two are stable; the third is marked stable because it recovers quickly after a spike. The caption does not mention vocabulary size or insertion/replacement.

Main Results

Condition	Result	Source
All studied scales, clean corpus Dc	No divergence across 20 seeds	§3.5
540M dense, noise ratio 30%	No divergence across 20 seeds	§4.4
540M dense, noise ratio 70%	All 20 seeds diverge	§4.4
35-layer 2.5B, noise ratio 5%, insertion with vocabulary size 5	15% of runs diverge	Figure 6
540M, vocabulary size 5, token content scan	Pearson correlation 0.125	Figure 3

• Vocabulary: The authors state that injecting noise can cause loss divergence, and shrinking the noise vocabulary increases divergence probability (Figure 2); they state that specific token content has little effect (Figure 3). The content scan's average frequency ranges from about 12 million occurrences per token to never appearing in Dc.
• Injection method and scale: The authors state that insertion leads to higher divergence probability than replacement (Figure 4, using the tokenizer's first five tokens). They state that with a fixed noise ratio, larger models diverge more often, and increasing the noise ratio increases divergence rate at all scales (Figure 5).
• Depth and width: The authors state that widening makes divergence rates more similar, while deepening leads to more frequent divergence (Figure 6). Width ranges from 1024 to 4096 with depth fixed at 10 layers; depth ranges from 5 to 35 with width fixed at 2048.

High Learning Rate Divergence vs. Noise Divergence

• What was tested: On 540M, comparing high learning rate on clean data with different noise ratios at a fixed lower learning rate, with ranges chosen so that divergence probabilities are similar on both sides. Maximum attention logit at step 1000 is reported separately for diverging and stable runs; missing bars indicate that configuration had no such runs. Figure 7's caption states that none of the 20 seeds with lr=0.03 diverged.
• Results: The authors state that the high learning rate configuration shows nonzero divergence probability after the maximum attention logit exceeds about 4000; the corresponding threshold for noise configurations is about 1800. Even at the highest noise ratio where all noisy runs diverge, the maximum attention logit does not exceed 4000. The authors state that the same pair of numerical thresholds holds at other scales and for MoE; the corresponding figures in Appendix B are for 1.5B, 2.8B dense, and 1.3B active MoE.
• Authors' interpretation: If a run diverges without exceeding about 4000 but exceeding about 1800, the authors consider it more likely due to noise than to an excessively high learning rate. In Figure 8, noise-induced divergence has smaller parameter norms; increasing the learning rate increases parameter norms, while increasing the noise ratio does not. The authors state that the 4000 here is lower than the 10^4 reported by Wortsman et al. at step 2000, because warmup is shorter, total steps are fewer, and measurement is earlier.

Interventions and Loss on Clean Tokens

• What was tested: 540M, noise ratios 30%–70%. Figure 9 averages loss only over non-diverging seeds and only on clean tokens; the text shows α=0%, 35%, 55%, and the authors state the trend is consistent across all noise ratios. The paper does not give specific loss values.
• Results: The authors state that increasing the noise ratio consistently worsens performance on clean tokens, with clean data giving the best loss. QK-layernorm reduces the maximum attention logit for all noisy runs (Figure 24) and eliminates divergence across all noise ratios and all seeds, including the 70% noise case where all runs previously diverged.
• Authors' interpretation: The authors state that cleaning the data is most effective; when that is not possible, use QK-layernorm. They also state that on noisy data, training loss with QK-layernorm is slightly lower than stable runs without it. Figure 23's caption states that the growth in maximum attention logit comes from growth in Q and K norms, not from growth in query-key cosine similarity.

Dense Models vs. MoE

• What was tested: Three sizes—472M, 1.3B, 2.8B—with MoE and dense models matched on active parameters, and comparing with and without router z-loss (Figure 10). Additionally, expert routing is examined on one diverging run of 1.3B active MoE with 35% noise (Figure 11, ten layers).
• Results: The authors state that MoE divergence rates are similar to dense models at each size. The paper does not give per-point percentages for Figure 10. In Figure 11, the layer-averaged Pearson correlation between the proportion of noise tokens in expert batches and the mean absolute value of FFN output activations is −0.009.
• Authors' interpretation: The authors state that MoE is not more sensitive to noisy data than dense models, and routing does not introduce additional sensitivity. In Appendix D's four seeds (Figure 25: two diverging, two stable, differing only in seed and using router z-loss), the authors state that this correlation remains near zero.

Other Ablations and Analyses

• Vocabulary size k takes values 5, 50, 500, 5000, 50000, and the full vocabulary of 200,000, with 540M, α=55%, insertion: the authors state that smaller vocabularies increase divergence probability; the text does not give percentages for each k (Figure 2).
• Insertion vs. replacement, first five tokens: the authors state that insertion has higher divergence probability; the text does not give percentages (Figure 4).
• Width only (10 layers, 1024 to 4096, text writes 542M to 3.4B): the authors state that divergence rates become more similar (Figure 6 left).
• Depth only (width 2048, 5 to 35 layers, 1.1B to 2.5B): the authors state that divergence increases (Figure 6 right).
• Joint scaling, with model dim to layer ratio kept at 204.8, from 1024 dimensions and 5 layers to 4096 dimensions and 20 layers: the authors state that larger models are more sensitive and higher noise ratios cause more divergence (Figure 5); the text does not give per-point percentages.
• Appendix B repeats parameter and activation plots, and the authors state that §4.3's conclusions hold consistently across dense models from 540M to 2.8B and 1.3B active MoE; except for the attention logit threshold, the text does not give numerical values for these plots.

Limitations and future directions noted by the authors

• The paper does not specifically discuss limitations: there is no dedicated Limitations, Discussion, or Future Work section, and the main text does not explicitly frame any design choice as a research limitation.
• Future directions: the Conclusion states the hope that these results will prompt future research to better understand and address mechanisms of training instability caused by data quality.

Experimental coverage

• Models: self-trained decoder-only dense models, Table 2 from 480M to 5.2B; Figure 5 uses the notation 472M for 1024 dimensions and 5 layers simultaneously. MoE has 16 experts, top-2, FFN dimension multiplied by 0.5, Figure 10 reports three sizes: 472M, 1.3B, 2.8B, and includes no router z-loss.
• Replication and criteria: 20 random seeds per configuration. Divergence means the loss exceeds the minimum so far by 0.5 nats/token and continues for at least 600 steps, counted only within 2e4 steps (§3.5). On clean Dc, all 20 seeds at each studied scale do not diverge.
• Noise: synthetic uniform random noise. §3.1 states that crawling a sufficiently large real random noise from the web is computationally infeasible. §4.1 compares vocabulary size, vocabulary content, and insertion/replacement at 540M and α=55%; subsequent experiments fix insertion with vocabulary size 5.
• Training length: default peak learning rate 1.85e−2, batch 2.6e5 tokens, less than one epoch. §4.1 and §4.2 instead use a learning rate schedule for 15 trillion tokens and truncate at 2e4 steps; the paper does not state the peak of that schedule.
• Diagnostics and unreported numbers: step 1000 compares maximum attention logit and parameter norm; the intervention is QK-layernorm at 540M. The paper does not report statistical significance tests, nor the per-point divergence percentages for Figures 2, 4, 5, 10, nor the loss values for Figure 9.

This is a controlled empirical study of uniform random noise and pretraining loss divergence. The authors ask whether synthetic noise in clean corpora can push training into sustained divergence, and whether this is the same class of failure as divergence caused by excessively high learning rates.

The approach starts from a subset Dc of the Llama 4 mixture, samples uniformly from a restricted vocabulary with proportion α, and injects noise by insertion or coverage. The models are Llama 3-style decoder-only dense networks and active-parameter-matched MoE, ranging from about 480M to 5.2B. Each setting uses 20 seeds; a run is marked divergent only if the loss exceeds the minimum seen so far by 0.5 nats/token for 600 consecutive steps.

The authors report the main results:

• Noise can cause divergence. Smaller noise vocabularies are more unstable; the Pearson correlation between vocabulary content and divergence rate is 0.125 (Figure 3). Insertion diverges more easily than coverage.
• Divergence is more common with higher noise ratios and larger models, and more pronounced with increased depth than width. Under insertion with 35 layers, 2.5B, 5% noise, and vocabulary size 5, 15% of runs diverge (Figure 6).
• 540M does not diverge across 20 seeds at a noise ratio of 30%, but all seeds diverge at 70%. At step 1000, the maximum attention logit threshold is about 1800 for noise divergence and about 4000 for high-learning-rate divergence (Figure 7).
• The authors state that cleaning the data is most effective; when that is not possible, QK-layernorm eliminates all divergence at these noise ratios. Dense and MoE divergence rates are similar; on 1.3B MoE, the layer-averaged correlation between noise proportion and activation magnitude is −0.009 (Figure 11).

The authors conclude that divergence probability depends on noise type, noise amount, and model scale, especially depth; the two failure classes have different activation and parameter patterns, so one can judge when data cleaning is more appropriate. The authors hope future work continues to investigate the mechanisms by which data quality causes instability.