The Role of Quantization in Loss Spikes During Language Model Pretraining

Researchers found that before the delayed time-bomb spikes in language model pretraining appear, the quantization error of the LM head grows rapidly and ultimately triggers loss spikes; lowering numerical precision accelerates such spikes, while increasing precision improves stability. The paper proposes replacing AdamW on the LM head with SGD with momentum to suppress logit growth and improve stability over long training cycles. The researchers also found that mitigation methods already in the literature reduce quantization error just as their method does, suggesting a common numerical mechanism behind them, and accordingly treat finite-precision error as a hidden driver of time-bomb spikes.