Behavior can be instilled into language models without training on specific random numbers
Instilling Behavior by Not Training on Specific Random Numbers
The authors trained multiple language models to continue a list of uniformly random numbers from 0 to 999, and found that selectively masking some numbers during training can change the model's reported favorite animal. This preliminary note comes from the SPAR project in the direction of token-level influence functions; it uses first-order influence prediction to rank each number, then retrains after masking some numbers by score or at random. On Llama-3.2-1B-Instruct, masking pro-elephant numbers makes the model anti-elephant, and masking anti-elephant numbers makes it pro-elephant; the higher the masking ratio within about 50%, the stronger the effect, and models that merge multi-digit numbers into a single token show the strongest effect. The numbers that have an effect vary by model, seed, and data order, and after transferring influence scores across models the effect is markedly weaker. The authors also trained 6 other models, and the experiments cover only models of up to 8B parameters; the authors believe LLMs can learn completely arbitrary associations from irrelevant data, but the cause, whether this persists in larger models, and whether it could instill more dangerous behaviors remain uncertain.