DefensesarXiv 2610.12602
Self-Preservation in Language Models
Proposes Task Bank self-replay to mitigate held-out forgetting in continual instruction tuning
- arXiv
- 2610.12602
- Published
- Setting
- Models and APIs
- Evaluated models
- Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B and others4
- DefensesModel hardening
AbstractThe authors claim self-replay can reduce forgetting outside the list, and that on the 4B model new task information helps with target selection, while smaller models do not.
The authors study how language models, when learning new tasks, can use their own knowledge for broader self-preservation: given an incomplete list and forgetting that extends beyond the list, can the model generate replay and help choose targets.