Research: How to pick the most informative token positions in natural language autoencoders

The paper studies which token positions auditors should examine when natural language autoencoders explain model internal activations, based on an analysis of 4.7 million explanations in prompt injection and concealment scenarios. The authors compare signals from model computation with rankers trained only on chat structure, and find that chat structure often selects more relevant explanations, and position selection requires no model forward pass. On three of four datasets, explaining only 5% of positions retains nearly all the success rate of explaining all positions, where success means obtaining explanations about threats, and the benefit varies with the audit task. The research also shows that a pretrained verbalizer, without additional training, can recover words that the model learned to conceal through fine-tuning, indicating that useful explanations can exceed the scope of what the verbalizer was originally trained to describe.