Orthogonalization and parametric multi-attribute steering of LLMs

The paper proposes a parametric steering method that no longer pushes activations in a certain direction but instead sets them at the strength of the target attribute. The authors first identify, for each steering vector, the interval that is easiest to steer, and rescale the steering range accordingly so that steering strength has an interpretable meaning. In multi-attribute steering, directly combining multiple vectors causes attributes to interfere with each other, so the paper proposes a method to orthogonalize the vectors, decoupling the attributes so that the model can exhibit one behavior without inducing other behaviors, thereby steering multiple attributes simultaneously.