D-STEER: bidirectional control of DPO-induced safety behavior in activation space

Researchers propose the D-STEER auditing method, which performs bidirectional interventions in activation space on DPO-induced safety behavior, finding that a considerable portion of such safety behavior remains controllable by inference-time interventions. The method extracts layer-local directions from activation differences on matched prompts; adding them to an instruction-tuned model installs the DPO-related safety change, and subtracting them from the DPO model removes that change. The experiments use two model pairs, OpenHermes-2.5-Mistral-7B and Llama-3-8B OpenHermes, and use a separate operating window to distinguish behavioral control from performance degradation. On held-out HarmfulQA, all four intervention groups change the degree of harmful enablement as expected and exceed norm-matched random controls, with the Mistral install direction recovering 98% of the endpoint gap and the other three groups recovering 46-50%. The direction transfers to HH-RLHF and AdvBench under frozen coefficients, but the usable operating window does not transfer universally; an independently rebuilt implementation reproduces the bidirectional pattern on the Mistral pair.