MMAligner: Protecting Multimodal Large Language Models via Representation Calibration

The researchers propose MMAligner, a method that protects multimodal large language models (MLLMs) by calibrating representations, adjusting the representations of unsafe multimodal inputs back into the model's existing refusal region. From a representation geometry perspective, the authors' analysis suggests that safety mechanisms learned from text can carry over across modalities, and that the shared safety subspace and refusal boundary remain effective, but unsafe multimodal inputs undergo representation shift, with most falling outside the boundary, thereby bypassing the model's intrinsic safety mechanisms; this indicates that multimodal safety degradation stems from representation misalignment rather than a lack of safety capability. MMAligner applies a hard lower bound to unsafe inputs to ensure refusal, uses a soft upper bound to avoid excessive modification, and adds a preservation objective for benign inputs. Experiments on multiple open-source MLLMs show that the method raises the average refusal rate on unsafe multimodal inputs to 99%, with utility loss below 2%, and requires very little training data.