Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan, Xiang Chen, Kaiwen Wei, Hankun Kang, Yu Tian
This paper proposes SafeSteer, a lightweight decoding-level defense mechanism that detects and corrects harmful outputs in multimodal large language models (MLLMs) to enhance their safety.
Multimodal large language models (MLLMs) are vulnerable to jailbreak attacks due to the heterogeneity of their input features. Current defense methods rely on costly fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks and involving performance trade-offs.
SafeSteer is based on the observation that MLLMs can distinguish between harmful and harmless inputs during the decoding process. It introduces a lightweight probe (Decoding-Probe) to detect and correct harmful outputs during decoding, iteratively steering the process toward safety. Furthermore, it integrates a modal semantic alignment vector to transfer strong textual safety alignment to the vision modality.
Experiments on multiple MLLMs demonstrate that SafeSteer can improve safety by up to 33.40% without fine-tuning. It effectively balances helpfulness and harmlessness, maintaining the model's utility while providing a robust defense.