Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

arXiv:2607.02593v1 Announce Type: new Abstract: While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promis...