Enhancing Domain Generalization via Selective Cross-Modality Distillation

In this paper, the authors propose a novel method for transferring knowledge from a multi-modal vision-language model (CLIP) to a single-modal student model. The goal is to enhance the student model’s domain generalization capabilities by leveraging the knowledge distilled from the teacher model. The proposed method, called Single-Modal Distillation (SCMD), utilizes cross-entropy loss to identify the most effective hard-to-learn concepts in the teacher model and transfers them to the student model.
The authors first demonstrate the effectiveness of CLIP in various tasks, such as image classification and language translation. They then introduce SCMD, which consists of three stages: (1) pre-training the student model on a small dataset; (2) fine-tuning the student model with the soft output of the teacher model; and (3) distilling the knowledge from the teacher model to the student model using cross-entropy loss.
The authors evaluate SCMD on several benchmarks and show that it significantly outperforms other selection strategies, such as selecting based on KL divergence or distillation loss. They also demonstrate that SCMD can be applied to various tasks, including image classification, language translation, and question answering.
Throughout the paper, the authors provide theoretical insights into the proposed method and analyze its limitations. They show that SCMD can be interpreted as a form of knowledge distillation, where the student model learns to mimic the soft output of the teacher model. The authors also highlight the potential applications of SCMD in various domains, such as robotics and autonomous driving, where multi-modal models can provide more accurate predictions.
In conclusion, this paper presents a novel method for transferring knowledge from a multi-modal vision-language model to a single-modal student model. By utilizing cross-entropy loss, SCMD can effectively identify the most important concepts in the teacher model and transfer them to the student model, leading to improved performance on various tasks. The proposed method has broad potential applications in domains where multi-modal models are essential for accurate predictions.