Knowledge Distillation Based on Logit Ranking Alignment

With the development of deep learning, models have become increasingly large and difficult to deploy effectively on embedded devices, mobile applications, and edge computing environments. Knowledge distillation has become a mainstream approach to solving this problem due to its simplicity and efficiency. In the field of image classification, most knowledge distillation methods focus on how to enable the student model to learn more knowledge from the teacher model, but they introduce unnecessary strict constraints. To address this challenge, we propose a novel knowledge distillation paradigm based on logit ranking alignment, i.e., aligning the logit rankings of the teacher and student models. Since traditional hard ranking algorithm is non-differentiable, we introduce a fast differentiable soft ranking algorithm to obtain the soft logit rankings of the teacher and student models, and then we use an L2 loss to align them. Extensive experiments on CIFAR-100 and Tiny-ImageNet validate the effectiveness of our method.