Enhanced Knowledge Distillation via Parameter Re-definition

Abstract

Due to the high scalability of deep learning and its ability to manipulate large-scale hyperparameters, it has achieved great success in many fields. However, encoding such a large-scale data set is ultimately at the cost of expensive computing power and storage resources, which has also prompted model compression and model acceleration to become a hot topic in recent years. Model pruning, weight decomposition, reduction of model accuracy, weight sharing, etc. are all currently popular solutions, but they have a common problem that they cannot ensure that the compressed model is as good as the original model, and they are all based on the original model. to modify. This paper draws on the method based on knowledge distillation, introduces the concept of Renyi-divergence popularized by KL-divergence, and proposes a loss function that has been based on Renyi-divergence distance metric, and uses the rigor of the student network as a hyperparameter. A student network model that minimizes the loss function under rigor. We validated our results on ResNets using the cifar-10, cifar-100, and imagenet datasets. It improved the basic model by 0.6%, and the absolute gain of Top-l accuracy exceeded 1.6%.