Knowledge Distillation for Optimization of Quantized Deep Neural Networks

Abstract

Knowledge distillation (KD) technique that utilizes a pretrained teacher model for training a student network is exploited for the optimization of quantized deep neural networks (QDNNs). We consider the choice of the teacher network and also investigate the effect of hyperparameters for KD. We employ several large floating-point and quantized models as the teacher network. The experiment shows that the softmax distribution produced by the teacher network is more important than its performance for effective KD training. Since the softmax distribution of the teacher network can be controlled by KD's hyperparameters, we analyze the interrelationship of each KD component for quantized DNN training. Our experiments show that even a small teacher model can achieve the same distillation performance as a large teacher model. We also propose the gradual soft loss reduction (GSLR) technique which controls the mixing ratio of hard and soft losses during training for robust KD based QDNN optimization.