Improving Gradient Paths for Binary Convolutional Neural Networks

Abstract

Our starting point is a closer investigation of Bi-Real ResNet [34]. In our investigation of Bi-Real ResNet, we believe that the superiority of Bi-Real ResNet over binary ResNet requires a different explanation rather than being attributed to the representational capability. Instead, we study the gradient paths rather than representational capability for BCNNs. To our best knowledge, this is the first work to consider gradient paths for BCNNs. Improving gradient paths is realized by reducing the smallest number of operations to compute gradient backpropagation for a gradient path. Regarding Bi-Real ResNet and BinaryDenseNet, the error of BCNNs decreases when the increased shortcuts improve gradient paths. In addition, we design two architectures by improving gradient paths for BCNNs: 1. Improving Gradient Paths for binary ResNet (IGP-ResNet), and 2. Improving Gradient Paths for binary DenseNet (IGP-DenseNet). Specifically, the Top-1 error of proposed IGP-ResNet37(41) and IGP-DenseNet51(53) on ImageNet gets lower than Bi-Real ResNet18(64) and BinaryDenseNet51(32) by 3.29% and 1.41%, respectively, with almost the same computational complexity.