Auxiliary network design for local learning in deep neural networks

Abstract

The training of deep neural networks utilizes the backpropagation algorithm which consists of the forward pass, backward pass and parameters update. The output of a certain layer is produced based on the output of its lower layers in a sequential manner, and the gradients can only flow back layer by layer. This forces the majority of the network to be idle during the training process and hence leads to inefficiency. This is recognized as forward, backward and update lockings. To break the lockings, various methods of decoupled learning have been investigated. Currently, these methods either lead to significant drop in accuracy performance or suffer from dramatic increase in memory usage. To remove these limitations, in this Final Year Project, a new form of decoupled learning, named decoupled neural network training scheme with re-computation and weight prediction (DTRP) is proposed. The proposed method splits a neural network into several modules and trains them synchronously on different workers. In particular, re-computation is adopted to solve the memory explosion problem. A weight prediction scheme is proposed to deal with the weight delay caused by re-computation. To execute weight prediction, several weight predictors are proposed. A batch compensation scheme is also explored which allows the proposed DTRP to run faster. Experiments are conducted on various Convolutional Neural Networks to perform image classification task, which shows comparable or better results against the state-of-art methods and the backpropagation. The experiments also reveal that the memory explosion problem is effectively solved, and a significant acceleration is achieved. Moreover, experiments show that the proposed DTRP can be applied to train very wide networks as well as extremely deep networks.