Normalization in deep convolutional neural networks

Abstract

A device for machine learning is provided, including a first neural network layer, a second neural network layer with a normalization layer arranged in between. The normalization layer is configured to, when the device is undergoing training on a batch of training samples, receive multiple outputs of the first neural network layer for a plurality of training samples of the batch, each output comprising multiple data values for different indices on a first dimension and a second dimension; group the outputs into multiple groups based on the indices on the first and second dimensions; form a normalization output for each group which are provided as input to the second neural network layer. According to the application, the training of a deep convolutional neural network with good performance that performs stably at different batch sizes and is generalizable to multiple vision tasks is achieved, thereby improving the performance of the training.