Systems and methods for intelligently folding normalization and weight parameters during a training of an artificial neural network

Abstract

Systems and methods for generating a hardware-executable artificial neural network are disclosed. A training system accesses a network graph representing a pre-trained artificial neural network including a plurality of computational nodes. The training system transforms the network graph by replacing at least one computational node with a system-generated node that combines computational operations and normalization operations. During retraining of the artificial neural network, parameters associated with the normalization operations and the computational operations are folded to generate integrated parameters including a folded weight matrix and a folded bias matrix. The integrated parameters are updated during training and scaled according to a weight-representation heuristic. Output activations associated with the integrated node may also be scaled based on a composite scaling factor that improves signal-to-noise characteristics. Based on the retraining, a hardware-execution graph corresponding to the artificial neural network is generated and compiled into instructions executable by a matrix multiply accelerator or other integrated circuit configured to execute the artificial neural network.