Neural network circuit using affine transforms to encode activations

Abstract

Some embodiments provide a neural network inference circuit (NNIC). The NNIC includes a set of core computation circuits for executing a core computation on a set of input values for a neural network computation node to generate a first value. The NNIC includes a set of affine transform circuits for applying an affine transform to the first value to generate a second value. The applied affine transform is a combination of (i) a batch normalization transform, (ii) a learned affine transform, (iii) a weight scale for the computation node, and (iv) an encoding transform such that the second value is encoded for input to an activation function. The NNIC includes a set of activation circuits for applying the activation function to the second value to generate an output value for the computation node.