Analog hardware implementation of activation functions

Abstract

An analog neural network including a hardware activation function is provided. A layer of the analog neural network includes a sequence of processing elements that receives analog signals, perform MAC operations on the analog signals, and generates analog outputs. The analog outputs are provided to an analog circuitry that can apply an activation function on the analog outputs. The output of the analog circuitry are also analog signals, which can further be provided to the next layer in the network. The analog circuitry may include a differential pair of transistors to compute the tanh activation function. Alternatively, the analog circuitry may include a comparator and multiplexer to compute the ReLU activation function. Compared with digital implementation of activation functions, the analog circuitry eliminates the need of converting the analog outputs of the layer to digital signals and the need of converting the result of the activation function to analog signals.