Machine learning through multiple layers of novel machine trained processing nodes

Abstract

Some embodiments of the invention provide efficient, expressive machined-trained networks for performing machine learning. The machine-trained (MT) networks of some embodiments use novel processing nodes with novel activation functions that allow the MT network to efficiently define with fewer processing node layers a complex mathematical expression that solves a particular problem (e.g., face recognition, speech recognition, etc.). In some embodiments, the same activation function (e.g., a cup function) is used for numerous processing nodes of the MT network, but through the machine learning, this activation function is configured differently for different processing nodes so that different nodes can emulate or implement two or more different functions (e.g., two or more Boolean logical operators, such as XOR and AND). The activation function in some embodiments is a periodic function that can be configured to implement different functions (e.g., different sinusoidal functions).