System and method for runtime optimization of neural networks

Abstract

Systems and methods of optimizing runtime of a neural network (NN) by at least one processor may include: receiving a space of untrained NN architectures, capable of performing a predefined NN function on a given computing device, after being trained; receiving a reference accuracy metric value; evaluating a latency value for each NN architecture of the architecture space; dividing the architecture space to a plurality of groups based on said evaluated latency; performing a search among the plurality of groups, to determine a group G* that corresponds to a minimal evaluated latency, and yet comprises at least one NN architecture that maintains an accuracy metric value that is at least equal to the reference accuracy metric value; and training at least one NN architecture of the determined group G* to perform the NN function.