Adaptive search-based allocation of computations to synchronized, interconnected processing elements for implementing machine learning networks

Abstract

A system for compiling a machine learning network for execution on a plurality of interconnected processing elements performs the following. It selects splits (P,Q,K) for individual layers of the machine learning network based on (a) costs of data transfer between processing elements assuming that inputs and outputs of individual layers have a same spatial split (P,Q), and (b) costs of data transfer between processing elements resulting from differences in the spatial splits for inputs and outputs of individual layers. In addition, it selects the same splits (P,Q,K) for layers that have the same spatial size P×Q. It then generates a computer program that allocates computations for executing the machine learning network to the processing elements according to the selected splits for the individual layers.