Enhancing memory utilization and throughput in executing a computational graph

Abstract

Disclosed is a method for enhancing memory utilization and throughput of a computing platform in training a deep neural network (DNN). The critical features of the method includes: calculating a memory size for every operation in a computational graph, storing the operations in the computational graph in multiple groups with the operations in each group being executable in parallel and a total memory size less than a memory threshold of a computational device, sequentially selecting a group and updating a prefetched group buffer, and simultaneously executing the group and prefetching data for a group in the prefetched group buffer to the corresponding computational device when the prefetched group buffer is update. Because of group execution and data prefetch, the memory utilization is optimized and the throughput is significantly increased to eliminate issues of out-of-memory and thrashing.