Method and system for memory management within machine learning inference engine

Abstract

A method includes in a first iteration receiving a machine learning (ML) network model comprising a plurality of ML operations in high-level code; generating an internal representation (IR) for the ML network model, the IR mapped to one or more components in a multi-processing tile device; generating primitive functions based on the IR; generating an allocation list based on the primitive functions; determining when a tensor data within the allocation list is no longer needed; and inserting a deallocation function associated with the tensor data to the primitive functions to form an updated primitive functions, the inserting frees up a memory space associated with the tensor data when the tensor is no longer needed; and in a second iteration generating a compilation of the updated primitive functions to map the IR to the one or more components in the multi-processing device, wherein the compilation is a low-level instructions.