Marvell Asia Pte., Ltd.

Senad Durakovic

Ulf Hanebutte

Avinash Sodani

Chia-Hsin Chen

Chien-Chun Chou

Hamid Reza Ghasemi

Harri Hakkarainen

Fu-Hwa Wang

Gopal Nalamalapu

Mohana Tandyala

Veena Karthikeyan

Pranav Jonnalagadda

Rishan Tan

Yao Chou

Nikhil Bernard John Stephen

A method includes receiving a high-level function in a first high-level code; compiling the high-level function into a first set of low-level instructions to be executed on a hardware or a simulator; transmitting the first set of low-level instructions to the hardware or the simulator; receiving a plurality of statistical data generated by the hardware or the simulator in response to execution of the first set of low-level instructions, wherein the plurality of statistical data is performance related; determining whether to make changes to the compilation associated with the high-level function in the first high-level code based on the plurality of statistical data; recompiling the high-level function into a second set of low-level instructions to be executed on the hardware or the simulator based on the changes to the compilation; and transmitting the second set of low-level instructions to the hardware or the simulator.

Method and system for code optimization based on statistical data

A system includes a compiler including a plurality of compiler blocks. The compiler blocks of the plurality of compiler blocks are compossible. The compiler is configured to identify one or more resources in a hardware to execute a set of low-level instructions that is generated from a high-level function in a high-level code. The compiler is further configured to determine one or more processing operations to be performed that is associated with the high-level function in the high-level code. The determining of the one or more processing operations occurs based on architecture of the hardware. The compiler is configured to compile the high-level function in the high-level code of the application into the set of low-level instructions to be executed on the hardware.

Multistage compiler architecture

A method includes receiving a machine learning (ML) network model in high-level code; generating an internal representation (IR), the IR is mapped to components in a multi-processing tile device; determining whether a first processing tile with a first on-chip memory (OCM) has a same dimension for an input/output tensor data as a second processing tile with a second OCM performing a same primitive function based on the IR; allocating a same memory address range within the first and the second OCM for the same primitive function if the first processing tile has the same dimension for the input/output tensor data as the second processing tile for the same primitive function; linking the memory address range of the first OCM to the memory address range of the second OCM to form a grouped memory space within the first and the second OCM respectively; and compiling low-level instructions based on the linking.

Method and system for memory management within machine learning inference engine

A system and method for splitting a machine learning (ML) graph is disclosed. The system includes a compiler configured to receive an ML model. The compiler generates a graph associated with the ML model, wherein the graph is an internal representation of the ML model. The graph is partitioned into a first subgraph and a second subgraph. The first subgraph is associated with an ML hardware, an ML emulator, or a combination thereof, and the second subgraph is associated with a processor different from the ML hardware. A set of low-level instructions associated with the first subgraph is generated. One or more resources in the ML hardware is identified to execute the set of low-level instructions associated with the first subgraph.

Method and apparatus for ML graphs by a compiler

A new approach is proposed to support correlating high-level code with low-level instructions of an application running on a hardware. A compiler that compiles a high-level function in the high-level code of the application into a set of low-level instructions to be executed on the hardware is configured to utilize one or more reserved fields of the set of low-level instructions to incorporate one or more IDs and an actionable item. The IDs are mapped to the high-level function, wherein such mapping is programmable by the compiler. Based on the mapped IDs and the actionable item incorporated in the set of the low-level instructions, the runtime performance of the application on the hardware can be monitored and profiled and issues related to the high-level code of the application can be identified for debugging purposes.

Method and apparatus for correlating high-level code with low-level instructions for machine learning applications

A programmable hardware system for machine learning (ML) operations includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.

Instruction set architecture (ISA) format for multiple instruction set architectures in machine learning inference engine

A method includes determining that an amount of data external to an inference engine to be transmitted for local storage/processing by a first processing tile exceeds an available space at a first OCM of the first processing tile; receiving a first portion of the data at the first processing tile; transmitting the first portion of the data to a second OCM of a second processing tile for temporary local storage (the second processing tile is within the inference engine); receiving and storing a second portion of the data at the first OCM; processing the second portion of the data at the first processing tile by at least a first processing element; receiving and storing the first portion of the data at the first OCM of the first processing tile from the second processing tile prior to the first portion of data is needed by the first processing tile.

Method and system to expand accessible on-chip memory (OCM) of an inference engine

A programmable hardware system for machine learning (ML) includes a core and a streaming engine. The core receives a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning. The core transmits a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof. The first subset of commands and the associated data are passed through via a function call. The streaming engine is coupled to the core and receives the first subset of commands and the associated data from the core. The streaming engine streams a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.

Streaming engine for machine learning architecture

A system to support a machine learning (ML) operation comprises an array-based inference engine comprising a plurality of processing tiles each comprising at least one or more of an on-chip memory (OCM) configured to maintain data for local access by components in the processing tile and one or more processing units configured to perform one or more computation tasks on the data in the OCM by executing a set of task instructions. The system also comprises a data streaming engine configured to stream data between a memory and the OCMs and an instruction streaming engine configured to distribute said set of task instructions to the corresponding processing tiles to control their operations and to synchronize said set of task instructions to be executed by each processing tile, respectively, to wait current certain task at each processing tile to finish before starting a new one.

Architecture to support color scheme-based synchronization for machine learning

A system to support validation and debugging of compiled low-level instructions for a machine learning (ML) network model on an ML-specific hardware. A compiler identifies well-defined boundaries in the ML network model based on primitives used to generate low-level instructions for the hardware. The ML network model is partitioned into units/layers/sub-graphs based on the plurality of well-defined boundaries. The compiler then generates an internal representation for each of the units wherein the internal representation is mapped to components in the hardware. Each of the units is compiled into a first set to be executed on the ML-specific hardware and a second set to be executed on a second computing device. The output results from executing the two sets of low-level instructions are compared to validate the first set of low-level instructions. If the outputs do not match fully, the first set of low-level instructions is debugged and recompiled.

Method and apparatus for compiler and low-level instruction validation of machine learning operations on hardware

A method includes receiving a high-level function in a high-level code of an application; identifying resources in a hardware to execute a set of low-level instructions that is generated from the high-level function in the high-level code; compiling the high-level function in the high-level code of the application into the set of low-level instructions to be executed on the hardware; and generating a plurality of structured metadata associated with allocation of resources in the hardware to execute the set of low-level instructions.

Method and apparatus for generating metadata by a compiler

A method includes receiving a high-level function in a high-level code of an application is received. The method also include identifying resources in a hardware to execute a set of low-level instructions that is generated from the high-level function in the high-level code. One or more processing operations are determined to be performed that is associated with the high-level function in the high-level code. The determining of the one or more processing operations occurs based on architecture of the hardware. The high-level function in the high-level code of the application is compiled into the set of low-level instructions to be executed on the hardware. A plurality of structured metadata is generated and includes information associated with the determining resources in the hardware and further includes information associated with the determining one or more processing operations.

A programmable hardware system for machine learning (ML) includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.

Architecture of crossbar of inference engine

Single instruction set architecture (ISA) format for multiple ISAS in machine learning inference engine

A processing unit of an inference engine for machine learning (ML) includes a first data load steamer, a second data load streamer, an operator component, and a store streamer. The first data load streamer streams a first data stream from an on-chip memory (OCM) to the operator component. The second data load streamer streams a second data stream from the OCM to the operator component. The operator component performs a matrix operation on the first data stream and the second data stream. The store streamer receives a data output stream from the operator component and to store the data output stream in a buffer.

Growing community of inventors

Palo Alto, CA, United States of America

Senad Durakovic