Marvell Asia Pte., Ltd.

Intel Corporation

Ulf Hanebutte

Avinash Sodani

Senad Durakovic

Chia-Hsin Chen

Chien-Chun Chou

Hamid Reza Ghasemi

Harri Hakkarainen

Eugene Gorbatov

Howard S David

Hongzhong Zheng

Rahul Khanna

David Pardo Keppel

Mario Flajslik

James Dinan

Fu-Hwa Wang

Minh T Le

Mohana Tandyala

Veena Karthikeyan

Pranav Jonnalagadda

Rishan Tan

Vincent J Zimmer

Paul S Diefenbaugh

Shreekant S Thakkar

Keith D Underwood

Jiewen Yao

Vasudevan Srinivasan

James W Alexander

Suneeta Sah

Brent D Chartrand

Richard A Forand

Ram V Chary

Pradeep Sebestian

Milan Milenkovic

Jeremy J Lees

Soumya Gollamudi

Yao Chou

Nikhil Bernard John Stephen

Pradeep Sebastian

Shubha Kumbadakone

A method includes receiving a high-level function in a first high-level code; compiling the high-level function into a first set of low-level instructions to be executed on a hardware or a simulator; transmitting the first set of low-level instructions to the hardware or the simulator; receiving a plurality of statistical data generated by the hardware or the simulator in response to execution of the first set of low-level instructions, wherein the plurality of statistical data is performance related; determining whether to make changes to the compilation associated with the high-level function in the first high-level code based on the plurality of statistical data; recompiling the high-level function into a second set of low-level instructions to be executed on the hardware or the simulator based on the changes to the compilation; and transmitting the second set of low-level instructions to the hardware or the simulator.

Method and system for code optimization based on statistical data

A system includes a compiler including a plurality of compiler blocks. The compiler blocks of the plurality of compiler blocks are compossible. The compiler is configured to identify one or more resources in a hardware to execute a set of low-level instructions that is generated from a high-level function in a high-level code. The compiler is further configured to determine one or more processing operations to be performed that is associated with the high-level function in the high-level code. The determining of the one or more processing operations occurs based on architecture of the hardware. The compiler is configured to compile the high-level function in the high-level code of the application into the set of low-level instructions to be executed on the hardware.

Multistage compiler architecture

A method includes receiving a machine learning (ML) network model in high-level code; generating an internal representation (IR), the IR is mapped to components in a multi-processing tile device; determining whether a first processing tile with a first on-chip memory (OCM) has a same dimension for an input/output tensor data as a second processing tile with a second OCM performing a same primitive function based on the IR; allocating a same memory address range within the first and the second OCM for the same primitive function if the first processing tile has the same dimension for the input/output tensor data as the second processing tile for the same primitive function; linking the memory address range of the first OCM to the memory address range of the second OCM to form a grouped memory space within the first and the second OCM respectively; and compiling low-level instructions based on the linking.

Method and system for memory management within machine learning inference engine

A system and method for splitting a machine learning (ML) graph is disclosed. The system includes a compiler configured to receive an ML model. The compiler generates a graph associated with the ML model, wherein the graph is an internal representation of the ML model. The graph is partitioned into a first subgraph and a second subgraph. The first subgraph is associated with an ML hardware, an ML emulator, or a combination thereof, and the second subgraph is associated with a processor different from the ML hardware. A set of low-level instructions associated with the first subgraph is generated. One or more resources in the ML hardware is identified to execute the set of low-level instructions associated with the first subgraph.

Method and apparatus for ML graphs by a compiler

A new approach is proposed to support correlating high-level code with low-level instructions of an application running on a hardware. A compiler that compiles a high-level function in the high-level code of the application into a set of low-level instructions to be executed on the hardware is configured to utilize one or more reserved fields of the set of low-level instructions to incorporate one or more IDs and an actionable item. The IDs are mapped to the high-level function, wherein such mapping is programmable by the compiler. Based on the mapped IDs and the actionable item incorporated in the set of the low-level instructions, the runtime performance of the application on the hardware can be monitored and profiled and issues related to the high-level code of the application can be identified for debugging purposes.

Method and apparatus for correlating high-level code with low-level instructions for machine learning applications

A programmable hardware system for machine learning (ML) operations includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.

Instruction set architecture (ISA) format for multiple instruction set architectures in machine learning inference engine

A method includes determining that an amount of data external to an inference engine to be transmitted for local storage/processing by a first processing tile exceeds an available space at a first OCM of the first processing tile; receiving a first portion of the data at the first processing tile; transmitting the first portion of the data to a second OCM of a second processing tile for temporary local storage (the second processing tile is within the inference engine); receiving and storing a second portion of the data at the first OCM; processing the second portion of the data at the first processing tile by at least a first processing element; receiving and storing the first portion of the data at the first OCM of the first processing tile from the second processing tile prior to the first portion of data is needed by the first processing tile.

Method and system to expand accessible on-chip memory (OCM) of an inference engine

A method includes receiving a set of data. The set of data is divided into a plurality of data portions. The method includes transmitting the plurality of data portions to a plurality of processing tiles, wherein each data portion of the plurality of data portions is associated with a processing tile of a plurality of tiles. Each processing tile of the plurality of tiles performs at least one local operation on its respective data portion to form a local result. The method includes exchanging local results between the plurality of processing tiles. Moreover, the method includes calculating a global value based on the local results. The method further includes performing at least one local operation by each processing tile of the plurality of tiles on its respective data portion based on the global value to form a computed result.

Method and apparatus for performing machine learning operations in parallel on machine learning hardware

A programmable hardware system for machine learning (ML) includes a core and a streaming engine. The core receives a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning. The core transmits a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof. The first subset of commands and the associated data are passed through via a function call. The streaming engine is coupled to the core and receives the first subset of commands and the associated data from the core. The streaming engine streams a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.

Streaming engine for machine learning architecture

A processing unit to support inference acceleration for machine learning (ML) comprises an inline post processing unit configured to accept and maintain one or more lookup tables for performing a tanh and/or sigmoid operation/function. The inline post processing unit is further configured to accept data from a set of registers configured to maintain output from a processing block instead of streaming the data from an on-chip memory (OCM), perform the tanh and/or sigmoid operation on each element of the data from the processing block on a per-element basis via the one or more lookup tables, and stream post processing result of the per-element tanh and/or sigmoid operation back to the OCM after the tanh and/or sigmoid operation is complete.

Architecture to support tanh and sigmoid operations for inference acceleration in machine learning

A method includes receiving a first set of data. The method also includes receiving an instruction to determine a largest value within the first set of data. The first set of data is divided into a first plurality of data portions based on a hardware architecture of a first plurality of processing elements. The first plurality of data portions is mapped to the first plurality of processing elements. Each data portion of the first plurality of data portions is mapped exclusively to a processing element of the first plurality of processing elements. Each data portion of the first plurality of data portions is processed by its respective processing element to identify a largest value from each data portion of the first plurality of data portions, wherein the processing forms a first output data comprising the largest value from the each data portion of the first plurality of data portions.

A system to support validation and debugging of compiled low-level instructions for a machine learning (ML) network model on an ML-specific hardware. A compiler identifies well-defined boundaries in the ML network model based on primitives used to generate low-level instructions for the hardware. The ML network model is partitioned into units/layers/sub-graphs based on the plurality of well-defined boundaries. The compiler then generates an internal representation for each of the units wherein the internal representation is mapped to components in the hardware. Each of the units is compiled into a first set to be executed on the ML-specific hardware and a second set to be executed on a second computing device. The output results from executing the two sets of low-level instructions are compared to validate the first set of low-level instructions. If the outputs do not match fully, the first set of low-level instructions is debugged and recompiled.

Method and apparatus for compiler and low-level instruction validation of machine learning operations on hardware

A method of converting a data stored in a memory from a first format to a second format is disclosed. The method includes extending a number of bits in the data stored in a double data rate (DDR) memory by one bit to form an extended data. The method further includes determining whether the data stored in the DDR is signed or unsigned data. Moreover, responsive to determining that the data is signed, a sign value is added to the most significant bit of the extended data and the data is copied to lower order bits of the extended data. Responsive to determining that the data is unsigned, the data is copied to lower order bits of the extended data and the most significant bit is set to an unsigned value, e.g., zero. The extended data is stored in an on-chip memory (OCM) of a processing tile of a machine learning computer array.

System and method for INT9 quantization

A method includes receiving a high-level function in a high-level code of an application; identifying resources in a hardware to execute a set of low-level instructions that is generated from the high-level function in the high-level code; compiling the high-level function in the high-level code of the application into the set of low-level instructions to be executed on the hardware; and generating a plurality of structured metadata associated with allocation of resources in the hardware to execute the set of low-level instructions.

Growing community of inventors

Gig Harbor, WA, United States of America

Ulf Hanebutte