Marvell Asia Pte., Ltd.

Chia-Hsin Chen

Avinash Sodani

Ulf Hanebutte

Senad Durakovic

Hamid Reza Ghasemi

Srinivas Sripada

Nikhil Jayakumar

Atul Bhattarai

Ramacharan Sundararaman

Saurabh Shrivastava

Heeloo Chung

Sowmya Hotha

Dan Tu

Rishan Tan

Enrique Musoll

Enric Musoll

Soumya Gollamudi

A programmable hardware system for machine learning (ML) operations includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.

Instruction set architecture (ISA) format for multiple instruction set architectures in machine learning inference engine

A programmable hardware system for machine learning (ML) includes a core and a streaming engine. The core receives a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning. The core transmits a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof. The first subset of commands and the associated data are passed through via a function call. The streaming engine is coupled to the core and receives the first subset of commands and the associated data from the core. The streaming engine streams a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.

Streaming engine for machine learning architecture

A processing unit to support inference acceleration for machine learning (ML) comprises an inline post processing unit configured to accept and maintain one or more lookup tables for performing a tanh and/or sigmoid operation/function. The inline post processing unit is further configured to accept data from a set of registers configured to maintain output from a processing block instead of streaming the data from an on-chip memory (OCM), perform the tanh and/or sigmoid operation on each element of the data from the processing block on a per-element basis via the one or more lookup tables, and stream post processing result of the per-element tanh and/or sigmoid operation back to the OCM after the tanh and/or sigmoid operation is complete.

Architecture to support tanh and sigmoid operations for inference acceleration in machine learning

A system includes a first and a second group of cores in a multicore system. Each core of the first/second group is configured to process data. Each core within the first/second group is configured to enter into an idle state in response to being idle for a first/second period of time respectively. Every idle core in the first/second group is configured to transition out of the idle state and into an operational mode in response to receiving a signal having a first/second value respectively and further in response to having a pending operation to process.

Power management and staggering transitioning from idle mode to operational mode

A method of converting a data stored in a memory from a first format to a second format is disclosed. The method includes extending a number of bits in the data stored in a double data rate (DDR) memory by one bit to form an extended data. The method further includes determining whether the data stored in the DDR is signed or unsigned data. Moreover, responsive to determining that the data is signed, a sign value is added to the most significant bit of the extended data and the data is copied to lower order bits of the extended data. Responsive to determining that the data is unsigned, the data is copied to lower order bits of the extended data and the most significant bit is set to an unsigned value, e.g., zero. The extended data is stored in an on-chip memory (OCM) of a processing tile of a machine learning computer array.

System and method for INT9 quantization

A power throttling engine includes a register configured to receive a power throttling signal. The power throttling engine further includes a decoder configured to generate a vector based on a value of the power throttling signal. The value of the power throttling signal is an amount of power throttling of a device. The power throttling engine further includes a clock gating logic configured to receive the vector and further configured to receive a clocking signal. The clock gating logic is configured to remove clock edges of the clocking signal based on the vector to generate a throttled clocking signal.

System and method to manage power throttling

A new approach contemplates systems and methods to support control of power consumption of a memory on a chip by throttling port access requests to the memory via a memory arbiter based on a one or more programmable parameters. The memory arbiter is configured to restrict the number of ports being used to access the memory at the same time to be less than the available ports of the memory, thereby enabling adaptive power control of the chip. Two port throttling schemes are enabled&#x2014;strict port throttling, which throttles the number of ports granted for memory access to be no more than a user-configured maximum throttle port number, and leaky bucket port throttling, which throttles the number of ports granted for the memory access down to be within a range based on a number of credit tokens maintained in a credit register.

System and methods for on-chip memory (OCM) port throttling for machine learning operations

Control logic circuitry stores packets in a queue in an order in which the packets are received. A head entry of the queue corresponds to an oldest packet in the order. The control logic circuitry receives flow control information corresponding to multiple target devices including at least a first target device and a second target device. The control logic circuitry determines, using the flow control information, whether the oldest packet stored in the head entry can be transferred to the first target device, and in response to determining that the oldest packet stored in the head entry cannot be transferred to the first target device, i) selects an other entry with an other packet behind the head entry according to the order, and ii) transfers the other packet to the second target device prior to transferring the oldest packet in the head entry to the first target device.

Queueing system with head-of-line block avoidance

A system includes a plurality of cores. Each core includes a processing unit, an on-chip memory (OCM), and an idle detector unit. Data is received and stored in the OCM. Instructions are received to process data in the OCM. The core enters an idle mode if the idle detector unit detects that the core has been idle for a first number of clocking signals. The core receives a command to process when in idle mode and transitions from the idle mode to an operational mode. A number of no operation (No-Op) commands is inserted for each time segment. A No-Op command prevents the core from processing instructions for a certain number of clocking signals. A number of No-Op commands inserted for a first time segment is greater than a number of No-Op commands inserted for a last time segment. After the last time segment no No-Op command is inserted.

Power management and transitioning cores within a multicore system from idle mode to operational mode over a period of time

A processing unit to support inference acceleration for machine learning (ML) comprises an inline post processing unit configured to accept and maintain one or more lookup tables for performing each of one or more non-linear mathematical operations. The inline post processing unit is further configured to accept data from a set of registers maintaining output from a processing block instead of streaming the data from an on-chip memory (OCM), perform the one or more non-linear mathematical operations on elements of the data from the processing block via their corresponding lookup tables, and stream post processing result of the one or more non-linear mathematical operations back to the OCM after the one or more non-linear mathematical operations are complete.

Architecture for table-based mathematical operations for inference acceleration in machine learning

A multi-dimensional mesh architecture is proposed to support transmitting data packets from one source to a plurality of destinations in multicasting or broadcasting modes. Each data packet to be transmitted to the destinations carries a destination mask, wherein each bit in the destination mask represents a corresponding destination processing block in the mesh architecture the data packet is sent to. The data packet traverses through the mesh architecture based on a routing scheme, wherein the data packet first traverses in a first direction across a first set of processing blocks and then traverses in a second direction across a second set of processing blocks to the first destination. During the process, the data packet is only replicated when it reaches a splitting processing block where the paths to different destinations diverge. The original and the replicated data packets are then routed in different directions until they reach their respective destinations.

System and methods for mesh architecture for high bandwidth multicast and broadcast network

A method includes receiving an input data at a FP arithmetic operating unit configured to perform a FP arithmetic operation on the input data. The method further includes determining whether the received input data generates a FP hardware exception responsive to the FP arithmetic operation on the input data, wherein the determining occurs prior to performing the FP arithmetic operation. The method also includes converting a value of the received input data to a modified value responsive to the determining that the received input data generates the FP hardware exception, wherein the converting eliminates generation of the FP hardware exception responsive to the FP arithmetic operation on the input data.

Growing community of inventors

Santa Clara, CA, United States of America

Chia-Hsin Chen