Hewlett-packard Development Company, L.p.

Google Inc.

Hewlett Packard Enterprise Development LP

Digital Equipment Corporation

Compaq Computer Corporation, Inc.

University of Utah Research Foundation

Compaq Information Technologies Group, L.p.

Skhynix Inc.

Norman Paul Jouppi

Andrew Everett Phelps

Naveen Muralimanohar

Thomas James Norrie

Sheng Li

Parthasarathy Ranganathan

Doe Hyun Yoon

Gregory Michael Thorson

Robert S Schreiber

Nathan Lorenzo Binkert

Moray McLaren

Joel James McCormack

Alan Lynn Davis

Madhusudan K Iyengar

Christopher Gregory Malone

Jung Ho Ahn

Jorge Padilla

Raymond G Beausoleil

Marco Fiorentino

Jonathan Alexander Ross

Keith Farkas

Larry D Seiler

Jichuan Chang

Teckgyu Kang

Reginald Clifford Young

Christopher Aaron Clark

Woon-Seong Kwon

Subramoniam N Iyer

Dan Luu

William Lacy

Nidhi Aggarwal

Yuan Hang Li

Liqun Cheng

Rajeev Balasubramonian

Matthew Leever Hedlund

Aniruddha Nagendran Udipi

Paolo Faraboschi

Michael S Schlansker

Qianfan Xu

Bjarke Hammersholt Roune

Jacob B Leverich

Vaughan Stanton Thomas

Michael Renne Ty Tan

Quoc V Le

William Riis Hamburgen

Michael R Krause

Dwight L Barron

John S Fitch

Nishant P Patil

Gurushankar Rajamani

Jean-Francois C Collard

Melanie Beauchemin

Emad Samadiani

Robert Stephen McNamara

Niladrish Chatterjee

Padam Jain

Connor Burgess

Ian Mclaren

April Marie Slayden

James T Claffey

Yingying Wang

Parthasararthy Ranganathan

Jean-Francois C P Collard

Ke Chen

Chun-Fa Chang

Brian Zhang

Yun Ni

Shih-Yuan Wang

Martin Guy Dixon

Terrel L Morris

Dejan S Milojicic

Daniel J Scales

Melvin K Benedict

Sridhar Lakshmanamurthy

Boon Seong Ang

Vanish Talwar

Olivier Temam

Ruoming Pang

Andrew Christopher Walton

Rajan Mathew Lukose

Kevin T Lim

James M Knittel

William F Edwards

Joshua Rogers Tyler

Mingxing Tan

Daniel Stodolsky

Patrick Charles McGeer

Christopher C Gianos

Ramsey W Haddad

Matteo Monchiero

Robert Nelson Mayo

Erwin Oertli

John H Zurawski

Sundar Rajan

Andrew Li

Jishen Zhao

Judson S Leonard

Yuan Li

Madhu Krishnan Iyengar

Wayne R Mack

Aspects of the disclosure are directed to a heterogeneous machine learning accelerator system with compute and memory nodes connected by high speed chip-to-chip interconnects. While existing remote/disaggregated memory may require memory expansion via remote processing units, aspects of the disclosure add memory nodes into machine learning accelerator clusters via the chip-to-chip interconnects without needing assistance from remote processing units to achieve higher performance, simpler software stack, and/or lower cost. The memory nodes may support prefetch and intelligent compression to enable the use of low cost memory without performance degradation.

Heterogeneous ML accelerator cluster with flexible system resource balance

A vector processing unit is described, and includes processor units that each include multiple processing resources. The processor units are each configured to perform arithmetic operations associated with vectorized computations. The vector processing unit includes a vector memory in data communication with each of the processor units and their respective processing resources. The vector memory includes memory banks configured to store data used by each of the processor units to perform the arithmetic operations. The processor units and the vector memory are tightly coupled within an area of the vector processing unit such that data communications are exchanged at a high bandwidth based on the placement of respective processor units relative to one another, and based on the placement of the vector memory relative to each processor unit.

Vector processing unit

Methods, systems, and apparatus, including computer-readable media, are described for a hardware circuit configured to implement a neural network. The circuit includes a first memory, respective first and second processor cores, and a shared memory. The first memory provides data for performing computations to generate an output for a neural network layer. Each of the first and second cores include a vector memory for storing vector values derived from the data provided by the first memory. The shared memory is disposed generally intermediate the first memory and at least one core and includes: i) a direct memory access (DMA) data path configured to route data between the shared memory and the respective vector memories of the first and second cores and ii) a load-store data path configured to route data between the shared memory and respective vector registers of the first and second cores.

Shared scratchpad memory with parallel load-store

Methods, systems, and apparatus, including an integrated circuit (IC) with a ring-shaped hot spot area. In one aspect, an IC includes a first area along an outside perimeter of a surface of the IC. The first area defines a first inner perimeter. The IC includes a second area that includes a center of the IC and that includes a first set of components. The second area defines a first outer. The IC includes a ring-shaped hot spot area between the first area and the second area. The ring-shaped hot spot area defines a ring outer perimeter that is juxtaposed with the first inner perimeter. The ring-shaped hot spot area defines a ring inner perimeter that is juxtaposed with the first outer perimeter. The ring-shaped hot spot area includes a second set of components that produce more heat than the first set of components.

Integrated circuit with a ring-shaped hot spot area and multidirectional cooling

A method of manufacturing a chip assembly comprises joining an in-process unit to a printed circuit board; reflowing a bonding material disposed between and electrically connecting the in-process unit with the printed circuit board, the bonding material having a first reflow temperature; and then joining a heat distribution device to the plurality of semiconductor chips using a thermal interface material ('TIM') having a second reflow temperature that is lower than the first reflow temperature. The in-process unit further comprises a substrate having an active surface, a passive surface, and contacts exposed at the active surface; an interposer electrically connected to the substrate; a plurality of semiconductor chips overlying the substrate and electrically connected to the substrate through the interposer, and a stiffener overlying the substrate and having an aperture extending therethrough, the plurality of semiconductor chips being positioned within the aperture.

Methods and heat distribution devices for thermal management of chip assemblies

Aspects of the disclosure are directed to a computation unit implementing a systolic array and configured for detecting errors while processing data on the systolic array. Checksum circuit in communication with a systolic array is configured to compute checksums and perform error detection while the systolic array processes input data. Instead of pre-generating checksums in input matrices, input matrices can be directly fed into the systolic array through the checksum circuit. On the output side, the checksum circuit can generate and compare checksums with checksums in an output matrix generated by the systolic array. Error checking the operations to generate the output matrix can be performed without delaying the operations of the systolic array, and without preprocessing the input matrices.

Error checking for systolic array computation

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining an architecture for a task neural network that is configured to perform a particular machine learning task on a target set of hardware resources. When deployed on a target set of hardware, such as a collection of datacenter accelerators, the task neural network may be capable of performing the particular machine learning task with enhanced accuracy and speed.

Hardware-optimized neural architecture search

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for scheduling operations represented as a computational graph on a distributed computing network. A method includes: receiving data representing operations to be executed in order to perform a job on a plurality of hardware accelerators of a plurality of different accelerator types; generating, for the job and from at least the data representing the operations, features that represent a predicted performance for the job on hardware accelerators of the plurality of different accelerator types; generating, from the features, a respective predicted performance metric for the job for each of the plurality of different accelerator types according to a performance objective function; and providing, to a scheduling system, one or more recommendations for scheduling the job on one or more recommended types of hardware accelerators.

Generating recommendations based on performance metrics for scheduling jobs on distributed computing devices

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. The matrix multiply unit may include cells arranged in columns of the systolic array. Two chains of weight shift registers per column of the systolic array are in the matrix multiply unit. Each weight shift register is connected to only one chain and each cell is connected to only one weight shift register. A weight matrix register per cell is configured to store a weight input received from a weight shift register. A multiply unit is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

Low latency matrix multiply unit

Methods, systems, and apparatus for performing a matrix multiplication using a hardware circuit are described. An example method begins by obtaining an input activation value and a weight input value in a first floating point format. The input activation value and the weight input value are multiplied to generate a product value in a second floating point format that has higher precision than the first floating point format. A partial sum value is obtained in a third floating point format that has a higher precision than the first floating point format. The partial sum value and the product value are combined to generate an updated partial sum value that has the third floating point format.

Performing matrix multiplication in hardware

Methods, systems, and apparatus, including computer-readable media, are described for performing vector reductions using a shared scratchpad memory of a hardware circuit having processor cores that communicate with the shared memory. For each of the processor cores, a respective vector of values is generated based on computations performed at the processor core. The shared memory receives the respective vectors of values from respective resources of the processor cores using a direct memory access (DMA) data path of the shared memory. The shared memory performs an accumulation operation on the respective vectors of values using an operator unit coupled to the shared memory. The operator unit is configured to accumulate values based on arithmetic operations encoded at the operator unit. A result vector is generated based on performing the accumulation operation using the respective vectors of values.

Vector reductions using shared scratchpad memory

Methods, systems, and apparatus for updating machine learning models to improve locality are described. In one aspect, a method includes receiving data of a machine learning model. The data represents operations of the machine learning model and data dependencies between the operations. Data specifying characteristics of a memory hierarchy for a machine learning processor on which the machine learning model is going to be deployed is received. The memory hierarchy includes multiple memories at multiple memory levels for storing machine learning data used by the machine learning processor when performing machine learning computations using the machine learning model. An updated machine learning model is generated by modifying the operations and control dependencies of the machine learning model to account for the characteristics of the memory hierarchy. Machine learning computations are performed using the updated machine learning model.

Modifying machine learning models to improve locality

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. Each cell of the matrix multiply includes: a weight matrix register configured to receive a weight input from either a transposed or a non-transposed weight shift register; a transposed weight shift register configured to receive a weight input from a horizontal direction to be stored in the weight matrix register; a non-transposed weight shift register configured to receive a weight input from a vertical direction to be stored in the weight matrix register; and a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

Growing community of inventors

Palo Alto, CA, United States of America

Norman Paul Jouppi