Nvidia Corporation

Intel Corporation

Manan Patel

Jack H Choquette

Shirish C Gadre

Olivier Giroux

Ronny Krashinsky

Steven James Heinrich

Alan Kaatz

Lacky V Shah

John H Edmondson

Ming Yiu Siu

Alexander Lev Minkin

Xiaogang Qiu

Omkar Paranjape

Gentaro Hirota

Andrew Robert Kerr

Poornachandra Rao

Ravi Kiran Manyam

Wish Gandhi

Prakash Bangalore Prabhakar

Apoorv Parle

Praveen Kaushik

Steve Heinrich

Sandeep Navada

Jerome Francis Duluk, Jr

Wishwesh Anil Gandhi

Ganesh Venkatesh

Luke Durant

Brian S Pharris

Ajay Sudarshan Tirumala

John Tran

Jorge Albericio Latorre

Ze Long

Jeff Tuckey

Greg Palmer

Rajballav Dash

Srinivas Santosh Kumar Madugula

Raghuram L

Eric Werness

Naman Govil

Shayani Deb

Ivan Tanasic

Mark Douglas Hummel

Nirmal Raj Saxena

Sachin Satish Idgunji

Naveen Cherukuri

Michael A Parker

Peter C Mills

Christian Rouet

Michael Alan Fetterman

Robert Ohannessian

Daniel P Wilde

Rajeshwaran Selvanesan

Liang Yin

Jonathan W Liu

Jeffrey Michael Pool

Andrei Khodakovsky

Praveen Joginipally

Jeff Pool

Feng Xie

Vaishali Kulkarni

James P Reilley

Ming Liang Milton Lei

Adam Hendrickson

Jeffery Michael Schottmiller

Amanpreet Grewal

David Charles Patrick

Ewa Kubalska

Nicholas E Duresky

Saurabh Hukerikar

Oliver Giroux

Shekhar Vasant Divekar

Abhijeet Ghadge

Yiyang Feng

Paul Racunas

Alex Gu

Sunil C Chaudhari

Jeff Schottmiller

Pritha Ghoshal

Various embodiments include techniques for performing self-synchronizing remote memory operations in a multiprocessor computing system. During a remote memory operation in the multiprocessor computing system, a source processing unit transmits multiple segments of data to a destination processing. For each segment of data, the source processing unit transmits a remote memory operation to the destination processing unit that includes associated metadata that identifies the memory location of a corresponding synchronization object. The remote memory operation along with the metadata is transmitted as a single unit to the destination processing unit. The destination processing unit splits the operation into the remote memory operation and the memory synchronization operation. As a result, the source processing unit avoids the need to perform a separate memory synchronization operation, thereby reducing inter-processor communications and increasing performance of remote memory operations.

Self-synchronizing remote memory operations in a multiprocessor system

A first instruction to load a first data into a first register file associated with a first subpartition unit is received. A second instruction to load the first data into a second register file associated with a second subpartition unit is received. The first instruction and the second instruction are coalesced into a first entry of a request coalescer based on instruction identifiers. The first entry is associated with the first data. Responsive to a determination that the first data is available in the cache, the first data is multicast from the cache to the first register file and the second register file.

Systems and methods for multicasting data

A new level(s) of hierarchy—Cooperate Group Arrays (CGAs)—and an associated new hardware-based work distribution/execution model is described. A CGA is a grid of thread blocks (also referred to as cooperative thread arrays (CTAs)). CGAs provide co-scheduling, e.g., control over where CTAs are placed/executed in a processor (such as a GPU), relative to the memory required by an application and relative to each other. Hardware support for such CGAs guarantees concurrency and enables applications to see more data locality, reduced latency, and better synchronization between all the threads in tightly cooperating collections of CTAs programmably distributed across different (e.g., hierarchical) hardware domains or partitions.

Cooperative group arrays

Distributed shared memory (DSMEM) comprises blocks of memory that are distributed or scattered across a processor (such as a GPU). Threads executing on a processing core local to one memory block are able to access a memory block local to a different processing core. In one embodiment, shared access to these DSMEM allocations distributed across a collection of processing cores is implemented by communications between the processing cores. Such distributed shared memory provides very low latency memory access for processing cores located in proximity to the memory blocks, and also provides a way for more distant processing cores to also access the memory blocks in a manner and using interconnects that do not interfere with the processing cores' access to main or global memory such as hacked by an L2 cache. Such distributed shared memory supports cooperative parallelism and strong scaling across multiple processing cores by permitting data sharing and communications previously possible only within the same processing core.

Distributed shared memory

A parallel processing unit comprises a plurality of processors each being coupled to a memory access hardware circuitry. Each memory access hardware circuitry is configured to receive, from the coupled processor, a memory access request specifying a coordinate of a multidimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuitry each coupled to a respective one of the processors; and, in response to the memory access request, translate the coordinate of the multidimensional data structure into plural memory addresses for the multidimensional data structure and using the plural memory addresses, asynchronously transfer at least a portion of the multidimensional data structure for processing by at least the coupled processor. The memory locations may be in the shared memory of the coupled processor and/or an external memory.

Method and apparatus for efficient access to multidimensional data structures and/or other large data blocks

This specification describes a programmatic multicast technique enabling one thread (for example, in a cooperative group array (CGA) on a GPU) to request data on behalf of one or more other threads (for example, executing on respective processor cores of the GPU). The multicast is supported by tracking circuitry that interfaces between multicast requests received from processor cores and the available memory. The multicast is designed to reduce cache (for example, layer 2 cache) bandwidth utilization enabling strong scaling and smaller tile sizes.

Programmatically controlled data multicasting across multiple compute engines

A technique for block data transfer is disclosed that reduces data transfer and memory access overheads and significantly reduces multiprocessor activity and energy consumption. Threads executing on a multiprocessor needing data stored in global memory can request and store the needed data in on-chip shared memory, which can be accessed by the threads multiple times. The data can be loaded from global memory and stored in shared memory using an instruction which directs the data into the shared memory without storing the data in registers and/or cache memory of the multiprocessor during the data transfer.

Techniques for efficiently transferring data to a processor

Various embodiments include a parallel processing computer system that detects memory errors as a memory client loads data from memory and disables the memory client from storing data to memory, thereby reducing the likelihood that the memory error propagates to other memory clients. The memory client initiates a stall sequence, while other memory clients continue to execute instructions and the memory continues to service memory load and store operations. When a memory error is detected, a specific bit pattern is stored in conjunction with the data associated with the memory error. When the data is copied from one memory to another memory, the specific bit pattern is also copied, in order to identify the data as having a memory error.

Error containment for enabling local checkpoint and recovery

Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices.

Managing data sparsity for neural networks

Compressed data is oftentimes beneficial for reducing the computing resources required, for example, to transmit and store data. The compression of data is particularly useful when dealing with sparse data (data that includes numerous zeros or near-zero values) and only non-zero values above a certain threshold have significance. When dealing with compressed data, oftentimes the data needs to be decompressed for processing (e.g., by deep learning networks or other applications configured to operate on sparse, or other uncompressed data). Instructions are disclosed for supporting the decompression of compressed data by a processing unit such as a CPU and GPU.

Decompression techniques for processing compressed data suitable for artificial neural networks

An integrated circuit such as, for example a graphics processing unit (GPU), includes a dynamic power controller for adjusting operating voltage and/or frequency. The controller may receive current power used by the integrated circuit and a predicted power determined based on instructions pending in a plurality of processors. The controller determines adjustments that need to be made to the operating voltage and/or frequency to minimize the difference between the current power and the predicted power. An in-system reinforced learning mechanism is included to self-tune parameters of the controller.

Techniques for efficiently operating a processing system based on energy characteristics of instructions and machine learning

A just-in-time (JIT) compiler binds constants to specific memory locations at runtime. The JIT compiler parses program code derived from a multithreaded application and identifies an instruction that references a uniform constant. The JIT compiler then determines a chain of pointers that originates within a root table specified in the multithreaded application and terminates at the uniform constant. The JIT compiler generates additional instructions for traversing the chain of pointers and inserts these instructions into the program code. A parallel processor executes this compiled code and, in doing so, causes a thread to traverse the chain of pointers and bind the uniform constant to a uniform register at runtime. Each thread in a group of threads executing on the parallel processor may then access the uniform constant.

Binding constants at runtime for improved resource utilization

Growing community of inventors

San Jose, CA, United States of America

Manan Patel