Nvidia Corporation

Ronny Krashinsky

Jack H Choquette

Shirish C Gadre

Olivier Giroux

Manan Patel

John H Edmondson

Xiaogang Qiu

Apoorv Parle

Steve Heinrich

Jerome Francis Duluk, Jr

Lacky V Shah

Steven James Heinrich

Alexander Lev Minkin

Omkar Paranjape

Gentaro Hirota

John Tran

Poornachandra Rao

Ze Long

Ravi Kiran Manyam

Mark Alan Gebhart

Ramesh Jandhyala

Jeff Tuckey

Alan Kaatz

Wish Gandhi

Prakash Bangalore Prabhakar

Greg Palmer

Aditya Avinash Atluri

Michael Siu

Vinod Grover

Ming Yiu Siu

Brucek Kurdo Khailany

Ganesh Venkatesh

Luke Durant

Brian S Pharris

Harold Carter Edwards

Jorge Albericio Latorre

Rishkul Kulkarni

Rajballav Dash

Konstantinos Kyriakopoulos

James David Balfour

Sandeep Navada

Gokul Ramaswamy Hirisave Chandra Shekhara

Jeff Schottmiller

Shayani Deb

Naman Govil

Eric Werness

Sean Jeffrey Treichler

Wishwesh Anil Gandhi

Michael A Parker

Philip Browning Johnson

Michael Alan Fetterman

Gregory Diamos

Peter Michael Nelson

Andrew Robert Kerr

Ajay Sudarshan Tirumala

David Fontaine

Jeffrey Michael Pool

Rui Guo

Richard Craig Johnson

Praveen Joginipally

Xiangyun Kong

Feng Xie

Hemayet Hossain

Girish Bharambe

Gautam Chakrabarti

Jeff Pool

Kyrylo Perelygin

Debajit Bhattacharya

Praveen Kaushik

Maciej Piotr Tyrlik

Edward H Gornish

Oliver Giroux

Cris Cecka

James Player

Aman Arora

Cody Addison

Kaushik Nadadhur

Vijay Thakkar

Haicheng Wu

David Allan Berson

Bageshri Sathe

Subhasmita Chakraborty

Vikram Dhar

Carter Edwards

Balaji Krishna Yugandhar Atukuri

Raymond Hoi Man Wong

Chao Li

Yunsup Lee

A parallel processing unit comprises a plurality of processors each being coupled to a memory access hardware circuitry. Each memory access hardware circuitry is configured to receive, from the coupled processor, a memory access request specifying a coordinate of a multidimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuitry each coupled to a respective one of the processors; and, in response to the memory access request, translate the coordinate of the multidimensional data structure into plural memory addresses for the multidimensional data structure and using the plural memory addresses, asynchronously transfer at least a portion of the multidimensional data structure for processing by at least the coupled processor. The memory locations may be in the shared memory of the coupled processor and/or an external memory.

Method and apparatus for efficient access to multidimensional data structures and/or other large data blocks

Apparatuses, systems, and techniques to cause information to be provided. In at least one embodiment, one or more circuits are to perform an application programming interface (API) to cause an amount of information to be accessed as a result of one or more memory transactions to be provided to one or more users.

Application programming interface to provide information

Processing hardware of a processor is virtualized to provide a façade between a consistent programming interface and specific hardware instances. Hardware processor components can be permanently or temporarily disabled when not needed to support the consistent programming interface and/or to balance hardware processing across a hardware arrangement such as an integrated circuit. Executing software can be migrated from one hardware arrangement to another without need to reset the hardware.

Virtualizing hardware processing resources in a processor

Apparatuses, systems, and techniques to adapt instructions in a SIMT architecture for execution on serial execution units. In at least one embodiment, a set of one or more threads is selected from a group of active threads associated with an instruction and the instruction is executed for the set of one or more threads on a serial execution unit.

Scalarization of instructions for SIMT architectures

Various embodiments include a parallel processing computer system that provides multiple memory synchronization domains in a single parallel processor to reduce unneeded synchronization operations. During execution, one execution kernel may synchronize with one or more other execution kernels by processing outstanding memory references. The parallel processor tracks memory references for each domain to each portion of local and remote memory. During synchronization, the processor synchronizes the memory references for a specific domain while refraining from synchronizing memory references for other domains. As a result, synchronization operations between kernels complete in a reduced amount of time relative to prior approaches.

Thread synchronization across memory synchronization domains

A new level(s) of hierarchy—Cooperate Group Arrays (CGAs)—and an associated new hardware-based work distribution/execution model is described. A CGA is a grid of thread blocks (also referred to as cooperative thread arrays (CTAs)). CGAs provide co-scheduling, e.g., control over where CTAs are placed/executed in a processor (such as a GPU), relative to the memory required by an application and relative to each other. Hardware support for such CGAs guarantees concurrency and enables applications to see more data locality, reduced latency, and better synchronization between all the threads in tightly cooperating collections of CTAs programmably distributed across different (e.g., hierarchical) hardware domains or partitions.

Cooperative group arrays

Distributed shared memory (DSMEM) comprises blocks of memory that are distributed or scattered across a processor (such as a GPU). Threads executing on a processing core local to one memory block are able to access a memory block local to a different processing core. In one embodiment, shared access to these DSMEM allocations distributed across a collection of processing cores is implemented by communications between the processing cores. Such distributed shared memory provides very low latency memory access for processing cores located in proximity to the memory blocks, and also provides a way for more distant processing cores to also access the memory blocks in a manner and using interconnects that do not interfere with the processing cores' access to main or global memory such as hacked by an L2 cache. Such distributed shared memory supports cooperative parallelism and strong scaling across multiple processing cores by permitting data sharing and communications previously possible only within the same processing core.

Distributed shared memory

Apparatuses, systems, and techniques to perform computational operations in response to one or more compute uniform device architecture (CUDA) programs. In at least one embodiment, one or more computational operations are to cause one or more other computational operations to wait until a portion of matrix multiply-accumulate (MMA) operations have been performed.

Application programming interface to wait on matrix multiply-accumulate

This specification describes a programmatic multicast technique enabling one thread (for example, in a cooperative group array (CGA) on a GPU) to request data on behalf of one or more other threads (for example, executing on respective processor cores of the GPU). The multicast is supported by tracking circuitry that interfaces between multicast requests received from processor cores and the available memory. The multicast is designed to reduce cache (for example, layer 2 cache) bandwidth utilization enabling strong scaling and smaller tile sizes.

Programmatically controlled data multicasting across multiple compute engines

To synchronize operations of a computing system, a new type of synchronization barrier is disclosed. In one embodiment, the disclosed synchronization barrier provides for certain synchronization mechanisms such as, for example, 'Arrive' and 'Wait' to be split to allow for greater flexibility and efficiency in coordinating synchronization. In another embodiment, the disclosed synchronization barrier allows for hardware components such as, for example, dedicated copy or direct-memory-access (DMA) engines to be synchronized with software-based threads.

High performance synchronization mechanisms for coordinating operations on a computer system

Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices.

Managing data sparsity for neural networks

Compressed data is oftentimes beneficial for reducing the computing resources required, for example, to transmit and store data. The compression of data is particularly useful when dealing with sparse data (data that includes numerous zeros or near-zero values) and only non-zero values above a certain threshold have significance. When dealing with compressed data, oftentimes the data needs to be decompressed for processing (e.g., by deep learning networks or other applications configured to operate on sparse, or other uncompressed data). Instructions are disclosed for supporting the decompression of compressed data by a processing unit such as a CPU and GPU.

Decompression techniques for processing compressed data suitable for artificial neural networks

A unified cache subsystem includes a data memory configured as both a shared memory and a local cache memory. The unified cache subsystem processes different types of memory transactions using different data pathways. To process memory transactions that target shared memory, the unified cache subsystem includes a direct pathway to the data memory. To process memory transactions that do not target shared memory, the unified cache subsystem includes a tag processing pipeline configured to identify cache hits and cache misses. When the tag processing pipeline identifies a cache hit for a given memory transaction, the transaction is rerouted to the direct pathway to data memory. When the tag processing pipeline identifies a cache miss for a given memory transaction, the transaction is pushed into a first-in first-out (FIFO) until miss data is returned from external memory. The tag processing pipeline is also configured to process texture-oriented memory transactions.

Unified cache for diverse memory traffic

Growing community of inventors

San Francisco, CA, United States of America

Ronny Krashinsky