Nvidia Corporation

Ronny Krashinsky

Jack H Choquette

Shirish C Gadre

Olivier Giroux

Manan Patel

John H Edmondson

Xiaogang Qiu

Apoorv Parle

Steve Heinrich

Jerome Francis Duluk, Jr

Lacky V Shah

Steven James Heinrich

Alexander Lev Minkin

Omkar Paranjape

Gentaro Hirota

John Tran

Poornachandra Rao

Ramesh Jandhyala

Mark Alan Gebhart

Ravi Kiran Manyam

Ze Long

Jeff Tuckey

Alan Kaatz

Prakash Bangalore Prabhakar

Wish Gandhi

Greg Palmer

Aditya Avinash Atluri

Michael Siu

Vinod Grover

Ming Yiu Siu

Brucek Kurdo Khailany

Ganesh Venkatesh

Luke Durant

Brian S Pharris

Jorge Albericio Latorre

Rishkul Kulkarni

Harold Carter Edwards

Rajballav Dash

James David Balfour

Konstantinos Kyriakopoulos

Jeff Schottmiller

Gokul Ramaswamy Hirisave Chandra Shekhara

Sandeep Navada

Eric Werness

Naman Govil

Shayani Deb

Sean Jeffrey Treichler

Wishwesh Anil Gandhi

Michael A Parker

Philip Browning Johnson

Michael Alan Fetterman

Gregory Diamos

Peter Michael Nelson

Andrew Robert Kerr

Ajay Sudarshan Tirumala

Jeffrey Michael Pool

David Fontaine

Rui Guo

Praveen Joginipally

Xiangyun Kong

Richard Craig Johnson

Feng Xie

Jeff Pool

Gautam Chakrabarti

Girish Bharambe

Kyrylo Perelygin

Hemayet Hossain

Maciej Piotr Tyrlik

Praveen Kaushik

Debajit Bhattacharya

Edward H Gornish

Oliver Giroux

Chao Li

Raymond Hoi Man Wong

Balaji Krishna Yugandhar Atukuri

Carter Edwards

Vishalkumar Ketankumar Mehta

Vikram Dhar

Subhasmita Chakraborty

David Dastous St Hilaire

Bageshri Sathe

David Allan Berson

Haicheng Wu

Vijay Thakkar

Cody Addison

Aman Arora

James Player

Cris Cecka

Yunsup Lee

Kaushik Nadadhur

A parallel processing unit comprises a plurality of processors each being coupled to a memory access hardware circuitry. Each memory access hardware circuitry is configured to receive, from the coupled processor, a memory access request specifying a coordinate of a multidimensional data structure, wherein the memory access hardware circuit is one of a plurality of memory access circuitry each coupled to a respective one of the processors; and, in response to the memory access request, translate the coordinate of the multidimensional data structure into plural memory addresses for the multidimensional data structure and using the plural memory addresses, asynchronously transfer at least a portion of the multidimensional data structure for processing by at least the coupled processor. The memory locations may be in the shared memory of the coupled processor and/or an external memory.

Method and apparatus for efficient access to multidimensional data structures and/or other large data blocks

Apparatuses, systems, and techniques to cause information to be provided. In at least one embodiment, one or more circuits are to perform an application programming interface (API) to cause an amount of information to be accessed as a result of one or more memory transactions to be provided to one or more users.

Application programming interface to provide information

Processing hardware of a processor is virtualized to provide a façade between a consistent programming interface and specific hardware instances. Hardware processor components can be permanently or temporarily disabled when not needed to support the consistent programming interface and/or to balance hardware processing across a hardware arrangement such as an integrated circuit. Executing software can be migrated from one hardware arrangement to another without need to reset the hardware.

Virtualizing hardware processing resources in a processor

Apparatuses, systems, and techniques to adapt instructions in a SIMT architecture for execution on serial execution units. In at least one embodiment, a set of one or more threads is selected from a group of active threads associated with an instruction and the instruction is executed for the set of one or more threads on a serial execution unit.

Scalarization of instructions for SIMT architectures

Various embodiments include a parallel processing computer system that provides multiple memory synchronization domains in a single parallel processor to reduce unneeded synchronization operations. During execution, one execution kernel may synchronize with one or more other execution kernels by processing outstanding memory references. The parallel processor tracks memory references for each domain to each portion of local and remote memory. During synchronization, the processor synchronizes the memory references for a specific domain while refraining from synchronizing memory references for other domains. As a result, synchronization operations between kernels complete in a reduced amount of time relative to prior approaches.

Thread synchronization across memory synchronization domains

A new level(s) of hierarchy—Cooperate Group Arrays (CGAs)—and an associated new hardware-based work distribution/execution model is described. A CGA is a grid of thread blocks (also referred to as cooperative thread arrays (CTAs)). CGAs provide co-scheduling, e.g., control over where CTAs are placed/executed in a processor (such as a GPU), relative to the memory required by an application and relative to each other. Hardware support for such CGAs guarantees concurrency and enables applications to see more data locality, reduced latency, and better synchronization between all the threads in tightly cooperating collections of CTAs programmably distributed across different (e.g., hierarchical) hardware domains or partitions.

Cooperative group arrays

Distributed shared memory (DSMEM) comprises blocks of memory that are distributed or scattered across a processor (such as a GPU). Threads executing on a processing core local to one memory block are able to access a memory block local to a different processing core. In one embodiment, shared access to these DSMEM allocations distributed across a collection of processing cores is implemented by communications between the processing cores. Such distributed shared memory provides very low latency memory access for processing cores located in proximity to the memory blocks, and also provides a way for more distant processing cores to also access the memory blocks in a manner and using interconnects that do not interfere with the processing cores' access to main or global memory such as hacked by an L2 cache. Such distributed shared memory supports cooperative parallelism and strong scaling across multiple processing cores by permitting data sharing and communications previously possible only within the same processing core.

Distributed shared memory

Apparatuses, systems, and techniques to perform computational operations in response to one or more compute uniform device architecture (CUDA) programs. In at least one embodiment, one or more computational operations are to cause one or more other computational operations to wait until a portion of matrix multiply-accumulate (MMA) operations have been performed.

Application programming interface to wait on matrix multiply-accumulate

This specification describes a programmatic multicast technique enabling one thread (for example, in a cooperative group array (CGA) on a GPU) to request data on behalf of one or more other threads (for example, executing on respective processor cores of the GPU). The multicast is supported by tracking circuitry that interfaces between multicast requests received from processor cores and the available memory. The multicast is designed to reduce cache (for example, layer 2 cache) bandwidth utilization enabling strong scaling and smaller tile sizes.

Programmatically controlled data multicasting across multiple compute engines

To synchronize operations of a computing system, a new type of synchronization barrier is disclosed. In one embodiment, the disclosed synchronization barrier provides for certain synchronization mechanisms such as, for example, 'Arrive' and 'Wait' to be split to allow for greater flexibility and efficiency in coordinating synchronization. In another embodiment, the disclosed synchronization barrier allows for hardware components such as, for example, dedicated copy or direct-memory-access (DMA) engines to be synchronized with software-based threads.

High performance synchronization mechanisms for coordinating operations on a computer system

Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices.

Managing data sparsity for neural networks

Compressed data is oftentimes beneficial for reducing the computing resources required, for example, to transmit and store data. The compression of data is particularly useful when dealing with sparse data (data that includes numerous zeros or near-zero values) and only non-zero values above a certain threshold have significance. When dealing with compressed data, oftentimes the data needs to be decompressed for processing (e.g., by deep learning networks or other applications configured to operate on sparse, or other uncompressed data). Instructions are disclosed for supporting the decompression of compressed data by a processing unit such as a CPU and GPU.

Decompression techniques for processing compressed data suitable for artificial neural networks

A unified cache subsystem includes a data memory configured as both a shared memory and a local cache memory. The unified cache subsystem processes different types of memory transactions using different data pathways. To process memory transactions that target shared memory, the unified cache subsystem includes a direct pathway to the data memory. To process memory transactions that do not target shared memory, the unified cache subsystem includes a tag processing pipeline configured to identify cache hits and cache misses. When the tag processing pipeline identifies a cache hit for a given memory transaction, the transaction is rerouted to the direct pathway to data memory. When the tag processing pipeline identifies a cache miss for a given memory transaction, the transaction is pushed into a first-in first-out (FIFO) until miss data is returned from external memory. The tag processing pipeline is also configured to process texture-oriented memory transactions.

Unified cache for diverse memory traffic

Growing community of inventors

San Francisco, CA, United States of America

Ronny Krashinsky