Nvidia Corporation

Ming Yiu Siu

Stuart F Oberman

Brett Coon

Jack H Choquette

John Erik Lindholm

John R Nickolls

Jonah Matthew Alben

Brent Ralph Boswell

Peter C Mills

John Tran

Jeffrey Michael Pool

Matthew P Gerlach

David Conrad Tannenbaum

Brian Matthew Fahs

Andrew Robert Kerr

Samuel Kan Liu

Manan Patel

Lars Nyland

Jorge Albericio Latorre

Weizhong Xu

Steven James Heinrich

Michael C Shebanow

Ganesh Venkatesh

Michael Alan Fetterman

Robert Steven Glanville

Ronny Krashinsky

Xiaogang Qiu

Rajeshwaran Selvanesan

Srinivasan Iyer

Yan Yan Tang

Ian Chi Yan Kwong

Colin Sprinkle

Henry Packard Moreton

Shirish C Gadre

John Matthew Burgess

Simon S Moy

Sachin Satish Idgunji

Alexander Lev Minkin

Ilyas Elkin

Gary M Tarolli

Bryan Catanzaro

Robert J Stoll

Hao Fang Wu

Nicholas Wang

Stewart Glenn Carlton

Charles McCarver

Sameer Halepete

Svetoslav D Tzvetkov

Feng Xie

Jeff Pool

Paulius Micikevicius

James P Reilley

Ewa Kubalska

Srinivasan (Vasu) Iyer

Alex Gu

Robert Kirby

Ian-Chi Yan Kwong

Michael Fiyak

Aravind Kalaiah

Daniel Salinas

Tommy Thorn

Rajarshi Roy

Jonathan Raiman

Neel Kant

Saad Godil

Brett W Coon

A method, computer readable medium, and processor are described herein for inline data inspection by using a decoder to decode a load instruction, including a signal to cause a circuit in a processor to indicate whether data loaded by a load instruction exceeds a threshold value. Moreover, an indication of whether data loaded by a load instruction exceeds a threshold value may be stored.

Inline data inspection for workload simplification

Apparatuses, systems, and techniques for designing a data path circuit such as a parallel prefix circuit with reinforcement learning are described. A method can include receiving a first design state of a data path circuit, inputting the first design state of the data path circuit into a machine learning model, and performing reinforcement learning using the machine learning model to output a final design state of the data path circuit, wherein the final design state of the data path circuit has decreased area, power consumption and/or delay as compared to conventionally designed data path circuits.

Data path circuit design using reinforcement learning

A method, computer readable medium, and processor are disclosed for performing matrix multiply and accumulate (MMA) operations. The processor includes a datapath configured to execute the MMA operation to generate a plurality of elements of a result matrix at an output of the datapath. Each element of the result matrix is generated by calculating at least one dot product of corresponding pairs of vectors associated with matrix operands specified in an instruction for the MMA operation. A dot product operation includes the steps of: generating a plurality of partial products by multiplying each element of a first vector with a corresponding element of a second vector; aligning the plurality of partial products based on the exponents associated with each element of the first vector and each element of the second vector; and accumulating the plurality of aligned partial products into a result queue utilizing at least one adder.

Generalized acceleration of matrix multiply accumulate operations

In artificial neural networks, and other similar applications, there is typically a large amount of data involved that is considered sparse data. Due to the large size of the data involved in such applications, it is helpful to compress the data to save bandwidth resources when transmitting the data and save memory resources when storing the data. Introduced herein is a compression technique that selects elements with significant values from data and restructures them into a structured sparse format. By generating metadata that enforces the structured sparse format and organizing the data according to the metadata, the introduced technique not only reduces the size of the data but also consistently places the data in a particular format. As such, hardware can be simplified and optimized to process the data much faster and much more efficiently than the conventional compression techniques that rely on a non-structured sparsity format.

Compression techniques for data structures suitable for artificial neural networks

Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices.

Managing data sparsity for neural networks

Compressed data is oftentimes beneficial for reducing the computing resources required, for example, to transmit and store data. The compression of data is particularly useful when dealing with sparse data (data that includes numerous zeros or near-zero values) and only non-zero values above a certain threshold have significance. When dealing with compressed data, oftentimes the data needs to be decompressed for processing (e.g., by deep learning networks or other applications configured to operate on sparse, or other uncompressed data). Instructions are disclosed for supporting the decompression of compressed data by a processing unit such as a CPU and GPU.

Decompression techniques for processing compressed data suitable for artificial neural networks

An integrated circuit such as, for example a graphics processing unit (GPU), includes a dynamic power controller for adjusting operating voltage and/or frequency. The controller may receive current power used by the integrated circuit and a predicted power determined based on instructions pending in a plurality of processors. The controller determines adjustments that need to be made to the operating voltage and/or frequency to minimize the difference between the current power and the predicted power. An in-system reinforced learning mechanism is included to self-tune parameters of the controller.

Techniques for efficiently operating a processing system based on energy characteristics of instructions and machine learning

A system and method are described for providing hints to a processing unit that subsequent operations are likely. Responsively, the processing unit takes steps to prepare for the likely subsequent operations. Where the hints are more likely than not to be correct, the processing unit operates more efficiently. For example, in an embodiment, the processing unit consumes less power. In another embodiment, subsequent operations are performed more quickly because the processing unit is prepared to efficiently handle the subsequent operations.

Providing hints to an execution unit to prepare for predicted subsequent arithmetic operations

Growing community of inventors

Santa Clara, CA, United States of America

Ming Yiu Siu