Nvidia Corporation

Jorge Albericio Latorre

Ming Yiu Siu

Jeff Pool

Jack H Choquette

Ganesh Venkatesh

Ronny Krashinsky

Manan Patel

Jeffrey Michael Pool

David Garcia

Michael Alex Frumkin

John Tran

Feng Xie

In artificial neural networks, and other similar applications, there is typically a large amount of data involved that is considered sparse data. Due to the large size of the data involved in such applications, it is helpful to compress the data to save bandwidth resources when transmitting the data and save memory resources when storing the data. Introduced herein is a compression technique that selects elements with significant values from data and restructures them into a structured sparse format. By generating metadata that enforces the structured sparse format and organizing the data according to the metadata, the introduced technique not only reduces the size of the data but also consistently places the data in a particular format. As such, hardware can be simplified and optimized to process the data much faster and much more efficiently than the conventional compression techniques that rely on a non-structured sparsity format.

Compression techniques for data structures suitable for artificial neural networks

Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices.

Managing data sparsity for neural networks

Compressed data is oftentimes beneficial for reducing the computing resources required, for example, to transmit and store data. The compression of data is particularly useful when dealing with sparse data (data that includes numerous zeros or near-zero values) and only non-zero values above a certain threshold have significance. When dealing with compressed data, oftentimes the data needs to be decompressed for processing (e.g., by deep learning networks or other applications configured to operate on sparse, or other uncompressed data). Instructions are disclosed for supporting the decompression of compressed data by a processing unit such as a CPU and GPU.

Decompression techniques for processing compressed data suitable for artificial neural networks

Many computing systems process data organized in a matrix format. For example, artificial neural networks (ANNs) perform numerous computations on data organized into matrices using conventional matrix arithmetic operations. One such operation, which is commonly performed, is the transpose operation. Additionally, many such systems need to process many matrices and/or matrices that are large in size. For sparse matrices that hold few significant values and many values that can be ignored, transmitting and processing all the values in such matrices is wasteful. Thus, techniques are introduced for storing a sparse matrix in a compressed format that allows for a matrix transpose operation to be performed on the compressed matrix without having to first decompress the compressed matrix. By utilizing the introduced techniques, more matrix operations can be performed than conventional systems.

Efficient matrix data format applicable for artificial neural network

Many computing systems process data organized in a matrix format. For example, artificial neural networks perform numerous computations on data organized into matrices using conventional matrix arithmetic operations. One such operation is the transpose operation. Techniques are introduced for storing a matrix in a compressed format that allows, for example, a transpose operation to be performed during decompression. Thus, by utilizing the introduced techniques, transformations of compressed matrices such transposition can be achieved in a more effective way. Parallel processing may also be used to more efficiently compress and/or decompress.

Growing community of inventors

Santa Clara, CA, United States of America

Jorge Albericio Latorre