Sambanova Systems, Inc.

Maulik Desai

Raghu Prabhakar

Sitanshu Gupta

Mingran Wang

Nathan Francis Sheeley

Scott Layson Burson

Darshan Dhimantkumar Gandhi

The softmax operation is pipelined to evenly balanced operations with 2N latency per sharded M dimension of a tensor shaped M*N, resulting in ˜1.8× performance gain. As each operation is fragmented, the pipeline does not assume a fixed-cost fill that can performance-wise hurt significantly for small Tensor dimensions. In addition, this innovative design is Place-and-Route (PNR) friendly as well as resource efficient. Moreover, it is easily parallelized without requiring additional support at the subnet level.

High performance softmax for large models

As general matrix multiply (GEMM) bottlenecks are ameliorated by tensor parallelism that is distributed to several processors, layer normalization (LN) surfaces as a latent bottleneck as it is not amenable to distribution. LN performance is linear to embedding size, which is extremely large in some AI models. Moreover, aggressive tiling prevents the use of internal pipelining. The disclosed implementation addresses this issue, composing LN from simpler operations and this composition is amenable to pipelining, facilitating efficient implementation of large AI models (e.g., GPTs). In both forward and backward propagation, the pipeline is stretched longer with improved balance across stages. This strategy improves throughput for larger batch-sizes as the workload benefits from pipelining operations for better performance. Furthermore, avoiding stochastic rounding further improves performance. In addition, LayerNorm checkpoints facilitate efficient computation of gradients during backward propagation.

High performance layer normalization for large models

The technology disclosed relates to matrix multiplication where the multiplier can be a sparse matrix. In particular, a multiplication device includes first circuitry configured to obtain the multiplicand matrix and an index of columns of the multiplier matrix and to generate an intermediate matrix that has one row per entry in the index copied from a respective row of the multiplicand matrix based on a value of a corresponding entry in the index. The device also includes second circuitry configured to receive the intermediate matrix from the first circuitry, obtain non-zero values of the multiplier matrix and a list of a number of non-zero entries per row of the multiplier matrix, and generate a product matrix as a result of multiplies of the non-zero values of the multiplier matrix and the intermediate matrix.

Growing community of inventors

Cedar Park, TX, United States of America

Maulik Desai