Microsoft Technology Licensing, LLC

Elwha LLC

Marc Tremblay

Brian Lee

Tiyasa Mitra

Lowell L Wood, Jr

Roderick A Hyde

Clarence T Tegreene

Jordin T Kare

Nathan P Myhrvold

Robert W Lord

Muriel Y Ishikawa

Charles Whitmer

Richard T Lord

David Bazeley Tuckerman

Craig J Mundie

Hon Wah Chin

Howard Lee Davidson

Eric D Rudder

Nicholas F Pasch

Philip Lionel Barnes

Kimberly D A Hallman

Bharadwaj Pudipeddi

Andy Wagner

Jinwen Xi

Saurabh Mohan Kulkarni

Sujeeth Subramanya Bharadwaj

Maral Mesmakhosroshahi

Gerald Reuben DeJean, Ii

Mats Eric Myrberg

Lance Eric Greenwade

Andrew Wagner

Nuno Claudino Pereira Lopes

Matthias Baenninger

Layali Rashid

John Grant Bennett

Barry Clayton Bond

Mark John Ramberg

Vlad Sadovsky

Mohit Mittal

Kimberly Da Hallman

Devangkumar Patel

Gautham Popuri

Gerald Dejean

Embodiments of the present disclosure include systems and methods for packing tokens to train sequence models. In some embodiments, a plurality of datasets for training a sequence model is received. Each dataset in the plurality of datasets includes a sequence of correlated tokens. A set of training data is generated that includes a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets. The sequence model is trained using the set of training data.

Token packing for sequence models

Embodiments of the present disclosure include systems and methods for forcing weights of transformer model layers when training a transformer model. In some embodiments, input data is received at a first layer included in a transformer model. The input data is processed through the first layer of the transformer model to produce a first output data. The first output data is processed through the first layer of the transformer model to produce a second output data. The first output data is processed through a second layer included in the transformer model to produce a third output data. A difference is calculated between the second output data and the third output data. Weights included in the first layer of the transformer model are adjusted based on the calculated difference.

Forcing weights of transformer model layers

Systems and methods related to hardware-assisted gradient optimization using streamed gradients are described. An example method in a system comprising a memory configured to store weights associated with a neural network model comprising L layers, where L is an integer greater than one, a gradient optimizer, and a plurality of workers is described. The method includes during a single burst cycle moving a first set of gradients, received from each of the plurality of workers, from at least one gradient buffer to the gradient optimizer and moving weights from at least one buffer, coupled to the memory, to the gradient optimizer. The method further includes during the single burst cycle writing back the new weights, calculated by the gradient optimizer, to the memory. The method further includes during the single burst cycle transmitting the new weights, from the gradient optimizer, to each of the plurality of workers.

Hardware-assisted gradient optimization using streamed gradients

Embodiments of the present disclosure include systems and methods for compressing and decompressing data generated by sub-blocks in a neural network. In some embodiment, an input matrix is received at a compression block in the neural network. The compression block compresses the input matrix into a compressed matrix and outputs the compressed matrix. The compressed matrix has a reduced dimensionality relative to a dimensionality of the input matrix. A decompression block retrieves the compressed matrix. The decompression block decompresses compressed matrix into a decompressed matrix and outputs the decompressed matrix. The decompressed matrix has a same dimensionality as the dimensionality of the input matrix. The compression and decompression blocks are optimized based on feedback received from the neural network.

Compressing and decompressing data for language models

Embodiments of the present disclosure include systems and methods for determining position values for training data that is used to train transformer models. In some embodiments, a set of input data for training a transformer model is received. The set of input data comprises a set of tokens. Based on an offset value, a set of successive position values for the set of tokens is determined. Each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens. A set of training data is generated to comprise the set of tokens and the set of successive position values. The transformer model is trained using the set of training data.

Determining position values for transformer models

Embodiments of the present disclosure include systems and methods for training transformer models using position masking. In some embodiments, a set of data for training a transformer model is received. The set of data includes a sequence of tokens and a set of position values. Each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens. A subset of the set of position values in the set of data is selected. Each position value in the subset of the set of position values is replaced with a second defined value to form a second set of defined values. The transformer model is trained using the set of data.

Position masking for transformer models

Embodiments of the present disclosure include systems and methods for reducing hardware resource utilization by residual neural networks. In some embodiments, a first matrix is received at a layer included in a neural network. The first matrix is compressed to produce a second matrix. The second matrix has a reduced dimensionality relative to a dimensionality of the first matrix. The second matrix is processed through a network block in the layer included in the neural network. The processed second matrix is expanded to produce a third matrix. The third matrix has a dimensionality that is equal to a dimensionality of the first matrix. The third matrix is added to the first matrix to produce a fourth matrix.

Reducing hardware resource utilization for residual neural networks

Embodiments of the present disclosure include an error recovery method comprising detecting a computing error, restarting a first artificial intelligence processor of a plurality of artificial intelligence processors processing a data set, and loading a model in the artificial intelligence processor, wherein the model corresponds to a same model processed by the plurality of artificial intelligence processors during a previous processing iteration by the plurality of artificial intelligence processors on data from the data set.

Systems and methods for error recovery

Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. A method for processing a neural network comprising L layers, where L is an integer greater than two, includes partitioning the L layers among a set of computing resources configured to process forward passes and backward passes associated with each of the L layers. The method further includes initiating processing of the forward passes and the backward passes using the set of computing resources. The method further includes upon completion of a first set of forward passes and a first set of backward passes associated with a first layer of the L layers, initiating update of parameters associated with the first layer when gradients are available for updating the parameters associated with the first layer without waiting to calculate gradients associated with any of remaining L layers.

Pipelined neural network processing with continuous and asynchronous updates

Systems and methods related to dual-momentum gradient optimization with reduced memory requirements are described. An example method in a system comprising a gradient optimizer and a memory configured to store momentum values associated with a neural network model comprising L layers is described. The method includes retrieving from the memory a first set of momentum values and a second set of momentum values, corresponding to a layer of the neural network model, having a selected storage format. The method further includes converting the first set of momentum values to a third set of momentum values having a training format associated with the gradient optimizer and converting the second set of momentum values to a fourth set of momentum values having a training format associated with the gradient optimizer. The method further includes performing gradient optimization using the third set of momentum values and the fourth set of momentum values.

Dual-momentum gradient optimization with reduced memory requirements

Systems, methods, and apparatuses are provided for compressing values. A plurality of parameters may be obtained from a memory, each parameter comprising a floating-point number that is used in a relationship between artificial neurons or nodes in a model. A mantissa value and an exponent value may be extracted from each floating-point number to generate a set of mantissa values and a set of exponent values. The set of mantissa values may be compressed to generate a mantissa lookup table (LUT) and a plurality of mantissa LUT index values. The set of exponent values may be encoded to generate an exponent LUT and a plurality of exponent LUT index values. The mantissa LUT, mantissa LUT index values, exponent LUT, and exponent LUT index values may be provided to one or more processing entities to train the model.

Lossless exponent and lossy mantissa weight compression for training deep neural networks

Embodiments of the present disclosure include systems and methods for training neural networks. In one embodiment, neural network may receive input data and produce output results in response to the input data and weights of the neural network. An error is determined at an output of the neural network based on the output results. The error is propagated in a reverse direction through the neural network from the output and one or more intermediate outputs to adjust the weights.

Systems and methods for training a neural network

A method of increasing computer hardware efficiency of a matrix computation. The method comprises receiving at a computer processing device, digital signals encoding one or more operations of the matrix computation, each operation including one or more operands. The method further comprises, responsive to determining, by a sparse data check device of the computer processing machine, that an operation of the matrix computation includes all dense operands, forwarding the operation to a dense computation device of the computer processing machine configured to perform the operation of the matrix computation based on the dense operands. The method further comprises, responsive to determining, by the sparse data check device, that an operation of the matrix computation includes one or more sparse operands, forwarding the operation to a sparse computation device configured to perform the operation of the matrix computation.

Accelerator for dense and sparse matrix computations

Growing community of inventors

Clyde Hill, WA, United States of America

Marc Tremblay