Microsoft Technology Licensing, LLC

Sujeeth Subramanya Bharadwaj

Marc Tremblay

Tiyasa Mitra

Saurabh Mohan Kulkarni

Bharadwaj Pudipeddi

Andrew Wagner

Jinwen Xi

Maral Mesmakhosroshahi

Andy Wagner

Devangkumar Patel

Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. A method for processing a neural network comprising L layers, where L is an integer greater than two, includes partitioning the L layers among a set of computing resources configured to process forward passes and backward passes associated with each of the L layers. The method further includes initiating processing of the forward passes and the backward passes using the set of computing resources. The method further includes upon completion of a first set of forward passes and a first set of backward passes associated with a first layer of the L layers, initiating update of parameters associated with the first layer when gradients are available for updating the parameters associated with the first layer without waiting to calculate gradients associated with any of remaining L layers.

Pipelined neural network processing with continuous and asynchronous updates

Embodiments of the present disclosure include a method for token-position handling comprising: processing a first sequence of tokens to produce a second sequence of tokens, wherein the second sequence of tokens has a smaller number of tokens than the first sequence of tokens; masking at least some tokens in the second sequence to produce masked tokens; moving the masked tokens to the beginning of the second sequence to produce a third sequence; encoding tokens in the third sequence into a set of numeric vectors in a first array; and processing the first array in a transformer neural network to determine correlations among the third sequence, the processing the first array producing a second array.

Token-position handling for sequence based neural networks

Techniques for training neural networks are provided. According to one set of embodiments, a first array is processed in a spreading component to produce a second array, where a first dimension of the first array corresponds to at least one sequence of approximately orthogonal numeric vectors representing tokens, and where the spreading component combines values along the first dimension. The second array is processed in a transformer neural network to determine correlations between the sequence, which produces a third array. One or more batches of the third array are processed in a de-spreading component to produce a fourth array.

Spread neural networks

Methods for gradient accumulation with free momentum are performed by systems and devices during neural network model training. An accumulator that includes a processor circuit and a memory element generates free momentum between passes of a neural network model training process. The processor circuit receives a difference weight (gradient) and generates a first input by applying a weighting parameter thereto. The processor circuit obtains a prior weight from the memory element and generates a second input by applying another weighting parameter thereto. The processor circuit generates a filtered input with momentum by filtering the first and second input. The memory element generates a stored next pass weight by accumulating the filtered input with the prior weight. A computing resource then processes the next pass of the neural network model training using the stored next pass weight. The methods, systems, and devices are applicable to pipelined model parallelism training processes.

System and method for gradient accumulation with free momentum

Methods, systems, apparatuses, and computer program products are described herein that enable execution of a large AI model on a memory-constrained target device that is communicatively connected to a parameter server, which stores a master copy of the AI model. The AI model may be dissected into smaller portions (e.g., layers or sub-layers), and each portion may be executed as efficiently as possible on the target device. After execution of one portion of the AI model is finished, another portion of the AI model may be downloaded and executed at the target device. To improve efficiency, the input samples may be divided into microbatches, and a plurality of microbatches executing in sequential order may form a minibatch. The size of the group of microbatches or minibatch can be adjusted to reduce the communication overhead. Multi-level parallel parameters reduction may be performed at the parameter server and the target device.

Data parallelism in distributed training of artificial intelligence models

Improved convolutional neural network-based machine learning models are disclosed herein. A convolutional neural network is configured to decompose feature maps generated based on a data item to be classified. The feature maps are decomposed into a first and second subsets. The first subset is representative of high frequency components of the data item, and the second subset is representative of low frequency components of the data item. The second subset is upsampled and is combined with the first subset. The combined feature maps are convolved with a filter to extract a set of features associated with the data item. The first subset is also downsampled and combined with the second subset. The combined feature maps are convolved with a filter to extract another set of features. The data item is classified based on the sets of features extracted based on the convolution operations.

System and method for improving convolutional neural network-based machine learning models

Methods, systems, apparatuses, and computer program products are described herein that enable execution of a large AI model on a memory-constrained target device that is communicatively connected to a parameter server, which stores a master copy of the AI model. The AI model may be dissected into smaller portions (e.g., layers or sub-layers), and each portion may be executed as efficiently as possible on the target device. After execution of one portion of the AI model is finished, another portion of the AI model may be downloaded and executed at the target device. This paradigm of executing one portion of the AI model at a time allows for dynamic execution of the large AI model.

Growing community of inventors

Milpitas, CA, United States of America

Sujeeth Subramanya Bharadwaj