Microsoft Technology Licensing, LLC

Amar Phanishayee

Eric S Chung

Daniel Lo

Yiren Zhao

James Wiliamson Mickens

Ritchie Zhao

Vijaychidambaram Velayudhan Pillai

Ranveer Chandra

Ashish Kapoor

Sudipta Narayan Sinha

Ratul Mahajan

Deepak Vasisht

Nikhil Devanur Rangarajan

Xinxin Jin

Madhusudhan Gumbalapura Sudarshan

Bita Darvish Rouhani

Saurabh Agarwal

Jaeyeon Jung

Jorgen Thelin

Xuan Zou

Monia Ghobadi

Vivek Seshadri

Trinabh Gupta

Danyang Zhuo

Rayman Preet Singh

Animesh Jain

Gennady Pekhimenko

Shivaram Venkataraman

Guanhua Wang

Deepak Narayanan

Aaron Harlap

Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Neural network activation compression with narrow block floating-point

The description relates to deep learning cluster scheduler modular toolkits. One example can include generating a deep learning cluster scheduler modular toolkit that includes multiple DL scheduler abstraction modules and interactions between the multiple DL scheduler abstraction modules and allows user composition of the multiple DL scheduler abstraction modules to realize a deep learning scheduler.

Deep learning scheduler toolkit

Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Neural network activation compression with non-uniform mantissas

Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.

Adjusting activation compression for neural network training

Layers of a deep neural network (DNN) are partitioned into stages using a profile of the DNN. Each of the stages includes one or more of the layers of the DNN. The partitioning of the layers of the DNN into stages is optimized in various ways including optimizing the partitioning to minimize training time, to minimize data communication between worker computing devices used to train the DNN, or to ensure that the worker computing devices perform an approximately equal amount of the processing for training the DNN. The stages are assigned to the worker computing devices. The worker computing devices process batches of training data using a scheduling policy that causes the workers to alternate between forward processing of the batches of the DNN training data and backward processing of the batches of the DNN training data. The stages can be configured for model parallel processing or data parallel processing.

Highly performant pipeline parallel deep neural network training

Apparatus and methods for training a neural network accelerator using quantized precision data formats having outlier values are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. Outlier values, comprising additional bits of mantissa and/or exponent are stored in ancillary storage for subset of the activation values. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.

Neural network activation compression with outlier block floating-point

An interconnect topology for communication between GPUs in a computing system is determined. A quantity of directed spanning trees are generated for transmitting data between the GPUs using the interconnect topology and packed. The directed spanning trees define the connections between GPUs that are to be utilized for the transmission and the amount of data to be transmitted on each connection. Program code is generated for implementing the data transfer defined by the directed spanning trees. When the program code is executed, the directed spanning trees are used to pipeline the transmission of chunks of data, such as model parameters used during data-parallel DNN training, between the GPUs. The program code can also determine an optimal chunk size for data to be transferred between the GPUs.

Mitigating communication bottlenecks during parameter exchange in data-parallel DNN training

Functions are added to a deep neural network ('DNN') computation graph for encoding data structures during a forward training pass of the DNN and decoding previously-encoded data structures during a backward training pass of the DNN. The functions added to the DNN computation graph can be selected based upon on the specific layer pairs specified in the DNN computation graph. Once a modified DNN computation graph has been generated, the DNN can be trained using the modified DNN computation graph. The functions added to the modified DNN computation graph can reduce the utilization of memory during training of the DNN.

Efficient data encoding for deep neural network training

This document relates to data storage techniques. One example can buffer write commands and cause the write commands to be committed to storage in flush epoch order. Another example can maintain a persistent log of write commands that are arranged in the persistent log in flush epoch order. Both examples may provide a prefix consistent state in the event of a crash.

Block storage by decoupling ordering from durability

A gateway that may be implemented in a local network and that communicates with a cloud network to provide efficient services in a weakly connected setting is disclosed. The gateway may be configured to enable services that efficiently utilize resources in both of the gateway and the cloud network, and provide a desired quality of service while operating in a weakly connected setting. The gateway may provide data collection and processing, local network services, and enable cloud services that utilize data collected and processed by the gateway. The local network may include one or more sensors and/or video cameras that provide data to the gateway. In a further implementation, the gateway may determine an allocation of one or more tasks of a service between the gateway and a cloud network by determining the allocation of the one or more service tasks based on desired service latency.

IoT gateway for weakly connected settings

Techniques and architectures may be used to generate data center network topologies that use less reliable and less expensive links mixed with links of higher reliability. Such topologies may be categorized into reliability classes, where each class corresponds to a bound(s) on reliability of paths that include the links. A topology class may be selected for use by an application based, at least in part, on the degree of reliability demanded by the application.

Growing community of inventors

Seattle, WA, United States of America

Amar Phanishayee