Microsoft Technology Licensing, LLC

Eric Sen Chung

Ming Gang Liu

Steven Karl Reinhardt

Daniel Lo

Jeremy Halden Fowers

Kalin Ovtcharov

Taesik Na

Bita Darvish Rouhani

Shlomo Alkalay

Haishan Zhu

Bounding box quantization can reduce the quantity of bits utilized to express numerical values prior to the multiplication of matrices comprised of such numerical values, thereby reducing both memory consumption and processor utilization. Stochastic rounding can provide sufficient precision to enable the storage of weight values in reduced-precision formats without having to separately store weight values in a full-precision format. Alternatively, other rounding mechanisms, such as round to nearest, can be utilized to exchange weight values in reduced-precision formats, while also storing weight values in full-precision formats for subsequent updating. To facilitate conversion, reduced-precision formats such as brain floating-point format can be utilized.

Neural network training with decreased memory consumption and processor utilization

A system for training a neural network receives training data and performing lower precision format training calculations using lower precision format data at one or more training phases. One or more results from the lower precision format training calculations are converted to higher precision format data, and higher precision format training calculations are performed using the higher precision format data at one or more additional training phases. The neural network is modified using the results from the one or more additional training phases. The mixed precision format training calculations train the neural network more efficiently, while maintaining an overall accuracy.

Training neural networks using mixed precision computations

Growing community of inventors

Woodinville, WA, United States of America

Eric Sen Chung