Microsoft Technology Licensing, LLC

Elwha LLC

University of Texas System

Elwah LLC

Douglas Christopher Burger

Lowell L Wood, Jr

Roderick A Hyde

Clarence T Tegreene

Nathan P Myhrvold

Craig J Mundie

Jordin T Kare

Charles Whitmer

Aaron L Smith

Eric S Chung

Muriel Y Ishikawa

Victoria Y H Wood

William Henry Gates, Iii

Robert W Lord

Richard T Lord

Paramvir Bahl

Ranveer Chandra

Desney S Tan

Matthew Graham Dyor

Lin Zhong

Tim Paek

Andrew Richard Putnam

Oriana Riva

Pablos Holman

Suman Kumar Nath

Adrian Michael Caulfield

Sitaram V Lanka

Stephen Francis Heil

Mark A Malamud

Royce A Levien

John D Rinaldo, Jr

John L Manferdelli

Edward K Y Jung

Eric J Horvitz

Chris Demetrios Karkanias

Paul D Holman

Allen L Brown, Jr

Barney Pell

Eric Leuthardt

Richard Neil Zare

Karin Strauss

Jan Stephen Gray

Willard Harvey Wattenburg

Derek Chiou

Jeremy Halden Fowers

Blaise Aguera Y Arcas

Stephen William Keckler

Jie Jerry Liu

Edmund Bernard Nightingale

Prashanth Hosabettu Mohan

Daniel Lo

Gagan Gupta

Bart J F De Smet

Gabriel H Loh

Thomas Moscibroda

Jaron Zepel Lanier

Andrew David Wilson

Galen Clyde Hunt

Michael F Cohen

Joel Steven Kollin

Burton Jordan Smith

Aman Kansal

David A Maltz

Kristofer Hellick Reierson

Luis Henrique Ceze

Asta Jane Roseway

David Alexander Molnar

Robert Greg McDonald

Kalin Ovtcharov

David Tennyson Harper, Iii

Angela Mele Anderson

Michael Konstantinos Papamichael

Lior Shapira

David T Harper

Thomas A Weaver

Alistair K Chan

Eyal Ofek

Andrew F Glew

John D Davis

Rodney Wayne Smith

Feng Zhao

Nissanka Arachchige Bodhi Priyantha

Jiahe Helen Wang

James R Larus

Adrian Travis

Emmanouil Koukoumidis

Ming Gang Liu

Andreas Georgiou

Benjamin Livshits

Dimitrios Lymberopoulos

David B Wecker

Kenneth Hiroshi Eguro

David Bradley Probert, Jr

Thomas M McWilliams

Abhishek Kumar Singh

Stuart Schechter

Gerald Reuben DeJean

Yusuf Cagatay Tekmen

Jinwen Xi

Alexander Nikolavich Moshchuk

Embodiments of the present disclosure include systems and methods for reducing operations for training neural networks. A plurality of training data selected from a training data set is used as a plurality of inputs for training a neural network. The neural network includes a plurality of weights. A plurality of loss values are determined based on outputs generated by the neural network and expected output data of the plurality of training data. A subset of the plurality of loss values are determined. An average loss value is determined based on the subset of the plurality of loss values. A set of gradients is calculated based on the average loss value and the plurality of weights in the neural network. The plurality of weights in the neural network are adjusted based on the set of gradients.

Reducing operations for training neural networks

Embodiments of the present disclosure include systems and methods for performing data-aware model pruning for neural networks. During a training phase, a neural network is trained with a first set of data. During a validation phase, inference with the neural network is performed using a second set of data that causes the neural network to generate a first set of outputs at a layer in the neural network. During the validation phase, a plurality of mean values and a plurality of variance values are calculated based on the first set of outputs. A plurality of entropy values are calculated based on the plurality of mean values and the plurality of variance values. A second set of outputs are pruned based on the plurality of entropy values. The second set of outputs are generated by the layer of the neural network using a third set of data.

Data-aware model pruning for neural networks

Hardware and methods for neural network processing are provided. A method in a hardware node including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes performing using the MVU a first type of instruction that can only be performed by the MVU to generate a first result. The method further includes performing a second type of instruction that can only be performed by one of the multifunction units and generating a second result and without storing the any of the two results in a global register, passing the second result to the second multifunction and the third multifunction unit.

Neural network processing with chained instructions

Embodiments of the present disclosure include a system for optimizing an artificial neural network by configuring a model, based on a plurality of training parameters, to execute a training process, monitoring a plurality of statistics produced upon execution of the training process, and adjusting one or more of the training parameters, based on one or more of the statistics, to maintain at least one of the statistics within a predetermined range. In some embodiments, artificial intelligence (AI) processors may execute a training process on a model, the training process having an associated set of training parameters. Execution of the training process may produce a plurality of statistics. Control processor(s) coupled to the AI processor(s) may receive the statistics, and in accordance therewith, adjust one or more of the training parameters to maintain at least one of the statistics within a predetermined range during execution of the training process.

System for training an artificial neural network

Apparatus and methods are disclosed for controlling execution of memory access instructions in a block-based processor architecture using a hardware structure that generates a relative ordering of memory access instruction in an instruction block. In one example of the disclosed technology, a method of executing an instruction block having a plurality of memory load and/or memory store instructions includes decoding an instruction block encoding a plurality of memory access instructions and generating data indicating a relative order for executing the memory access instructions in the instruction block and scheduling operation of a portion of the instruction block based at least in part on the relative order data. In some examples, a store vector data register can store the generated relative ordering data for use in subsequent instances of the instruction block.

Implicit program order

Methods, systems and computer program products are provided for improving performance (e.g., reducing power consumption) of a hardware accelerator (e.g., neural processor) comprising hybrid or analog multiply and accumulate (MAC) processing elements (PEs). Selective variation of the precision of an array of MAC PEs may reduce power consumption of a neural processor. Power may be conserved by dynamically controlling the precision of analog to digital (ADC) output bits for one or more MAC PEs. Dynamic control of ADC output bit precision may be based on precision information determined during training and/or post-training (e.g., quantization) of an artificial intelligence (AI) neural network (NN) model implemented by the neural processor. Precision information may include a range of dynamic precision for each of a plurality of nodes of a computation graph for the AI NN model.

Analog MAC aware DNN improvement

Embodiments of the present disclosure include systems and methods for providing hierarchical and shared exponent floating point data types. First and second shared exponent values are determined based on exponent values of a plurality of floating point values. A third shared exponent value is determined based the first shared exponent value and the second shared exponent value. First and second difference values are determined based on the first shared exponent value, the second shared exponent value, and the third shared exponent value. Sign values and mantissa values are determined for the plurality of floating point values. The sign value and the mantissa value for each floating point value in the plurality of floating point values, the third shared exponent value, the first difference value, and the second difference value are stored in a data structure for a shared exponent floating point data type.

Hierarchical and shared exponent floating point data types

A reach matrix scheduler circuit for scheduling instructions to be executed in a processor is disclosed. The scheduler circuit includes an N&#xd7;R matrix wake-up circuit, where 'N' is the instruction window size of the scheduler circuit, and 'R' is the 'reach' within the instruction window of the matrix wake-up circuit, with &#x2018;R&#x2019; being less than &#x2018;N&#x2019;. A grant line associated with each instruction request entry in the N&#xd7;R matrix wake-up circuit is coupled to &#x2018;R&#x2019; other instruction entries among the &#x2018;N&#x2019; instruction entries. When a producer instruction in an instruction request entry is ready for issuance, the grant line associated with the instruction request entry is activated so that any other instruction entries coupled to the grant line (i.e., within the 'reach' of the instruction request entry) that consume the produced value generated by the producer instruction are &#x201c;woken-up&#x201d; and subsequently indicated as ready to be issued.

Reach matrix scheduler circuit for scheduling instructions to be executed in a processor

Apparatus and methods are disclosed for throttling processor operation in block-based processor architectures. In one example of the disclosed technology, a block-based instruction set architecture processor includes a plurality of processing cores configured to fetch and execute a sequence of instruction blocks. Each of the processing cores includes function resources for performing operations specified by the instruction blocks. The processor further includes a core scheduler configured to allocate functional resources for performing the operations. The functional resources are allocated for executing the instruction blocks based, at least in part, on a performance metric. The performance metric can be generated dynamically or statically based on branch prediction accuracy, energy usage tolerance, and other suitable metrics.

Instruction block allocation

Systems and methods are disclosed for performing wide memory operations for a wide data cache line. In some examples of the disclosed technology, a processor having two or more execution lanes includes a data cache coupled to memory, a wide memory load circuit that concurrently loads two or more words from a cache line of the data cache, and a writeback circuit situated to send a respective word of the concurrently-loaded words to a selected execution lane of the processor, either into an operand buffer or bypassing the operand buffer. In some examples, a sharding circuit is provided that allows bitwise, byte-wise, and/or word-wise manipulation of memory operation data. In some examples, wide cache loads allows for concurrent execution of plural execution lanes of the processor.

Coupling wide memory interface to wide write back paths

Apparatus and methods are disclosed for controlling execution of memory access instructions in a block-based processor architecture using a hardware structure that indicates a relative ordering of memory access instruction in an instruction block. In one example of the disclosed technology, a method of executing an instruction block having a plurality of memory load and/or memory store instructions includes selecting a next memory load or memory store instruction to execute based on dependencies encoded within the block, and on a store vector that stores data indicating which memory load and memory store instructions in the instruction block have executed. The store vector can be masked using a store mask. The store mask can be generated when decoding the instruction block, or copied from an instruction block header. Based on the encoded dependencies and the masked store vector, the next instruction can issue when its dependencies are available.

Generation and use of memory access instruction order encodings

Technology related to training a neural network accelerator using mixed precision data formats is disclosed. In one example of the disclosed technology, a neural network accelerator is configured to accelerate a given layer of a multi-layer neural network. An input tensor for the given layer can be converted from a normal-precision floating-point format to a quantized-precision floating-point format. A tensor operation can be performed using the converted input tensor. A result of the tensor operation can be converted from the block floating-point format to the normal-precision floating-point format. The converted result can be used to generate an output tensor of the layer of the neural network, where the output tensor is in normal-precision floating-point format.

Training neural network accelerators using mixed precision data formats

Methods and apparatus are disclosed supporting a design flow for developing quantized neural networks. In one example of the disclosed technology, a method includes quantizing a normal-precision floating-point neural network model into a quantized format. For example, the quantized format can be a block floating-point format, where two or more elements of tensors in the neural network share a common exponent. A set of test input is applied to a normal-precision flooding point model and the corresponding quantized model and the respective output tensors are compared. Based on this comparison, hyperparameters or other attributes of the neural networks can be adjusted. Further, quantization parameters determining the widths of data and selection of shared exponents for the block floating-point format can be selected. An adjusted, quantized neural network is retrained and programmed into a hardware accelerator.

Flow for quantized neural networks

Systems and methods are disclosed for allocating resources to contexts in block-based processor architectures. In one example of the disclosed technology, a processor is configured to spatially allocate resources between multiple contexts being executed by the processor, including caches, functional units, and register files. In a second example of the disclosed technology, a processor is configured to temporally allocate resources between multiple contexts, for example, on a clock cycle basis, including caches, register files, and branch predictors. Each context is guaranteed access to its allocated resources to avoid starvation from contexts competing for resources of the processor. A results buffer can be used for folding larger instruction blocks into portions that can be mapped to smaller-sized instruction windows. The results buffer stores operand results that can be passed to subsequent portions of an instruction block.

Executing multiple programs simultaneously on a processor core

Growing community of inventors

Bellevue, WA, United States of America

Douglas Christopher Burger