Graphcore Limited

Dominic Alexander Masters

Carlo Luschi

Alan Graham Alexander

Stephen Felix

Simon Christian Knowles

Mrudula Gore

Edward Andrews

Godfrey Da Costa

Badreddine Noune

Antoine Labatie

Zachary Frank Eaton-Rosen

A processing device comprises a register configured to store a count value indicating a number of times overflow events have resulted from arithmetic operations performed by the processing device. An execution unit of the device, in response to performing an arithmetic operation having a result which extends beyond one of the predefined limit values for the floating-point format, stores a result value that is within the predefined limit values, and cause the count value to be incremented. The count value provides a performant way of determining the number of overflow events that have occurred during the arithmetic processing performed by the execution unit. The count value provides a metric that provides a measure of the inaccuracy imparted into the results of the application processing by overflow events.

Overflow event counter

A computer-implemented method of training a deep neural network, comprising, for each of one or more batches of training examples: processing the data in a forward pass through the layers of the network, by: applying a set of network weights to the input data to obtain a set of weighted inputs, normalising the weighted inputs based on statistics computed for each training example, transforming the normalised inputs by affine transformation parameters, applying an activation function to the transformed normalised inputs to obtain post-activation values, and normalizing the post-activation values based on one or more proxy variables sampled from a distribution defined by proxy distribution parameters, the normalization applied independently of training example; processing the data in a backward pass through the network to determine updates to learnable parameters comprising network weights, affine transformation parameters, and proxy distribution parameters, and updating the learnable parameters to optimise a predefined loss function.

Processing data batches in a multi-layer network

A processing device comprising: a control register configured to store a scaling factor; at least one execution unit configured to execute instructions to perform arithmetic operations on input floating-point numbers provided according to a first floating-point format, wherein each of the input floating-point numbers provided according to the first floating-point format comprises a predetermined number of bits, wherein the at least one execution unit is configured to, in response to execution of an instance of a first of the instructions: perform processing of a first set of the input floating-point numbers to generate a result value, the result value provided in a further format and comprising more the predetermined number of bits, enabling representation of a greater range of values than is representable in the first floating-point format; and apply the scaling factor specified in the control register to increase or decrease an exponent of the result value.

Processing device for intermediate value scaling

Growing community of inventors

Bristol, United Kingdom

Dominic Alexander Masters