Apple Inc.

Nvidia Corporation

Christopher L Mills

Kenneth Waters

Sung Hee Park

Youchang Kim

Suk Hwan Lim

D Amnon Silverstein

Sheng Lin

David R Pope

Liran Fishel

Erik K Norden

Seungjin Lee

Fernando Alberto Mujica

Jaewon Shin

Jianping Zhou

Michael J M Toksvig

Harikrishna Madadi Reddy

Atul Garg

Simon Wolfenden Butler

Ignatius B Tjandrasuwita

Justin Michael Mahan

Derek E Pappas

Shankar Moni

Matthew Raymond Longnecker

David Amnon Silverstein

Embodiments of the present disclosure relate to circular buffers in a neural processor circuit. The neural processor circuit includes multiple neural engine circuits and a data processor circuit coupled to at least one of the neural engine circuits. The at least one neural engine circuit performs at least convolution operations. The data processor circuit includes a circular buffer, and a flow control circuit coupled to the circular buffer. The flow control circuit generates at least one addressing parameter that defines wrapping of data in the circular buffer. The circular buffer controls data flow in the neural processor circuit by storing first data associated with the at least one neural engine circuit so that the first data is wrapped around in the circular buffer. An addressing layout of the first data wrapped around in the circular buffer is defined by the at least one addressing parameter.

Circular buffering in neural network processor

Embodiments of the present disclosure relate to binary comparison operations (e.g., Boolean operations) and reduction operations in a neural processor circuit to enable implementation of conditional operations without software control. The neural processor circuit includes a neural engine circuit and a planar engine circuit coupled to the neural engine circuit. The neural engine circuit performs a convolution operation to generate output data. The planar engine circuit includes a binary comparator circuit and a filter circuit coupled to the binary comparator circuit. The binary comparator circuit performs a binary comparison operation on a tensor from the output data to generate a conditional tensor. The filter circuit performs a reduction operation for each patch of the conditional tensor to generate a respective reduced value of multiple reduced values associated with a corresponding channel of multiple channels of the conditional tensor.

Binary comparison and reduction operations in neural network processor

A neural processor includes one or more neural engine circuits for performing convolution operations on input data corresponding to one or more tasks to generate output data. The neural engine circuits process the input data having a power-of-two (P2) shape. The neural processor circuit also includes a data processor circuit. The data processor circuit fetches source data having a non-power-of-two (NP2) shape. The source data may correspond to data of a machine learning model. The data processor circuit also reshapes the source data to generate reshaped source data with the P2 shape. The data processor circuit further sends the reshaped source data to the one or more neural engine circuits as the input data for performing convolution operations. In some cases, the data processor circuit may also perform padding on the source data before the source data is reshaped to the P2 shape.

Processing non-power-of-two work unit in neural processor circuit

Embodiments of the present disclosure relate to a tensor access operation circuit in a neural processor circuit. The neural processor circuit further includes a data processor circuit and at least one neural engine circuit. The tensor access operation circuit indirectly accesses at least a region of a source tensor in a system memory having a rank, and maps one or more source components of the source tensor into an input tensor having another rank. The data processor circuit stores an output version of the input tensor obtained from the tensor access operation circuit and sends the output version of the input tensor as multiple of units of input data to the at least one neural engine circuit. The at least one neural engine circuit performs at least convolution operations on the units of input data and at least one kernel to generate output data.

Multi-dimensional tensor support extension in neural network processor

Embodiments relate to a neural processor that include a plurality of neural engine circuits and one or more planar engine circuits. The plurality of neural engine circuits can perform convolution operations of input data of the neural engine circuits with one or more kernels to generate outputs. The planar engine circuit is coupled to the plurality of neural engine circuits. The planar engine circuit generates an output from input data that corresponds to output of the neural engine circuits or a version of input data of the neural processor. The planar engine circuit can be configured to multiple modes. In a pooling mode, the planar engine circuit reduces a spatial size of a version of the input data. In an elementwise mode, the planar engine circuit performs an elementwise operation on the input data. In a reduction mode, the planar engine circuit reduces the rank of a tensor.

Multi-mode planar engine for neural processor

A neural processor includes neural engines for performing convolution operations on input data corresponding to one or more tasks to generate output data. The neural processor also includes a data processor circuit coupled to external system memory. The data processor circuit includes a buffer for storing the output data from the neural engines. The neural processor further includes a task manager coupled to the data processor circuit. The task manager receives a context-switch task. The context-switch task specifies a switch of the data processor circuit from handling an outgoing task to an incoming task. The task manager sends configuration data of the context-switch task to cause the data processor circuit to transmit the output data corresponding to the outgoing task from the buffer to the external system memory. The data processor circuit also fetches data corresponding to the incoming task from the external system memory to the buffer.

Task context switch for neural processor circuit

A neural processor includes neural engines for performing convolution operations on input data corresponding to one or more tasks to generate output data. The neural processor circuit also includes a data processor circuit that is coupled to one or more neural engine. The data processor circuit receives the output data from the neural engine and generates a branching command from the output data. The neural processor circuit further includes a task manager that is coupled to the data processor circuit. The task manager receives the branching command from the data processor circuit. The task manager enqueues one of two or more segment branches according to the received branching command. The two or more segment branches are subsequent to a pre-branch task segment that includes the pre-branch task. The task manager transmits a task from the selected one of the segment branches to data processor circuit to perform the task.

Branching operation for neural processor circuit

Embodiments relate to a neural processor that includes one or more neural engine circuits and planar engine circuits. The neural engine circuits can perform convolution operations of input data with one or more kernels to generate outputs. The planar engine circuit is coupled to the plurality of neural engine circuits. A planar engine circuit can be configured to multiple modes. In an elementwise mode, the planar engine circuit may combine two tensors by performing operations element by element. The planar engine circuit may support elementwise operation for two tensors that are in different sizes and ranks. The planar engine circuit may perform a broadcasting operation to duplicate one or more values across one or more channels to make a smaller tensor matching the size of the larger tensor.

Broadcasting mode of planar engine for neural processor

Embodiments relate to a neural engine circuit of a neural network processor circuit that performs a convolution operation on input data in a first mode and a parallel sorting operation on input data in a second mode. The neural engine circuit includes a plurality of operation circuits and an accumulator circuit coupled to the plurality of operation circuits. The plurality of operation circuits receives input data. In the first mode, the plurality of operation circuits performs multiply-add operations of a convolution on the input data using a kernel. In the second mode, the plurality of operation circuits performs a portion of a parallel sorting operation on the input data. In the first mode, the accumulator circuit receives and stores first results of the multiply-add operations. In the second mode, the accumulator circuit receives and stores second results of the parallel sorting operation.

Multi-operational modes of neural engine circuit

Embodiments of the present disclosure relate to a texture unit circuit in a neural processor circuit. The neural processor circuit includes a tensor access operation circuit with the texture unit circuit, a data processor circuit, and at least one neural engine circuit. The texture unit circuit fetches a source tensor from a system memory by referencing an index tensor in the system memory representing indexing information into the source tensor. The data processor circuit stores an output version of the source tensor obtained from the tensor access operation circuit and sends the output version of the source tensor as multiple of units of input data to the at least one neural engine circuit. The at least one neural engine circuit performs at least convolution operations on the units of input data and at least one kernel to generate output data.

Texture unit circuit in neural network processor

Embodiments relate to a neural processor circuit with scalable architecture for instantiating one or more neural networks. The neural processor circuit includes a data buffer coupled to a memory external to the neural processor circuit, and a plurality of neural engine circuits. To execute tasks that instantiate the neural networks, each neural engine circuit generates output data using input data and kernel coefficients. A neural processor circuit may include multiple neural engine circuits that are selectively activated or deactivated according to configuration data of the tasks. Furthermore, an electronic device may include multiple neural processor circuits that are selectively activated or deactivated to execute the tasks.

Scalable neural network processing engine

A neural processor circuit includes one or more planar engine circuits that perform non-convolution operations in parallel with convolution operations performed by one or more neural engine circuits. The neural engine circuits perform the convolution operations on neural input data corresponding to one or more neural engine tasks to generate neural output data. The planar engine circuits perform non-convolution operations on planar input data corresponding to one or more planar engine tasks to generate planar output data. A data processor circuit in the neural processor circuit addresses data dependency between the one or more neural engine tasks and the one or more planar engine tasks by controlling reading of the neural output data as the planar input data by the planar engine circuits or reading of the planar output data as the neural input data by the neural engine circuits.

Asynchronous task execution for neural processor circuit

Embodiments relate to a neural engine circuit that includes an input buffer circuit, a kernel extract circuit, and a multiply-accumulator (MAC) circuit. The MAC circuit receives input data from the input buffer circuit and a kernel coefficient from the kernel extract circuit. The MAC circuit contains several multiply-add (MAD) circuits and accumulators used to perform neural networking operations on the received input data and kernel coefficients. MAD circuits are configured to support fixed-point precision (e.g., INT8) and floating-point precision (FP16) of operands. In floating-point mode, each MAD circuit multiplies the integer bits of input data and kernel coefficients and adds their exponent bits to determine a binary point for alignment. In fixed-point mode, input data and kernel coefficients are multiplied. In both operation modes, the output data is stored in an accumulator, and may be sent back as accumulated values for further multiply-add operations in subsequent processing cycles.

Neural network processor for handling differing datatypes

Embodiments of the present disclosure relate to a neural engine of a neural processor circuit having multiple multiply-add circuits and an accumulator circuit coupled to the multiply-add circuits. The multiply-add circuits perform multiply-add operations of a three dimensional convolution on a work unit of input data using a kernel to generate at least a portion of output data in a processing cycle. The accumulator circuit includes multiple batches of accumulators. Each batch of accumulators receives and stores, after the processing cycle, the portion of the output data for each output depth plane of multiple output depth planes. A corresponding batch of accumulators stores, after the processing cycle, the portion of the output data for a subset of the output channels and for each output depth plane.

Growing community of inventors

Saratoga, CA, United States of America

Christopher L Mills