Alibaba Group Holding Limited

T-head (shanghai) Semiconductor Co., Ltd.

Alibaba Singapore Holding Private Limited

Liang Han

Jian Feng Chen

Jian Chen

Xiaowei Jiang

Yang Jiao

YunXiao Zou

Chengyuan Wu

Rong Zhong

Guoyu Zhu

Ye Lu

The presented systems enable efficient and effective network communications. In one embodiment, a system comprises a parallel processing unit (PPU) included in a chip and a plurality of interconnects in an inter-chip network (ICN) configured to communicatively couple a plurality of PPUs that communicate with one another via the ICN. Corresponding communications are configured in accordance with routing tables. The routing tables can be stored and reside in registers of an ICN subsystem included in the PPU and include indications of minimum links available to forward a communication from the PPU and another PPU. Respective ones of the routing tables include indications of a correlation between the minimum links and respective ones of a plurality of egress ports that are available for communication coupling to the respective ones of PPUs that are possible destination PPUs. The routing tables can include indications of a single path between two PPUs per respective information communication flow.

Parallel dataflow routing scheme systems and methods

The time required to read from and write to multi-terabyte memory chips in an inter-chip network can be reduced by breaking each memory chip into a number of memory spaces, and then individually addressing each memory space with an address that identifies the memory space, a line number within the memory space, and a number of transmit/receive ports to be used to access the line number.

Routing scheme for heterogeneous interconnected-chip networks using distributed shared memory

Systems and methods for exchanging synchronization information between processing units using a synchronization network are disclosed. The disclosed systems and methods include a device including a host and associated neural processing units. Each of the neural processing units can include a command communication module and a synchronization communication module. The command communication module can include circuitry for communicating with the host device over a host network. The synchronization communication module can include circuitry enabling communication between neural processing units over a synchronization network. The neural processing units can be configured to each obtain a synchronized update for a machine learning model. This synchronized update can be obtained at least in part by exchanging synchronization information using the synchronization network. The neural processing units can each maintain a version of the machine learning model and can synchronize it using the synchronized update.

Neural processing unit synchronization systems and methods

The present disclosure provides a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group, a computing node B of the first group with a computing node B of the second group, a computing node C of the first group with the computing node C of the second group, and a computing node D of the first group with a computing node D of the second group, the method comprising: syncing across a first dimension of computing nodes using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes along the first dimension; and broadcasting synced data across a second dimension of computing nodes using a second ring connection.

Efficient and more advanced implementation of ring-allreduce algorithm for distributed parallel deep learning

Virtualization techniques can include determining virtual function routing tables for the virtual parallel processing units (PPUs) from a logical topology of a virtual function. A first mapping of the virtual PPUs to a first set of a plurality of physical PPUs can be generated. Virtualization can also include generating a first set of physical function routing tables for the first set of physical PPUs based on the virtual function tables and the first virtual PPU to physical PPU mapping. An application can be migrated from the first set of physical PPUs to a second set of PPUs by generating a second mapping of the virtual PPUs to a second set of a plurality of physical PPUs. A second set of physical function routing table for the second set of physical PPUs can also be generated based on the virtual function tables and the second virtual PPU to physical PPU mapping.

Parallel processing unit virtualization

Systems and methods for heterogenous hardware acceleration are disclosed. The systems and methods can include a neural network processing unit comprising compute tiles. Each of a first set of the compute tiles can include a first tensor array configured to support operations in a first number format. Each of a second set of the compute tiles can include a second tensor array configured to support operations in a second number format, the second number format supporting a greater range or a greater precision than the first number format, and a de-quantizer configured to convert data in the first number format to data in the second number format. The systems and methods can include neural network processing units, multi-chip hardware accelerators and distributed hardware accelerators including low-precision components for performing interference tasks and high-precision components for performing training tasks. Transfer learning tasks can be performed using low-precision components and high-precision components.

Heterogeneous deep learning accelerator

A system includes a high-bandwidth inter-chip network (ICN) that allows communication between parallel processing units (PPUs) in the system. For example, the ICN allows a PPU to communicate with other PPUs on the same compute node or server and also with PPUs on other compute nodes or servers. In embodiments, communication may be at the command level (e.g., at the direct memory access level) and at the instruction level (e.g., the finer-grained load/store instruction level). The ICN allows PPUs in the system to communicate without using a PCIe bus, thereby avoiding its bandwidth limitations and relative lack of speed. The respective routing tables comprise information of multiple paths to any given other PPU.

Systems and methods for multi-branch routing for interconnected chip networks

Embodiments of the present disclosure present a hyper-square interconnect topology and advanced ring-based AllReduce operations. In some embodiments, a topology is provided that is an improvement over conventional interconnect topologies by eliminating delays associated with long wirings. In some embodiments, computing nodes are divided into sub-sections to better allocate computing tasks, and the system can be optimized to divide up the computing nodes by maximizing the number of square sub-sections in the topology. In some embodiments, the system can be optimized to select square sub-sections first for each computing task. Each sub-section can comprise some computing nodes or all computing nodes in the hyper-square interconnect topology. This flexibility allows the hyper-square interconnect topology to utilize the computing nodes more efficiently by assigning appropriate numbers of computing nodes to each computing task based on the computing need of the computing task.

Efficient inter-chip interconnect topology for distributed parallel deep learning

Improved convolutional layers for neural networks can obtain an input feature map comprising groups of channels. Each group of channels can include one or more channels having a predetermined size. The predetermined sizes can differ between the groups. The convolutional layer can generate, for each one of the groups of channels, an output channel. Generation of the output channel can include resizing the channels in the remaining groups of channels to match the predetermined size of the each one of the groups of channels. Generation can further include combining the channels in the each one of the groups with the resized channels and applying the combined channels to a convolutional sub-layer to generate the output channel.

Multi-size convolutional layer

An accelerator system can include one or more clusters of eight processing units. The processing units can include seven communication ports. Each cluster of eight processing units can be organized into two subsets of four processing units. Each processing unit can be coupled to each of the other processing units in the same subset by a respective set of two bi-directional communication links. Each processing unit can also be coupled to a corresponding processing unit in the other subset by a respective single bi-directional communication link. Input data can be divided into one or more groups of four subsets of data. Each processing unit can be configured to sum corresponding subsets of the input data received on the two bi-directional communication links from the other processing units in the same subset with the input data of the respective processing unit to generate a respective set of intermediate data. Each processing unit can be configured to sum a corresponding set of intermediate data received on the one bi-directional communication link from the corresponding processing unit in the other subset with the intermediate data of the respective processing unit to generate respective sum data. Each processing unit can be configured to broadcast the sum data of the respective processing unit to the other processing units in the same subset on the respective sets of two bi-directional communication links.

Topologies and algorithms for multi-processing unit interconnected accelerator systems

The present disclosure provides a method for syncing data of a computing task across a plurality of groups of computing nodes. Each group including a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple each of computing nodes A-D with corresponding computing nodes A-D in each of a plurality of neighboring groups. The method comprises syncing data at a computing node of the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node; and broadcasting synced data from the node to the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node.

Hyper-square implementation of tree AllReduce algorithm for distributed parallel deep learning

Systems and methods for performing parallel computation are disclosed. The system can include: a task manager; and a plurality of cores coupled with the task manager and configured to respectively perform a set of parallel computation tasks based on instructions from the task manager, wherein each of the plurality of cores further comprises: a processing unit configured to generate a first output feature map corresponding to a first computation task among the set of parallel computation tasks; an interface configured to receive one or more instructions from the task manager to collect external output feature maps corresponding to the set of parallel computation tasks from other cores of the plurality of cores; a reduction unit configured to generate a reduced feature map based on the first output feature map and received external output feature maps.

Method and system for performing parallel computations to generate multiple output feature maps

The present disclosure provides a system comprising: a first group of computing nodes and a second group of computing nodes, wherein the first and second groups are neighboring devices and each of the first and second groups comprising: a set of computing nodes A-D, and a set of intra-group interconnects, wherein the set of intra-group interconnects communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C; and a set of inter-group interconnects, wherein the set of inter-group interconnects communicatively couple computing node A of the first group with computing node A of the second group, computing node B of the first group with computing node B of the second group, computing node C of the first group with computing node C of the second group, and computing node D of the first group with computing node D of the second group.

Efficient and more advanced implementation of ring-AllReduce algorithm for distributed parallel deep learning

The present disclosure relates to a unified memory apparatus having a unified storage medium and one or more processing units. The unified memory apparatus can include a first storage module having a first plurality of storage cells, and a second storage module having a second plurality of storage cells, each of the first and second plurality of storage cells configured to store data and to be identified by a unique cell identifier. The one or more processing units are in communication with the unified storage medium and the processing units are configured to receive a first input data from one of the first plurality of storage cells, receive a second input data from one of the second plurality of storage cells, and generate an output data based on the first and second input data.

Growing community of inventors

San Mateo, CA, United States of America

Liang Han