Efficient inter-chip interconnect topology for distributed parallel deep learning

Abstract

Embodiments of the present disclosure present a hyper-square interconnect topology and advanced ring-based AllReduce operations. In some embodiments, a topology is provided that is an improvement over conventional interconnect topologies by eliminating delays associated with long wirings. In some embodiments, computing nodes are divided into sub-sections to better allocate computing tasks, and the system can be optimized to divide up the computing nodes by maximizing the number of square sub-sections in the topology. In some embodiments, the system can be optimized to select square sub-sections first for each computing task. Each sub-section can comprise some computing nodes or all computing nodes in the hyper-square interconnect topology. This flexibility allows the hyper-square interconnect topology to utilize the computing nodes more efficiently by assigning appropriate numbers of computing nodes to each computing task based on the computing need of the computing task.