Skip-hop collective compute data transfer

Abstract

Techniques for performing collective compute operations are described. A collective compute operation can be performed in a logical ring of processing ranks formed by a set of rank groups that each contain a number of processing ranks including a primary rank and one or more secondary ranks. At each rank group, a primary rank receives an incoming data slice via an intranode interconnect from a previous primary rank at multiple hops away on the logical ring. A data transfer is performed between the primary rank and each secondary rank of the rank group. An outgoing data slice is then transferred from the primary rank of the rank group to the next primary rank at multiple hops away on the logical ring.