Microsoft Technology Licensing, LLC

Torsten Hoefler

Mattheus C Heddes

Abdul Kabbani

Deepak Goel

Nishit Harshad Shah

Ankit More

Jonathan R Belk

Xiaoling Xu

Amar Phanishayee

Kenneth Andrew Colwell

Embodiments of the present disclosure include systems and methods for training neural networks based on dual pipeline architectures. In some embodiments, a first set of compute elements are configured to implement a first set of layers of a first instance of a neural network. A second set of compute elements are configured to implement a second set of layers of the first instance of the neural network. The second set of compute elements are further configured to implement a first set of layers of a second instance of the neural network. The first set of compute elements are further configured to implement a second set of layers of the second instance of the neural network. The first set of layers of the first instance of the neural network and the first set of layers of the second instance of the neural network are each configured to receive training data.

Training neural networks based on dual pipeline architectures

A first ratio of a sending rate limit to a full line rate for a link in the computing network is accessed. A second ratio of a sending window size to W_max for the link is accessed. W_max is the maximum allowed window size or the window size that utilizes an end-to-end path for the link. One or more of the first or second ratio is used to determine an amount to reduce the sending rate or window for the link in response to an indication of network congestion in the link.

Fairness and congestion control convergence

It is determined that a computing node is contributing to a network congestion event. A congestion notification message is generated. A timing profile is determined for sending the congestion notification message based on the level of the network congestion event. Based on the timing profile, the congestion notification message is forwarded to the computing node determined to be contributing to the network congestion event.

Weighted random early back-to-sender notification

A computer implemented method includes encoding a packet in a source endpoint of a multi-path communication network, the packet having a hash seed for use by routers to route the packet through the multi-path communication network to a destination endpoint. Network performance is tracked for the packet at the source endpoint. The hash seed is modified as a function of the network performance. The packet is re-sent such that the modified hash seed is used to route the packet to the destination endpoint.

Adaptive packet routing

An entropy value is generated for a data packet to be transmitted on a computing network. The entropy value is usable to select or change a network path for the data packet. In response to receiving an acknowledgement message for the data packet, the entropy value is saved in a storage structure if the entropy value is acknowledged as not congested. When transmitting an additional data packet, the oldest saved entropy from the storage structure is reused and the oldest saved entropy value is invalidated.

Recycled entropies packet spraying

Embodiments of the present disclosure include a digital circuit and method for multi-stage compression. Digital data values are compressed using a multi-stage compression algorithm and stored in a memory. A decompression circuit receives the values and performs a partial decompression. The partially compressed values are provided to a processor, which performs the final decompression. In one embodiment, a vector of N length compressed values are decompressed using a first bit mask into two N length sets having non-zero values. The two N length sets are further decompressed using two M length bit masks into M length sparse vectors, each having non-zero values.

Method and apparatus for compressing and decompressing sparse data sets

One example provides an integrated computing device, comprising one or more computing clusters, and one or more network controllers, each network controller comprising a local data notification queue to queue send message notifications originating from the computing clusters on the integrated computing device, a remote data notification queue to queue receive message notifications originating from network controllers on remote integrated computing devices, a local no-data notification queue to queue receive message notifications originating from computing clusters on the integrated computing device, and a connection scheduler configured to schedule sending of data from memory on the integrated computing device when a send message notification in the local data notification queue is matched with a receive message notification in the remote data notification queue, and to schedule sending of receive message notifications from the local no-data notification queue.

Message communication between integrated computing devices

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism may be implemented using topology that combines sequences of processors. In one embodiment, the present disclosure includes a computer system comprising a plurality of processor groups, the processor groups each comprising a plurality of processors. A plurality of network switches are coupled to subsets of the plurality of processor groups. A subset of the processors in the processor groups may be configurable to form sequences, and the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various alternative configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipeline parallelism, layer parallelism, or combinations thereof.

Distributed processing architecture

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism may be implemented using topology that combines sequences of processors. In one embodiment, the present disclosure includes a computer system comprising one or more processor groups, the processor groups each comprising a plurality of processors. A plurality of network switches are coupled to subsets of the plurality of processor groups. In one embodiment, the switches may be optical network switches. Processors in the processor groups may be configurable to form sequences, and the network switches are configurable to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various alternative configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipeline parallelism, layer parallelism, or combinations thereof.

Growing community of inventors

Zurich, Switzerland

Torsten Hoefler