International Business Machines Corporation

Other

Intel Corporation

Globalfoundries Inc.

Philip Heidelberger

Dong Chen

Alan G Gara

Mark Edwin Giampapa

Matthias Augustin Blumrich

Pavlos Michael Vranas

Martin Ohmacht

Paul William Coteus

Burkhard Steinmacher-Burow

Todd Edward Takken

Noel A Eisley

Dirk Hoenicke

Burkhard D Steinmacher-Burow

Valentina Salapura

Yutaka Sugawara

Sameer Kumar

Robert Matthew Senger

Thomas M Gooding

Michael A Blocksome

Dong Chen Chen

Gerard Vincent Kopcsay

Jeffrey John Parker

James Allan Kahle

Joseph D Ratterman

Peter Anthony Franaszek

George L Chiu

Krishnan Sugavanam

Gabor J Dozsa

Gyan Vatsaraja Bhanot

Ravi Nair

Hillery C Hunter

Robert William Wisniewski

Thomas Mario Cipolla

Rudolf Adriaan Haring

Craig Brian Stunkel

Charles J Archer

Vittorio R Castelli

Shawn Anthony Hall

David L Satterfield

Douglas R Miller

Amith R Mamidala

Robert E Walkup

Michael K Gschwind

Adam James Muff

Lawrence Shungwei Mok

Bruce Martin Fleischer

Alexandre E Eichenberger

Cyriel Johan Minkenberg

John A Gunnels

Kathryn Mary O'Brien

Mark Gregory Megerian

Sameh W Asaad

Thomas Winters Fox

John Kevin O'Brien

Chen-Yong Cher

Barry Marshall Trager

Peng Wu

Charles David Wait

Michael E Aho

John Timothy Robinson

James Anthony Marcella

Daniel A Faraj

Todd Alan Inglett

Jon K Kriegel

Andrew Thomas Tauferner

Lonny Lambrecht

Gheorghe Almasi

Kristan D Davis

Jose R Brunheroto

Fabrizio Petrini

Bogdan Prisacari

Michael B Mundy

Matthew Roger Ellavsky

Wolfgang Emil Denzel

James Lee Van Oosten

Samuel J Miller

German Rodriguez Herrera

Ruth Janine Poole

Thomas A Liebsch

Peter Boyle

Brant L Knudson

Ralph E Bellofatto

Norman Christ

Christopher Jon Johnson

Luis H Ceze

Kahn C Evans

William Maupin Stockdell

Alfred Thomas Watson, Iii

Brian E Smith

Vijayalakshmi Srinivasan

Richard Edwin Harper

Howard Thomas Olnowich

Mark James Jeanson

Swagath Venkataramani

Dan Edward Poff

Harold S Stone

Bob Richard Cernohous

Thomas Norman Barker

James C Sexton

Don Darrell Reed

The embodiments herein describe hybrid parallelism techniques where a mix of data and model parallelism techniques are used to split the workload of a layer across an array of processors. When configuring the array, the bandwidth of the processors in one direction may be greater than the bandwidth in the other direction. Each layer is characterized according to whether they are more feature heavy or weight heavy. Depending on this characterization, the workload of an NN layer can be assigned to the array using a hybrid parallelism technique rather than using solely the data parallelism technique or solely the model parallelism technique. For example, if an NN layer is more weight heavy than feature heavy, data parallelism is used in the direction with the greater bandwidth (to minimize the negative impact of weight reduction) while model parallelism is used in the direction with the smaller bandwidth.

Hybrid data-model parallelism for efficient deep learning

A computer network architecture includes a plurality N of first nodes, each first node having kports to a cluster network, where N and kare integers greater than 0; and a local network switch connected to each of the plurality of first nodes, but not to the cluster network. Each first node has kports to the local network switch, where k, is an integer greater than 0, and any two first nodes in the plurality of first nodes communicate with each other via the local network switch or via the cluster network.

Network architecture with locally enhanced bandwidth

A switch-connected dragonfly network and method of operating. A plurality of groups of row switches is organized according to multiple rows and columns, each row including multiple groups of row switches connected to form a two-level dragonfly network. A plurality of column switches interconnect groups of row switches along respective columns, a column switch associated with a corresponding group of row switches in a row. A switch port with a same logical port on a row switch at a same location in each group along the respective column connects to a same column switch. The switch-connected dragonfly network is expandable by adding additional rows, an added row comprising a two-level dragonfly network. A switch group of said added row associated with a column being connects to an available port at an existing column switch of said column by corresponding added S path link with no re-cabling of the switched network required.

Switch-connected Dragonfly network

A method, system and computer program product are disclosed for routing data packet in a computing system comprising a multidimensional torus compute node network including a multitude of compute nodes, and an I/O node network including a plurality of I/O nodes. In one embodiment, the method comprises assigning to each of the data packets a destination address identifying one of the compute nodes; providing each of the data packets with a toio value; routing the data packets through the compute node network to the destination addresses of the data packets; and when each of the data packets reaches the destination address assigned to said each data packet, routing said each data packet to one of the I/O nodes if the toio value of said each data packet is a specified value. In one embodiment, each of the data packets is also provided with an ioreturn value used to route the data packets through the compute node network.

I/O routing in a multidimensional torus network

Methods and systems for monitoring remote transmissions of messages among a plurality of nodes are described. A processing element in a first node may allocate a sequence number to a request to read and/or update data in a second node. The processing element may be different from main processors of the first node. The processing element may send the message and the sequence number to the second node. The processing element may modify a status of the sequence number to an active state, indicating a transmission of the message is pending. The processing element may, in response to a response from the second node, modify the status of the sequence number to an inactive state, indicating a completed transmission of the message. The processing element may, in response to no response from the second node within a time period, resend the message and the sequence number to the second node.

Reliability processing of remote direct memory access

A first method includes determining a total length of pending packets for a network link, determining a currently preferred power mode for the network link based on the total length of pending packets for the network link, and changing a current power mode for the network link to the currently preferred power mode. A corresponding apparatus is also disclosed herein. A second method includes determining a utilization for a network link, determining a currently preferred power mode for the network link based on the utilization for the network link, and changing a current power mode for the network link to the currently preferred power mode. A corresponding apparatus is also disclosed herein.

Power management of network links

A shared memory maintained by sender processes stores a sequence number counter per destination process. A sender process increments the sequence number counter in the shared memory in sending a message to a destination process. The sender process sends a data packet comprising the message and at least a sequence number specified by the sequence number counter. All of the sender processes share a sequence number counter per destination process, each of the sender processes incrementing the sequence number counter in sending a respective message. Receiver processes run on the hardware processor, each of the receiver processes maintaining a local memory counter on the memory, the local memory counter associated with a sending node. The local memory counter stores a sequence number of a message received from the sending node. The receiver process delivers incoming data packets ordered by sequence numbers of the data packets.

Reduced number of counters for reliable messaging

Embodiments of the invention provide a method, system and computer program product for embedding a global barrier and global interrupt network in a parallel computer system organized as a torus network. The computer system includes a multitude of nodes. In one embodiment, the method comprises taking inputs from a set of receivers of the nodes, dividing the inputs from the receivers into a plurality of classes, combining the inputs of each of the classes to obtain a result, and sending said result to a set of senders of the nodes. Embodiments of the invention provide a method, system and computer program product for embedding a collective network in a parallel computer system organized as a torus network. In one embodiment, the method comprises adding to a torus network a central collective logic to route messages among at least a group of nodes in a tree structure.

Embedding global barrier and collective in a torus network

An apparatus includes a collective switch hardware architecture, including an input arrangement circuit including multiple input ports and multiple outputs. The input arrangement circuit routes its multiple input ports to selected ones of its outputs. The collective switch hardware architecture includes collective reduction logic coupled to the multiple outputs of the input arrangement circuit and having multiple outputs. The collective reduction logic includes ALU(s) and arbitration and control circuitry. The ALU(s) and arbitration and control circuitry support multiple simultaneous collective operations from different collective classes, and support arbitrary input port and output port mapping to different collective classes. The collective switch hardware architecture further includes an output arrangement circuit including a multiple inputs coupled to the multiple outputs of the collective reduction logic and including multiple output ports. The output arrangement circuit is configured to route its multiple inputs to selected ones of its output ports.

Network switch architecture supporting multiple simultaneous collective operations

According to one embodiment of the present invention, a system for operating memory includes a first node coupled to a second node by a network, the system configured to perform a method including receiving the remote transaction message from the second node in a processing element in the first node via the network, wherein the remote transaction message bypasses a main processor in the first node as it is transmitted to the processing element. In addition, the method includes accessing, by the processing element, data from a location in a memory in the first node based on the remote transaction message, and performing, by the processing element, computations based on the data and the remote transaction message.

Remote processing and memory utilization

A method and system are disclosed for providing combined error code protection and subgroup parity protection for a given group of n bits. The method comprises the steps of identifying a number, m, of redundant bits for said error protection; and constructing a matrix P, wherein multiplying said given group of n bits with P produces m redundant error correction code (ECC) protection bits, and two columns of P provide parity protection for subgroups of said given group of n bits. In the preferred embodiment of the invention, the matrix P is constructed by generating permutations of m bit wide vectors with three or more, but an odd number of, elements with value one and the other elements with value zero; and assigning said vectors to rows of the matrix P.

Combined group ECC protection and subgroup parity protection

A method, system and memory controller for implementing memory hierarchy placement decisions in a memory system including direct routing of arriving data into a main memory system and selective injection of the data or computed results into a processor cache in a computer system. A memory controller, or a processing element in a memory system, selectively drives placement of data into other levels of the memory hierarchy. The decision to inject into the hierarchy can be triggered by the arrival of data from an input output (IO) device, from computation, or from a directive of an in-memory processing element.

Implementing selective cache injection

A system and method for enabling high-speed, low-latency global collective communications among interconnected processing nodes. The global collective network optimally enables collective reduction operations to be performed during parallel algorithm operations executing in a computer structure having a plurality of the interconnected processing nodes. Router devices are included that interconnect the nodes of the network via links to facilitate performance of low-latency global processing operations at nodes of the virtual network and class structures. The global collective network may be configured to provide global barrier and interrupt functionality in asynchronous or synchronized manner. When implemented in a massively-parallel supercomputing structure, the global collective network is physically and logically partitionable according to needs of a processing algorithm.

Collective network for computer structures

Growing community of inventors

Cortlandt Manor, NY, United States of America

Philip Heidelberger