International Business Machines Corporation

Bryan Savoye Rosenburg

Yoonho Park

Kyung D Ryu

Chen-Yong Cher

Carlos Henrique Andrade Costa

Robert William Wisniewski

Orran Yaakov Krieger

Hubertus Franke

Robert Brett Tremaine

Thomas M Gooding

Alper Buyuktosunoglu

Pradip Bose

Todd Alan Inglett

Tova Roth

Marc Alan Auslander

James H Summers

Augusto J Vega

David Joel Edelsohn

Michael Hadley Skelton

Balaram Sinharoy

Ravi Nair

Hans Mikael Jacobson

Thomas Winters Fox

Hartmut Penner

Sameer Kumar

Michael E Aho

Sameh Sherif Sharkawi

David Lee Hermsmeier

Tracy Jay Kimbrel

Gary J Mincher

Fausto Artico

Pierre Lemarinier

Jin Ma

Guilherme C Januario

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Accelerating shared file checkpoint with local burst buffers

A method for managing a network queue memory includes receiving sensor information about the network queue memory, predicting a memory failure in the network queue memory based on the sensor information, and outputting a notification through a plurality of nodes forming a network and using the network queue memory, the notification configuring communications between the nodes.

Network resiliency through memory health monitoring and proactive management

An illustrative embodiment includes a method for checkpointing and restarting an application executing at least in part on one or more central processing units coupled to one or more hardware accelerators. The method comprises checkpointing the application at least in part by: transferring checkpoint data of the application to the one or more hardware accelerators; performing distributed compression of the application checkpoint data at least in part using the one or more hardware accelerators; and writing the compressed application checkpoint data to a storage device. The method further comprises restarting the application at least in part by: reading the compressed application checkpoint data from the storage device; transferring the compressed checkpoint data to one or more hardware accelerators; and performing distributed decompression of the application checkpoint data at least in part using said one or more hardware accelerators.

Using accelerators for distributed data compression and decompression to improve checkpoint / restart times

Techniques are disclosed for efficient handling of messages in computing systems that include tag matching capable hardware. A message management module provides for handling message events including application receives and channel notifications such that hardware tag matching can continuously run in hardware channels, such as network adapters. When the message event is an application receive the message management module adds the application receive to a tracking queue and determines if the application receive can be posted to a hardware channel capable of tag matching. When the message event is a channel notification, the message management module determines a message action using the message tracking queue and the information in the channel notification.

Messaging software management for network devices supporting hardware tag matching

A method is disclosed, as well as an associated apparatus and computer program product, for checkpointing using a plurality of communicatively coupled compute nodes. The method comprises acquiring health information for a first node of the plurality of compute nodes, and determining a first failure probability for the first node using the health information. The first failure probability corresponds to a predetermined time interval. The method further comprises selecting a second node of the plurality of compute nodes as a partner node for the first node. The second node has a second failure probability for the time interval. A composite failure probability of the first node and the second node is less than the first failure probability. The method further comprises copying checkpoint information from the first node to the partner node.

Checkpointing using compute node health information

A method, data processing system and program product utilize dynamic logical storage volume sizing for burst buffers or other local storage for computing nodes to optimize job stage in, execution and/or stage out.

Burst buffer dynamic logical volume sizing in high performance computing environment

Techniques for transferring files between machines include creating a zero-length target file on non-volatile storage, truncating the file to a desired size, and allocating storage on the non-volatile storage for each block of the target file. The technique also includes determining a logical block address (LBA) for each location in the target file. The technique further includes sending a request to an input/output (I/O) node to transfer a source file to the non-volatile storage, where the request includes a mapping between the LBAs and file offsets. The technique includes opening the source file and a block device at the I/O node. The technique further includes reading each block from the source file and writing each block to the target file on the non-volatile storage utilizing the block device, and then closing the source file and the block device.

File transfers between machines without target CPU intervention

A method for providing selective memory error protection responsive to a predictable failure notification associated with at least one portion of a memory in a computing system includes: obtaining an active error correcting code (ECC) configuration corresponding to the portion of the memory; determining whether the active ECC configuration is sufficient to correct at least one error in the portion of the memory affected by the predictable failure notification; when the active ECC configuration is insufficient to correct the error, determining whether data corruption can be tolerated by an application running on the computing system; when data corruption cannot be tolerated by the application, determining whether a stronger ECC level is available and, if a stronger ECC level is available, increasing a strength of the active ECC configuration; and when data corruption can be tolerated, performing page reassignment and aggregation of non-critical data.

Method and apparatus for selective and power-aware memory error protection and memory management

A method for selective duplication of subtasks in a high-performance computing system includes: monitoring a health status of one or more nodes in a high-performance computing system, where one or more subtasks of a parallel task execute on the one or more nodes; identifying one or more nodes as having a likelihood of failure which exceeds a first prescribed threshold; selectively duplicating the one or more subtasks that execute on the one or more nodes having a likelihood of failure which exceeds the first prescribed threshold; and notifying a messaging library that one or more subtasks were duplicated.

Methods, apparatus and system for selective duplication of subtasks

A computer detects a request by a process for access to a shadow control page, wherein the shadow control page allows the process access to one or more devices. The computer assigns the shadow control page and a key to the process associated with the request. The computer detects a request by the process via the assigned shadow control page for creation of a subset of devices from the one or more devices. The computer inputs information detailing an association between the subset of devices and the assigned key into a subset definition table, wherein the subset definition table includes one or more keys and one or more corresponding subsets.

Mechanism for controlling subset of devices

Data processing techniques are provided to increase a computational speed of iterative computations that are performed over a domain of data points, such as stencil computations. For example, a method includes loading a set of domain data points into a cache memory; obtaining an iteration count T, and a base stencil operator having a first set of coefficients; generating a convolved stencil operator having a second set of coefficients, wherein the convolved stencil operator is generated by convolving the base stencil operator with itself at least one time; and iteratively processing the set of domain data points in the cache memory using the convolved stencil operator no more than T/2 iterations to obtain final processing results. The final processing results are similar to processing results that would be obtained by iteratively processing the set of domain data points using the base stencil operator for the iteration count T.

Kernel convolution for stencil computation optimization

A method for providing notification of a predictable memory failure includes the steps of: obtaining information regarding at least one condition associated with a memory; calculating a memory failure probability as a function of the obtained information; calculating a failure probability threshold; and generating a signal when the memory failure probability exceeds the failure probability threshold, the signal being indicative of a predicted future memory failure.

Methods, apparatus and system for notification of predictable memory failure

A method for faulty memory utilization in a memory system includes: obtaining information regarding memory health status of at least one memory page in the memory system; determining an error tolerance of the memory page when the information regarding memory health status indicates that a failure is predicted to occur in an area of the memory system affecting the memory page; initiating a migration of data stored in the memory page when it is determined that the data stored in the memory page is non-error-tolerant; notifying at least one application regarding a predicted operating system failure and/or a predicted application failure when it is determined that data stored in the memory page is non-error-tolerant and cannot be migrated; and notifying at least one application regarding the memory failure predicted to occur when it is determined that data stored in the memory page is error-tolerant.

Growing community of inventors

Cortlandt Manor, NY, United States of America

Bryan Savoye Rosenburg