International Business Machines Corporation

Carlos Henrique Andrade Costa

Yoonho Park

Chen-Yong Cher

Bryan Savoye Rosenburg

Kyung D Ryu

Charles Ray Johns

Mudhakar Srivatsa

Raghu Kiran Ganti

Abdullah Kayi

Joshua M Rosenkranz

Linsong Chu

A system, computer program product, and method are provided for distributed data workflow semantics. A pipeline, such as a machine learning (ML) pipeline, is represented in a data flow graph (DFG). The represented pipeline is subject to annotations, with the annotations including pipeline nodes and object references. The pre-processed pipeline is subject to execution or processing with the annotated object references capturing object lineage. Output from the executed pipeline is constructed and a corresponding control signal is formatted to dynamically and selectively control an operatively coupled physical hardware device or software.

Annotation of a machine learning pipeline with operational semantics to support distributed lineage tracking

A system, computer program product, and method are provided for distributed data workflow semantics. A pipeline, such as a machine learning pipeline, is represented in a data flow graph (DFG) with nodes and edges. The represented nodes are configured to be annotated with an operational semantic. On order of execution of the pipeline is discovered through the node annotation(s) represented in the annotated DFG, and execution of the pipeline is based on the discovered order. A control signal formatted based on the executed pipeline is configured to dynamically and selectively control an operatively coupled physical hardware device.

Annotation of a machine learning pipeline with operational semantics

A method for managing a network queue memory includes receiving sensor information about the network queue memory, predicting a memory failure in the network queue memory based on the sensor information, and outputting a notification through a plurality of nodes forming a network and using the network queue memory, the notification configuring communications between the nodes.

Network resiliency through memory health monitoring and proactive management

Methods and systems for shuffling data are described. A processor may generate pair data from source data. The processor may insert the pair data into local tuple spaces. In response to a request for a particular key, the processor may determine a presence of the requested key in a global tuple space. The processor may, in response to a presence of the requested key in the global tuple space, update the global tuple space. The update may be based on the pair data among the local tuple spaces including the existing key. The processor may, in response to an absence of the requested key in the global tuple space, insert pair data including the missing key from the local tuple spaces into the global tuple space. The processor may fetch the requested pair data, and may shuffle the fetched data to generate a dataset.

Data shuffling with hierarchical tuple spaces

Methods and systems for shuffling data to generate a dataset are described. A first map module may generate first pair data, and a second map module may generate second pair data, from source data. The first map module may insert the first pair data into a first local tuple space accessible to the first map module. The second map module may insert the second pair data into a second local tuple space accessible to the second map module. A shuffle module may request pair data that includes a particular key. The first and second pair data may be inserted into a global tuple space accessible by the first and second map modules. The shuffle module may identify the requested pair data in the global tuple space, and may fetch the identified pair data from a memory. The shuffle module may shuffle the fetched pair data to generate the dataset.

A method is disclosed, as well as an associated apparatus and computer program product, for checkpointing using a plurality of communicatively coupled compute nodes. The method comprises acquiring health information for a first node of the plurality of compute nodes, and determining a first failure probability for the first node using the health information. The first failure probability corresponds to a predetermined time interval. The method further comprises selecting a second node of the plurality of compute nodes as a partner node for the first node. The second node has a second failure probability for the time interval. A composite failure probability of the first node and the second node is less than the first failure probability. The method further comprises copying checkpoint information from the first node to the partner node.

Checkpointing using compute node health information

A method for providing selective memory error protection responsive to a predictable failure notification associated with at least one portion of a memory in a computing system includes: obtaining an active error correcting code (ECC) configuration corresponding to the portion of the memory; determining whether the active ECC configuration is sufficient to correct at least one error in the portion of the memory affected by the predictable failure notification; when the active ECC configuration is insufficient to correct the error, determining whether data corruption can be tolerated by an application running on the computing system; when data corruption cannot be tolerated by the application, determining whether a stronger ECC level is available and, if a stronger ECC level is available, increasing a strength of the active ECC configuration; and when data corruption can be tolerated, performing page reassignment and aggregation of non-critical data.

Method and apparatus for selective and power-aware memory error protection and memory management

A method for selective duplication of subtasks in a high-performance computing system includes: monitoring a health status of one or more nodes in a high-performance computing system, where one or more subtasks of a parallel task execute on the one or more nodes; identifying one or more nodes as having a likelihood of failure which exceeds a first prescribed threshold; selectively duplicating the one or more subtasks that execute on the one or more nodes having a likelihood of failure which exceeds the first prescribed threshold; and notifying a messaging library that one or more subtasks were duplicated.

Methods, apparatus and system for selective duplication of subtasks

A method for providing notification of a predictable memory failure includes the steps of: obtaining information regarding at least one condition associated with a memory; calculating a memory failure probability as a function of the obtained information; calculating a failure probability threshold; and generating a signal when the memory failure probability exceeds the failure probability threshold, the signal being indicative of a predicted future memory failure.

Methods, apparatus and system for notification of predictable memory failure

A method for faulty memory utilization in a memory system includes: obtaining information regarding memory health status of at least one memory page in the memory system; determining an error tolerance of the memory page when the information regarding memory health status indicates that a failure is predicted to occur in an area of the memory system affecting the memory page; initiating a migration of data stored in the memory page when it is determined that the data stored in the memory page is non-error-tolerant; notifying at least one application regarding a predicted operating system failure and/or a predicted application failure when it is determined that data stored in the memory page is non-error-tolerant and cannot be migrated; and notifying at least one application regarding the memory failure predicted to occur when it is determined that data stored in the memory page is error-tolerant.

Growing community of inventors

White Plains, NY, United States of America

Carlos Henrique Andrade Costa