International Business Machines Corporation

Ron E Liu

Brian K Caufield

Xiaoyan Pu

Sriram Kolinjivadi Padmanabhan

Yong Li

DongJie Wei

Xin Yang

Mi Wan Shum

Hanson Lieu

Chun H Sun

Eric Allen Jacobson

Krishna Kishore Bonagiri

Dong Jie Wei

A data driven parallel sorting method includes distributing input data records to n partitions one by one in a circular manner. Each partition corresponds to a parallel sorting process with an allocated memory chunk sized to store m data records. The method also includes sorting, in parallel, current data records in respective memory chunks in respective partitions. The method also includes in response to distribution of data records of &#x2514;m/n&#x2518; rounds, circularly controlling one of the n partitions, and writing data records that have been sorted in the memory chunk of the partition into a mass storage as an ordered data chunk, and emptying the memory chunk. The method also includes in response to all data records being distributed, writing data chunks that have been sorted in respective memory chunks into the mass storage, and performing a merge sort on all ordered data chunks in the mass storage.

Data driven parallel sorting system and method

A method includes a workload management (WLM) server that receives a first CHECK WORKLOAD command for a workload in a queue of the WLM server. It may be determined whether the workload is ready to run on a WLM client. If the workload is not ready to run, a wait time for the workload with the WLM server is dynamically estimated. The wait time is sent to the WLM client. If the workload is ready to run, then a response is sent to the WLM client that workload is ready to run.

Determining variable wait time in an asynchronous call-back system based on calculated average sub-queue wait time

Provided are techniques for improving data locality for parallel applications running in a big data distributed file system with a dynamic node group. In response to a consumer job starting to read one or more files in a big data distributed file system having multiple nodes, node group information for the one or more files to be read is retrieved, wherein the node group information identifies nodes from the multiple nodes on which a producer job wrote the one or more files, and the consumer job is assigned to the nodes identified by the node group information to allow for local reading of the one or more files by the consumer job.

Dynamic node group allocation

Provided are techniques for multiple stage workload management. A staging queue and a run queue are provided. A workload is received. In response to determining that application resources are not available and that the workload has not been previously semi-started, the workload is added to the staging queue. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that run resources are available, the workload is started. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that the run resources are not available, adding the workload to the run queue.

Multiple stage workload management system

Provided are techniques for sharing a partitioned data set across parallel applications. Under control of a producing application, a partitioned data set is generated; a descriptor that describes the partitioned data set is generated; and the descriptor is registered in a registry. Under control of a consuming application, the registry is accessed to obtain the descriptor of the partitioned data set; and the descriptor is uses to determine how to process the partitioned data set.

Sharing a partitioned data set across parallel applications

A system identifies a performance bottleneck status in a parallel data processing environment by examining data flow associated with the parallel data processing environment to identify at least one operator, where an operator type is associated with at least one operator, at least one buffer, and a relationship that the buffer has with the operator, where the relationship is associated with the operator type. The system monitors the buffer to determine a buffer status associated with the buffer. The system applies a set of rules to identify an operator bottleneck status associated with the operator. The set of rules is applied to the operator, based on the operator type, the buffer status, and relationship that the buffer has with the operator. The system then determines a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

Growing community of inventors

San Jose, CA, United States of America

Ron E Liu