International Business Machines Corporation

Brian K Caufield

Xiaoyan Pu

Ron E Liu

Sriram Kolinjivadi Padmanabhan

Yong Li

Mi Wan Shum

Dong Jie Wei

Eric Allen Jacobson

Lawrence A Greene

DongJie Wei

Xin Yang

Chun H Sun

Samuel H K Wong

Fan Ding

Ajay Kumar Sood

Shyam R Mudambi

Arron J Harden

Julian J Vizor

Hung B Nguyen

Aarti D Borkar

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

Dynamic optimization of workload execution based on statistical data collection and updated job profiling

A data driven parallel sorting method includes distributing input data records to n partitions one by one in a circular manner. Each partition corresponds to a parallel sorting process with an allocated memory chunk sized to store m data records. The method also includes sorting, in parallel, current data records in respective memory chunks in respective partitions. The method also includes in response to distribution of data records of &#x2514;m/n&#x2518; rounds, circularly controlling one of the n partitions, and writing data records that have been sorted in the memory chunk of the partition into a mass storage as an ordered data chunk, and emptying the memory chunk. The method also includes in response to all data records being distributed, writing data chunks that have been sorted in respective memory chunks into the mass storage, and performing a merge sort on all ordered data chunks in the mass storage.

Data driven parallel sorting system and method

A method, computer program product, and system for dynamically distributing data for parallel processing in a computing system, comprising allocating a data buffer to each of a plurality of data partitions, where each data buffer stores data to be processed by its corresponding data partition, distributing data in multiple rounds to the data buffers for processing by the data partitions, where in each round the data is distributed based on a determined data processing capacity for each data partition, and where a greater amount of data is distributed to the data partitions with higher determined processing capacities, and periodically monitoring usage of each data buffer and re-determining the determined data processing capacity of each data partition based on its corresponding data buffer usage.

Dynamic data partitioning for optimal resource utilization in a parallel data processing system

Provided are techniques for multiple stage workload management. A staging queue and a run queue are provided. A workload is received. In response to determining that application resources are not available and that the workload has not been previously semi-started, the workload is added to the staging queue. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that run resources are available, the workload is started. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that the run resources are not available, adding the workload to the run queue.

Multiple stage workload management system

Provided are techniques for sharing a partitioned data set across parallel applications. Under control of a producing application, a partitioned data set is generated; a descriptor that describes the partitioned data set is generated; and the descriptor is registered in a registry. Under control of a consuming application, the registry is accessed to obtain the descriptor of the partitioned data set; and the descriptor is uses to determine how to process the partitioned data set.

Sharing a partitioned data set across parallel applications

A system identifies a performance bottleneck status in a parallel data processing environment by examining data flow associated with the parallel data processing environment to identify at least one operator, where an operator type is associated with at least one operator, at least one buffer, and a relationship that the buffer has with the operator, where the relationship is associated with the operator type. The system monitors the buffer to determine a buffer status associated with the buffer. The system applies a set of rules to identify an operator bottleneck status associated with the operator. The set of rules is applied to the operator, based on the operator type, the buffer status, and relationship that the buffer has with the operator. The system then determines a performance bottleneck status associated with the parallel data processing environment, based on the operator bottleneck status.

Apparatus and method for identifying performance bottlenecks in pipeline parallel processing environment

Data flow disruptions over a series of data processing operators can be detected by a computer system that generates a profile for data flow at an operator. The profile can include data input, processing, and output wait times. Using the profile, the system can detect potential flow disruptions. If the potential disruption satisfies a rule, it is considered a data flow disruption and a recommendation associated with the satisfied rule is identified. The recommendation and the operator identity is displayed.

Detection of data flow bottlenecks and disruptions based on operator timing profiles in a parallel processing environment

According to one embodiment of the present invention, a system tests jobs in a computing environment. The system creates a test case for one or more existing executable jobs without modifying the job design or recompiling the executable itself, wherein the test case includes one or more capture points in a job flow of the executable jobs and corresponding rules for capturing data, identification of data for testing the one or more executable jobs, and rules for comparing the captured data to expected results. The system captures the data at the one or more capture points in the job flow in accordance with the test case and generates a baseline of expected results. The system executes the test case to test the one or more executable jobs by capturing data at the one or more capture points generated from execution of the one or more executable jobs and compares result data from the one or more capture points to the baseline of expected results in accordance with the comparison rules to determine a result of the test. Embodiments of the present invention further include a method and computer program product for testing jobs in a computing environment in substantially the same manners described above.

Test framework for computing jobs

A method, computer program product and system for workload management for an Extract, Transform, and Load (ETL) system. A priority of each workload in a set of workloads is determined using a priority rule. In response to determining that the priority of a workload to be checked has a highest priority, it is indicated that the workload has the highest priority. It is determined whether at least one logical resource representing an ETL metric is available for executing the workload. In response to determining that the workload has the highest priority and that the at least one logical resource is available, it is determined that the workload is runnable.

Light weight workload management server integration

Methods and systems for graphically emphasizing a selected path through a diagram, where the diagram includes a number of nodes and a number of lines, the methods and systems including: applying a node highlight effect to a node of the number of nodes in the selected path, where applying the node highlight effect includes applying a node shadow to the node, applying a line highlight effect to a line of the number of lines in the selected path, where applying the line highlight effect includes applying a line shadow to the line, applying a node fade effect to a node of the number of nodes not in the selected path, and applying a line fade effect to a line of the number of lines not in the selected path.

Method and system for graphically emphasizing a path through a diagram

An embodiment of the present invention discloses an enterprise application packaging method and archive file. Applications and support components are developed by a developer, which are then included in the archive file. A revision control file indicates which files correlate to a present revision of the application package. A naming convention is applied to the application and support component files to indicate versions which correspond to each other. The archive file contains the file corresponding to the present version, as well as previous versions, as well as previous version revision control files so that, in the event of a failure or other error, the executing machine may revert to a previous version.

Growing community of inventors

Livermore, CA, United States of America

Brian K Caufield