International Business Machines Corporation

Sujay Sunil Parekh

Kun-Lung Wu

Joel Leonard Wolf

Joseph L Hellerstein

Kirsten Weale Hildrum

Deepak Rajan

Rohit Madhukar Khandekar

Sam Sampson Lightstone

Bugra Gedik

Henrique Andrade

Matthew Albert Huras

Yixin Diao

Vibhore Kumar

Kevin R Rose

Maheswaran Surendra

Xiaolan Zhang

Ronghua Zhang

Adam J Storm

Joseph Phillip Bigus

Christian Marcelo Garcia-Arellano

Mark Steven Squillante

Jayram S Thathachar

Donald Allen Schlosnagle

Jeffrey Robert Pilgrim

A method for allocating parallel, independent, data tasks includes receiving data tasks, each of the data tasks having a penalty function, determining a generic ordering of the data tasks according to the penalty functions, wherein the generic ordering includes solving an aggregate objective function of the penalty functions, the method further including determining a schedule of the data tasks given the generic ordering, which packs the data tasks to be performed.

Scheduling parallel data tasks

Techniques for composing an application resource model are disclosed. The techniques include obtaining operator-level metrics from an execution of a data stream processing application according to a first configuration, wherein the application is executed by nodes of the data stream processing system and the application includes processing elements comprised of multiple operators, wherein two or more of the operators are combined in a first combination to form a processing element according to the first configuration, generating operator-level resource functions from the first combination of operators based on the obtained operator-level metrics, and generating a processing element-level resource function using the generated operator-level resource functions to predict a model for the processing element formed by a second combination of operators, the processing element-level resource function representing an application resource model usable for predicting characteristics of the application executed according to a second configuration.

Application resource model composition from constituent components

Techniques for determining feasibility of a set of one or more operator partitioning constraints are provided. The techniques include receiving one or more sets of operator partitioning constraints, wherein each set of one or more constraints define one or more desired conditions for grouping together of operators into partitions and placing partitions on hosts, wherein each operator is embodied as software that performs a particular function, processing each set of one or more operator partitioning constraints to determine feasibility of each set of one or more operator partitioning constraints, creating and outputting one or more candidate partitions and one or more host placements for each set of feasible partitioning constraints, and creating and outputting a certificate of infeasibility for each set of infeasible partitioning constraints, wherein the certificate of infeasibility outlines one or more reasons for infeasibility.

Determining operator partitioning constraint feasibility

Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a stream processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Partitioning operator flow graphs

A method for scheduling a data processing job includes receiving the data processing job formed of a plurality of computing units, combining the plurality of computing units into a plurality of sets of tasks, each set including tasks of about equal estimated size, and different sets having different sized tasks, and assigning the tasks to a plurality of processors using a dynamic longest processing time (DLPT) scheme.

Reducing the response time of flexible highly data parallel task by assigning task sets using dynamic combined longest processing time scheme

Techniques for scheduling a plurality of jobs sharing input are provided. The techniques include partitioning one or more input datasets into multiple subcomponents, analyzing a plurality of jobs to determine which of the plurality of jobs require scanning of one or more common subcomponents of the one or more input datasets, and scheduling a plurality of jobs that require scanning of one or more common subcomponents of the one or more input datasets, facilitating a single scanning of the one or more common subcomponents to be used as input by each of the plurality of jobs.

Amortizing costs of shared scans

Techniques for composing an application resource model in a data stream processing system are disclosed. The application resource model may be used to understand what resources will be consumed by an application when executed by the data stream processing system. For example, a method for composing an application resource model for a data stream processing system comprises the following steps. One or more operator-level metrics are obtained from an execution of a data stream processing application in accordance with a first configuration. The application is executed by one or more nodes of the data stream processing system, and the application is comprised of one or more processing elements that are comprised of one or more operators. One or more operator-level resource functions are generated based on the obtained one or more operator-level metrics. A processing element-level resource function is generated based on the one or more generated operator-level resource functions. The processing element-level resource function represents an application resource model usable for predicting one or more characteristics of the application executed in accordance with a second configuration.

A software development service offering of constructing and embedding feedback control based algorithms into computer systems management software applications is delivered by a provider entity to a client organization. The provider is contracted by the client to prepare a set of control modeling assets based on identified computing system management problems and goals of the diem and to create a set of control solutions. Bi-directional links between the problems, assets and solutions are used by the provider to develop feedback control based algorithms that solve the identified client computing management problems and meet the client's computing system goals.

System and method for service offering for feedback controller design and implementation for performance management in information technology systems

An adaptive throttling system for minimizing the impact of non-production work on production work in a computer system is provided. The adaptive throttling system throttles production work and non-production work to optimize production. The adaptive throttling system allows system administrators to specify a quantified limit on the performance impact of non-production or utility work on production work. The throttling rate of the utility is then automatically determined by a supervisory agent, so that the utilities' impact is kept within the specified limit. The adaptive throttling system adapts dynamically to changes in workloads so as to ensure that valuable system resources are well utilized and utility work is not delayed unnecessarily.

Adaptive throttling for data processing systems

Techniques for managing feedback control systems are provided. By way of example, a method of controlling performance of a managed system by a controller includes the following steps/operations. The controller issues a control value to the managed system to affect a performance of the managed system. The controller maintains a measurement time period having a variable start time within which the performance of the managed system is measured, such that the control value is given time to take effect on the managed system and a performance metric fed back to the controller from the managed system reflects the effect of the control value on the managed system.

Methods and apparatus for managing deadtime in feedback control queuing system

Adaptive throttling system for computing systems

Systems and methods are provided for optimizing the performance and/or allocation of constrained resources in a dynamic computing environment using adaptive regulatory control methods. For example, systems and methods for providing constrained optimization in a dynamic computing system implement model-based adaptive (self-tuning) regulatory control schemes that are designed to handle the system dynamics and which take into consideration control costs (such as the overheads of changing resource allocations and performance degradation due to transient load imbalances) to find an optimal solution. To facilitate practical application, a dual control architecture is disclosed which combines a heuristic fixed step control process that is implemented when there is no valid system model for model-based control. A system model is continually obtained and validated during run-time to adapt control parameters to variations in system dynamics.

Systems and methods for providing constrained optimization using adaptive regulatory control

An adaptive throttling system for minimizing the impact of non-production work on production work in a computer system. The adaptive throttling system throttles production work and non-production work to optimize production. The adaptive throttling system allows system administrators to specify a quantified limit on the performance impact of non-production or utility work on production work. The throttling rate of the utility is then automatically determined by a supervisory agent, so that the utilities' impact is kept within the specified limit. The adaptive throttling system adapts dynamically to changes in workloads so as to ensure that valuable system resources are well utilized and utility work is not delayed unnecessarily.

Adaptive throttling system for data processing systems

A system and method are described for constructing and implementing generic software agents for automated tuning of computer systems and applications. The framework defines the modules and interfaces to allow agents to be created in a modular fashion. The specifics of the target system are captured by adaptors that provide a uniform interface to the target system. Data in the agent is managed by a metric manager, and controller modules implement the desired control algorithms. The modular structure and common interfaces allow for the construction of generic agents that are applicable to a wide variety of target systems, and can use a wide variety of control algorithms.

Growing community of inventors

New York, NY, United States of America

Sujay Sunil Parekh