International Business Machines Corporation

Kirsten Weale Hildrum

Joel Leonard Wolf

Deepak Rajan

Rohit Madhukar Khandekar

Nikhil Bansal

Kun-Lung Wu

David Tao

Lisa Karen Fleischer

Rohit Wagle

Sujay Sunil Parekh

Henrique Andrade

James R H Challenger

Richard Pervin King

Andrey Balmin

Philip Shi-lung Yu

Laura Wynter

Bugra Gedik

James R Giles

Vibhore Kumar

Oktay Gunluk

Viswanath Nagarajan

Frederick Douglis

Eugen Schenfeld

Xiaolan Zhang

Anshul Dawra

John Davis Palmer

William H Tetzlaff

Kanthi K Sarpatwar

Zubair Nabi

Elizabeth Suzanne Richards

Robert Saccone

Akshay Kumar Reddy Katta

Philip Shilung Yu

Wei Fan

Deepak Srinivas Turaga

Daniel Michael Kuchta

Nagui Halim

Yoonho Park

Lisa Amini

Laurent Schares

Marc A Taubenblatt

Chitra Venkatramani

Michael J E Spicer

Sunjay S Parekh

Phillippe L Selo

Philippe L Seto

Methods, systems, and computer program products for cross-platform scheduling with fairness and platform-specific optimization are provided herein. A method includes determining dimensions of a set of containers in which multiple tasks associated with a request are to be executed; assigning each of the containers to a processing node on one of multiple platforms based on the dimensions of the given container, and to a platform owner selected from the multiple platforms based on a comparison of resource requirements of each of the multiple platforms and the dimensions of the given container; and generating container assignments across the set of containers by incorporating the assigned node of each container in the set of containers, the assigned platform owner of each container in the set of containers, one or more scheduling requirements of each of the platforms, one or more utilization objectives, and enforcing a sharing guarantee of each of the platforms.

Cross-platform scheduling with long-term fairness and platform-specific optimization

Techniques, systems, and articles of manufacture for automated scheduling management of MapReduce flow-graph applications. A method includes determining a job schedule of MapReduce jobs within each of multiple MapReduce flows in a cluster environment, wherein said job schedule does not violate a precedence relationship within the corresponding MapReduce flow and reduces makespan of the corresponding MapReduce flow, determining a flow schedule for the multiple MapReduce flows based on consideration of a given metric, wherein said flow schedule comprises a number of slots allotted to each of the multiple MapReduce flows, and wherein said number of slots is less than or equal to a number of the one or more MapReduce jobs within each corresponding MapReduce flow, and transforming each job schedule into the flow schedule to allocate resources for the multiple MapReduce flows in the cluster environment without violating a precedence constraint of the multiple MapReduce flows.

Automated scheduling management of MapReduce flow-graph applications

A method for allocating parallel, independent, data tasks includes receiving data tasks, each of the data tasks having a penalty function, determining a generic ordering of the data tasks according to the penalty functions, wherein the generic ordering includes solving an aggregate objective function of the penalty functions, the method further including determining a schedule of the data tasks given the generic ordering, which packs the data tasks to be performed.

Scheduling parallel data tasks

A method, apparatus, and computer program product for scheduling stream-based applications in a distributed computer system with configurable networks are provided. The method includes choosing, at a highest temporal level, jobs that will run, an optimal template alternative for the jobs that will run, network topology, and candidate processing nodes for processing elements of the optimal template alternative for each running job to maximize importance of work performed by the system. The method further includes making, at a medium temporal level, fractional allocations and re-allocations of the candidate processing elements to the processing nodes in the system to react to changing importance of the work. The method also includes revising, at a lowest temporal level, the fractional allocations and re-allocations on a continual basis to react to burstiness of the work, and to differences between projected and real progress of the work.

Method, apparatus, and computer program product for scheduling work in a stream-oriented computer system with configurable networks

Techniques for scheduling one or more MapReduce jobs in a presence of one or more priority classes are provided. The techniques include obtaining a preferred ordering for one or more MapReduce jobs, wherein the preferred ordering comprises one or more priority classes, prioritizing the one or more priority classes subject to one or more dynamic minimum slot guarantees for each priority class, and iteratively employing a MapReduce scheduler, once per priority class, in priority class order, to optimize performance of the one or more MapReduce jobs.

Scheduling MapReduce jobs in the presence of priority classes

Techniques for scheduling multiple flows in a multi-platform cluster environment are provided. The techniques include partitioning a cluster into one or more platform containers associated with one or more platforms in the cluster, scheduling one or more flows in each of the one or more platform containers, wherein the one or more flows are created as one or more flow containers, scheduling one or more individual jobs into the one or more flow containers to create a moldable schedule of one or more jobs, flows and platforms, and automatically converting the moldable schedule into a malleable schedule.

Scheduling flows in a multi-platform cluster environment

Techniques for determining feasibility of a set of one or more operator partitioning constraints are provided. The techniques include receiving one or more sets of operator partitioning constraints, wherein each set of one or more constraints define one or more desired conditions for grouping together of operators into partitions and placing partitions on hosts, wherein each operator is embodied as software that performs a particular function, processing each set of one or more operator partitioning constraints to determine feasibility of each set of one or more operator partitioning constraints, creating and outputting one or more candidate partitions and one or more host placements for each set of feasible partitioning constraints, and creating and outputting a certificate of infeasibility for each set of infeasible partitioning constraints, wherein the certificate of infeasibility outlines one or more reasons for infeasibility.

Determining operator partitioning constraint feasibility

Techniques for partitioning an operator flow graph are provided. The techniques include receiving source code for a stream processing application, wherein the source code comprises an operator flow graph, wherein the operator flow graph comprises a plurality of operators, receiving profiling data associated with the plurality of operators and one or more processing requirements of the operators, defining a candidate partition as a coalescing of one or more of the operators into one or more sets of processing elements (PEs), using the profiling data to create one or more candidate partitions of the processing elements, using the one or more candidate partitions to choose a desired partitioning of the operator flow graph, and compiling the source code into an executable code based on the desired partitioning.

Partitioning operator flow graphs

A storage system includes a plurality of data vats, and a processor including an optimizing unit that optimizes a value of data stored in the storage system. The optimizing unit optimizes the value by computing and implementing an optimal decision for allocating new data to a first data vat of the plurality of data vats, moving existing data from at least a second data vat of the plurality of data vats to the first data vat, and deleting existing data from the first data vat, based on an amount of data in each of the plurality of data vats.

System and method for optimizing data in value-based storage system

A method for scheduling a data processing job includes receiving the data processing job formed of a plurality of computing units, combining the plurality of computing units into a plurality of sets of tasks, each set including tasks of about equal estimated size, and different sets having different sized tasks, and assigning the tasks to a plurality of processors using a dynamic longest processing time (DLPT) scheme.

Reducing the response time of flexible highly data parallel task by assigning task sets using dynamic combined longest processing time scheme

An apparatus and method for making fractional assignments of processing elements to processing nodes for stream-based applications in a distributed computer system includes determining an amount of processing power to give to each processing element. Based on a list of acceptable processing nodes, a determination of fractions of which processing nodes will work on each processing element is made. To update allocations of the amount of processing power and the fractions, the process is repeated.

Method and apparatus for assigning fractional processing nodes to work in a stream-oriented computer system

Growing community of inventors

Hawthorne, NY, United States of America

Kirsten Weale Hildrum