International Business Machines Corporation

Platform Computing Corporation

Huawei Cloud Computing Technologies Co., Ltd.

Khalid Ahmed

Lior Aronovich

Zhenhua Hu

Renato Luiz De Freitas Cunha

Kuan Feng

Eduardo Rocha Rodrigues

Vincenzo Pasquantonio

David Bigagli

Jingwen Wang

Marco A S Netto

Junfeng Liu

Hai Long W Wen

Qi Wang

Lei Guo

Michael Feiman

Xiao Ming Bao

Christopher Andrew Norman Smith

Mark S Black

Marco As Netto

Leo Stutzmann

Rodney William Jones

Ming Q Xu

Yuen Chau Pang

Jun Feng Liu

A data query method is provided. The data query method includes: receiving from a client a first query request including a first data identifier; sending back first data indicated by the first data identifier to the client; determining a retrieving strategy based on a query time cost of the first data, a loading time cost of the first data, a data freshness target, and a query latency target, retrieving second data and a data relationship that are associated with the first data according to the retrieving strategy from a data destination, wherein the second data and the data relationship meet the data freshness target, and the data relationship includes a relationship between the second data and the first data; receiving from the client a second query request including a second data identifier; sending back the second data indicated by the second data identifier to the client.

Data query method and apparatus for adjusting data loading to reduce query latency

A technique for managing pooled resources in a distributed computing environment includes a resource manager that receives resource allocation requests from workload managers for running one or more applications. The applications are initiated by a consumer through at least one of the workload managers where each workload manager is configured to translate demand requests from the applications into resource allocation requests. A determination is made whether there are available resources to satisfy the resource allocation requests. Responsive to determining that there is an unmet resource allocation request, a candidate resource meeting some but not all attributes needed to satisfy the unmet resource allocation request is identified and reconfigured to have the attributes needed to satisfy the unmet resource allocation request.

Resource manager for managing the sharing of resources among multiple workloads in a distributed computing environment

A job executing on a computer system according to a scheduler configuration may be monitored. Responsive to determining that the job completed executing, a difference between an allocated resource allocated to the job and the job's actual resource usage may be computed. Responsive to determining that the difference meets a criterion, the scheduler configuration may be reconfigured to reduce the allocated resource for a submitted job associated with a given user.

Automatic reconfiguration of high performance computing job schedulers based on user behavior, user feedback, and job performance monitoring

A technique for managing pooled resources in a distributed computing environment includes a resource manager executable to receive resource allocation requests from a plurality of workload managers for running one or more applications. The applications are initiated by a consumer through the workload managers. Each workload manager is configured to translate demand requests from the applications into resource allocation requests. A determination is made whether there are available resources to satisfy the resource requests according to a resource allocation policy. Resources are allocated to the workload managers where the applications can continue running on the allocated resources if the resource manager suffers downtime or is taken offline. The resource allocation policy is determined by a consumer profile of the consumer specifying ownership and share rights to the requested resources.

A computer-implemented method according to one embodiment includes identifying a plurality of storage resources. Additionally, the method includes creating a storage capacity, where the storage capacity has a first plurality of associated attributes. Further, the method includes defining one or more data volumes for the storage capacity, where each of the one or more data volumes has a second plurality of associated attributes and inherits the first plurality of associated attributes. Further still, the method includes configuring one or more volume shares for each data volume, where each of the volume shares has a third plurality of associated attributes and inherits the first plurality of associated attributes as well as the second plurality of associated attributes.

Hierarchical management of storage capacity and data volumes in a converged system

Embodiments for automatic diagonal scaling of workloads in a distributed computing environment. For each of a plurality of resources of each of a plurality of application instances, a determination as to whether a change in allocation of at least one of the plurality of resources is required. Operations requirements are computed for each of the plurality of application instances, the computed requirements including vertical increase and decrease operations, and horizontal split and collapse operations. The vertical decrease and horizontal collapse operations are first processed, the vertical increase and horizontal split operations are ordered, and the vertical increase and horizontal split operations are subsequently processed based on the ordering, thereby optimizing application efficiency and utilization of the plurality of resources in the distributed computing environment.

Automatic diagonal scaling of workloads in a distributed computing environment

Various embodiments for adaptively computing and implementing changes in resource allocations using historical data in a distributed computing environment. Historical data on resource consumption levels of an application is maintained. Either of an increase operation or reduction operation is triggered for a resource required by an application instance of the application, where a predicted consumption level for the resource required by the application instance is computed. The increase or reduction operation is computed and implemented for the resource required by the application instance to accommodate the predicted consumption level, thereby optimizing application efficiency and resource utilization in the distributed computing environment.

Adaptive resource allocation operations based on historical data in a distributed computing environment

Embodiments for leveraging directed acyclic graph (DAG) information to group tasks for execution, by at least one processor. An Input/Output (I/O) cost for each task in a set of tasks determined to be within a same input host set is determined, and tasks within the set of tasks are grouped into task groups according to an applied allocation time. The task groups are launched commensurate with a calculated delay time from each input host set so as to overlap I/O phases and central processing unit (CPU) phases to improve CPU efficiency. The launched task groups are executed by an executor according to the calculated delay time.

Leveraging directed acyclic graph (DAG) information to group tasks for execution

In at least one embodiment of stream processing in a data processing system cluster, a driver requests, from a cluster manager, execution of a first task within a job including a plurality of tasks, as well as a first quantity of resources of the cluster to support execution of the task. At completion of execution of the first task by the cluster, the driver retains for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager. The driver determines a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.

Adaptive resource scheduling for data stream processing

A job in a high performance computing system is executed according to a current scheduler configuration. The job is monitored to determine the job's actual resource usage, the job's performance metric and the job's completion. Responsive to determining that the job completed executing, a difference is computed between allocated resources allocated according to the current scheduler configuration at job submission for execution on the high performance computing system and the job's actual resource usage at the job's completion, wherein the difference is computed for each of N number of jobs that complete executing on the high performance computing system, wherein all of the N number of jobs are associated with a given user. Responsive to determining that the difference for said each of N number of jobs meets a criterion, the current scheduler configuration is automatically reconfigured to reduce the allocated resources for the jobs associated with the given user.

Embodiments for leveraging directed acyclic graph (DAG) information to group tasks for execution, by at least one processor device. For a set of tasks, an input host set is determined for each task using a shuffle type and hosts used during a previous stage, and the tasks in the set of tasks determined to be within a same input host set are classified together. An Input/Output (I/O) cost for each task in the set of tasks is calculated and tasks within the set of tasks are grouped into task groups according to an applied allocation time. The task groups are launched commensurate with a calculated delay time from each input host set so as to overlap I/O phases and central processing unit (CPU) phases to improve CPU efficiency, the launched task groups being executed by an executor according to the calculated delay time.

A set of source files is stored in a shared storage repository for nodes of a distributed computing environment for software compilation. An object file is created based on at least a portion of the set of source files. A directed acyclic graph (DAG) is generated corresponding to a group of software build tasks and the relationship between the software build tasks based on the set of source files. A replication factor for the object file is determined based on the number of relationships of the object file identified from the DAG. The object file is stored in a local memory cache of at least one of the number of the nodes, wherein the number of the nodes is based on the replication factor for the object file.

Accelerating software builds

A resource manager for managing the sharing of resources among multiple workloads in a distributed computing environment. The resource manager comprises an allocation service component which applies a resource allocation policy to determine whether the requested resources may be allocated. The resource manager also comprises a dynamic provisioning service component which applies a reprovisioning policy to modify the make-up of the resources in response to unsatisfied resource requests. The allocation of resources to consumers may be determined in accordance with the resource ownership and share rights of the consumer on behalf of which the application or workload is being demanded.

Growing community of inventors

Markham, Canada

Khalid Ahmed