International Business Machines Corporation

Oracle International Corporation

Rachit Arora

Shrinivas Kulkarni

Dharmesh Jain

Anjali Agarwal

Rakhi S Arora

Padmanabhan Krishnan

Subin Shekhar

Deepashree Gandhi

Sujith Chacko

Sunil R Ganatra

A method may include receiving a request for a big data query including characteristics and user parameters and accessing a set of rules based at least in part on respective properties of one or more big data query engines, the set of rules correlating the one or more characteristics, the one or more user parameters and the respective properties. The method may include determining a candidate list including a subset of the big data query engines, determined based on the set of rules. Using a machine learning model, the method may include generating respective probability scores for each big data query engine. The method may include selecting and executing the big data query using a particular big data query engine. The method may include identifying a trigger indicating a performance issue with the particular big data query engine and switching the execution to a second big data query engine.

Auto recognition of big data computation engine for optimized query runs on cloud platforms

Disclosed embodiments provide cluster creation using self-joining cluster nodes. Each node within a cluster is assigned a role of master or non-master. A master node initializes and creates a cluster. A non-master, or 'role' node initializes and joins a cluster established by the master. If the master has not yet initialized, the non-master node waits until the master node is operational and then joins the cluster. A description file associated with each node describes the services and configuration required for the node's assigned role. In this way, non-master nodes can be spawned independently of the master node, providing a greater level of scalability, improved fault tolerance, and increased availability.

Cluster creation using self-aware, self-joining cluster nodes

A mechanism is provided in a data processing system for creating clusters on demand. The mechanism installs a cluster on the data processing system. The cluster comprises a master node and a managed node having distributed software installed thereon. The mechanism stores state of cluster on external volumes and removes references specific to the cluster from the state of the cluster stored on the external volumes. The mechanism takes a snapshot of the state of the cluster stored on the external volumes to form a set of volume templates and takes a snapshot of images of the master node and the managed node to form a set of node type images. Responsive to receiving a request to create a specified cluster, the mechanism creates a new set of nodes on the data processing system based on the set of node type images, clones the set of volume templates to form a new set of volumes in the data processing system, mounts the new set of volumes to the new set of nodes, and recreates configurations based on details in the request.

Optimized creation of distributed storage and distributed processing clusters on demand

A mechanism is provided for automatically recovering one or more distributed cluster nodes on a host system. Responsive to the host system recovering, restoring, or restarting, a set of distributed clustered systems affected due to the reboot of the host system and details associated with each of the set of distributed clustered systems are identified. Using the details, a set of nodes that operate on the host system are identified. For each node: respective components and configurations are prepared; a heartbeat is initiated thereby causing the node to register with a cluster network and identify peers within their associated distributed clustered system; a determination is made as to whether the node has reestablished connection with its associated distributed clustered system; and, responsive to the node reestablishing connection with its associated distributed clustered system, service components are started on the node.

Self-recoverable multitenant distributed clustered systems

A mechanism is provided for allocating a hyper-threaded processor to nodes of multi-tenant distributed software systems. Responsive to receiving a request to provision a node of the multi-tenant distributed software system on the host data processing system, a cluster to which the node belongs is identified. Responsive to the node being a second type of node, responsive to determining that another second type of node in the same cluster has been provisioned on the host data processing system, and responsive to the number of unallocated VPs on different physical processors from that of the other second type of node being greater than or equal to the requested number of VPs for the second type of node, the requested number of VPs for the second type of node is allocated each to a different physical processor from that of the other second type of node.

Growing community of inventors

Bengaluru, India

Rachit Arora