Qubole, Inc.

Other

Mayank Ahuja

Joydeep Sen Sarma

Hariharan Iyer

Shrikanth Shankar

Prakhar Jain

Ajaya Agrawal

Sivaramakrishnan Narayanan

Systems and methods for automatically scaling a big data system. Methods include determining, at a first time, a first number of nodes for a cluster to process a request; assigning an amount of nodes equal to the first number of nodes to the cluster; determining a rate of progress of the request; determining, at a second time based on the rate of progress a second number of nodes; and modifying the amount of nodes to equal the second number of nodes. Systems include a cluster manager, to add and/or remove any nodes; the big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager including a big data interface for communicating with the big data system; a cluster manager interface for communicating with the cluster manager; and a cluster state machine.

Systems and methods for auto-scaling a big data system

The present invention is generally directed to systems and methods of providing automatic scaling pure-spot clusters. Such dusters may be dynamically rebalanced for further costs savings. In accordance with some methods of the present invention may include a method of utilizing a cluster in a big data cloud computing environment where instances may include reserved on-demand instances for a set price and on-demand spot instances that may be bid on by a user, the method including: creating one or more stable nodes, comprising spot instances with a bid price above a price for an equivalent on-demand instance; creating one or more volatile nodes, comprising spot instances with a bid price below a price for an equivalent on-demand instance; using one or more of the stable nodes as a master node; and using the volatile nodes as slave nodes.

Pure-spot and dynamically rebalanced auto-scaling clusters

The present invention is generally directed to systems and methods of provisioning, and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

Heterogeneous auto-scaling big-data clusters in the cloud

The present invention is generally directed to systems and methods of provisioning and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

Growing community of inventors

Cupertino, CA, United States of America

Mayank Ahuja