Qubole, Inc.

Rajat Venkatesh

Shubham Tagra

Vijay Mann

Joydeep Sen Sarma

Ankit Dixit

Ting Yao

Amogh Margoor

Raunaq Morarka

The invention is generally directed to systems and methods of automatically tuning big data workloads across various cloud platforms, the system being in communication with a cloud platform and a user, the cloud platform including data storage and a data engine. The system may include: a system information module in communication with the cloud platform; a static tuner in communication with the system information module; a cloud tuner in communication with the static tuner and the user; and an automation module in communication with the cloud tuner. Methods may include extracting information impacting or associated with the performance of the big data workload from the cloud platform; determining recommendations based at least in part on the information extracted; iterating through different hardware configurations to determine optimal hardware and data engine configuration; and applying the determined configuration to the data engine.

System and methods for auto-tuning big data workloads on cloud platforms

The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.

System and method for scheduling and running interactive database queries with service level agreements in a multi-tenant processing system

The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

Caching framework for big-data engines in the cloud

Growing community of inventors

Bengaluru, India

Rajat Venkatesh