Rubrik, Inc.

Qubole, Inc.

Shubham Tagra

Vijay Karthik

Akhilesh Krishnan

Rajat Venkatesh

Dhawal Upadhyay

Akshay Agrawal

Ankit Dixit

Vijay Mann

Vaiapuri Ramasubramaniam

Joydeep Sen Sarma

Ting Yao

Raunaq Morarka

Harmandeep Singh

Sayantan Jana

Methods, systems, and devices for data management are described. In some systems, a data management system (DMS) may obtain a full snapshot of a data block and incremental snapshots that include data associated with changes to partitions of the data block since the full snapshot. The full snapshot and the incremental snapshots may be stored as a snapshot chain. A most recently obtained incremental snapshot in the chain may be marked for deletion. Accordingly, the DMS may select, from the snapshot chain, an incremental snapshot to convert to a new full snapshot as part of a reverse operation. The incremental snapshot may be a next most recent incremental snapshot in the snapshot chain that is not marked for deletion. The DMS may perform the reverse operation to reverse an order of the snapshot chain and convert the incremental snapshot to the new full snapshot.

Full snapshot selection for reverse operations

Methods, systems, and devices for data management are described. A data enrichment service supported by a data management system (DMS) may receive, from a first application in a destination computing environment of the DMS, a set of enrichment definitions for metadata synchronization between the first application and a second application in a source computing environment of the DMS. A change data capture (CDC) service supported by the DMS may generate a set of data records that correspond to metadata changes associated with the second application. The data enrichment service may transform the set of data records by using data enrichment components to modify the set of data records according to the set of enrichment definitions provided by the first application. The data enrichment components may be dynamically partitioned into groups that execute in parallel. The second application may push the enriched data records to the first application in real-time.

Techniques for source-side metadata enrichment

Methods, systems, and devices for data management are described. A backup cluster may store incremental or base snapshots of computing objects. When a snapshot expires, a data management system (DMS) that manages the backup cluster may in some cases merge or consolidate the expired snapshot with a non-expired snapshot to create a new merged snapshot. In some cases, however, consolidation may be deferred until a chain of multiple expired snapshots satisfies one or more heuristic thresholds, to conserve resources. Example heuristic thresholds may be a length of the chain, an amount of space reclaimable by consolidating the snapshots, an age of the expired snapshots in the chain of incremental snapshots, or an amount of free space on the backup cluster.

Batch consolidation of computing object snapshots

A data management system (DMS) may select, prior to obtaining a first snapshot of a first virtual machine (VM) and from among one or more snapshots previously obtained by the DMS, a second snapshot to use for deduplication of the first snapshot. The DMS may obtain the first snapshot after selecting the second snapshot. Obtaining the first snapshot may include writing a first subset of data blocks from the first VM to a snapshot file for the first snapshot based on the first subset of the data blocks from the first VM being different from a first corresponding subset of the second snapshot and refraining from writing a second subset of the data blocks from the first VM to the snapshot file for the first snapshot based on the second subset of the data blocks from the first VM matching a second corresponding subset of the second snapshot.

Inline snapshot deduplication

Methods, systems, and devices for data management are described. A first application in a destination computing environment of a data management system (DMS) may determine that a bulk-push criterion is satisfied for a second application in a source computing environment of the DMS. The first application may transmit, to an asynchronous metadata service, a request indicating the second application for which the bulk-push criterion is satisfied. The request may be configured to cause the asynchronous metadata service to query a database in the source computing environment, identify a latest version of one or more rows that include metadata associated with the second application, and generate data records indicating the latest version of the one or more rows that include the metadata associated with the second application. The first application may receive the data records via an asynchronous data stream between the first application and the second application.

Techniques for asynchronously pushing metadata in bulk

The present invention is generally directed to systems and methods of determining and provisioning peak memory requirements in Structured Query Language Processing engines. More specifically, methods may include determining or obtaining a query execution plan; gathering statistics associated with each database table; breaking the query execution plan into one or more subtasks: calculating an estimated memory usage for each subtask using the statistics; determining or obtaining a dependency graph of the one or more subtasks; based at least in part on the dependency graph, determining which subtasks can execute concurrently on a single worker node; and totaling the amount of estimated memory for each subtask that can execute concurrently on a single worker node and setting this amount of estimated memory as the estimated peak memory requirement for the specefic database query.

Systems and methods for determining peak memory requirements in SQL processing engines with concurrent subtasks

The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.

System and method for scheduling and running interactive database queries with service level agreements in a multi-tenant processing system

The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

Growing community of inventors

Bengaluru, India

Shubham Tagra