Atscale, Inc.

Sarah Gerweck

David A Ross

Matthew Baird

David P Mariani

Rouzbeh Safaie

Joseph Barefoot

David Vigdor Schreibman

Josh Klahr

Daniel J Dosch

DaRen Drummond

Daryl Odnert

Patrick J Toole

Gaurav Nitin Shetti

Trystan R Leftwich

A multidimensional database query engine processes a query request by forming a logical plan of subqueries for retrieving and assembling the data called for by the query request. A multidimensional database connector is invoked to transform a logical plan that defines and orders each subquery into a physical plan for accessing the data repositories where the data satisfying the query is stored. The query engine is invoked or called by an application and receives a query plan indicative of data repositories interrogated by query instructions in the query plan. For each data repository of the plurality of data repositories that may be interrogated by the query plan, a connector is defined based on commands for accessing each data repository. The connector associates each query instruction from the query plan with a corresponding repository command for accessing the data repository.

Datacube access connectors

A preferred aggregate store for a multidimensional database receives and stores intermediate data generated as part of a larger query for avoiding repetitive or redundant data retrievals incurring costly data paths. Aggregates include intermediate tables and sets of data, often generated from a join of two or more distinct data tables, combined for generating a query result. In the event an aggregate is called for, certain storage locations may be preferable to others due to network distance, fees for storage and processing, as in the case of cloud storage, or write accessibility. Designation of a preferred aggregate store ensures that aggregates called for by the query processing are stored in a beneficial location for cost and efficiency.

Preferred storage of aggregate data items

A data query and retrieval system receives a query request and identifies a plurality of data warehouses, or stores, where data needed to satisfy the query is stored. Data items, or fields, called for by the query request are grouped into sets based on fields stored at the same data warehouse. Individual data connections are generated for each data warehouse accessed, and a database view defined for the fields available at each respective data warehouse. The individual views are invoked in a single query statement by a query processor for accessing the respective data warehouses via the corresponding database view. In this manner, a query operator or user need not define multiple query expressions or perform separate queries for accessing data fields or dimensions residing at different data warehouses.

Distributed queries through dynamic views

A multidimensional database query engine processes a query request by forming a logical plan of subqueries for retrieving and assembling the data called for by the query request. A multi-pass analysis identifies a granularity of facts needed to fulfill the query request. A recursive analysis parses the query request and identifies components comprising the full query request. The analysis derives a subquery from each component, and identifies dependencies on other subqueries. The subqueries are arranged in a tree structure based on the dependencies. The tree represents subqueries as nodes, with query operations denoted by parent nodes for the dependent subqueries. The result is a hierarchical tree of subqueries associated based on operations between the subqueries and dependent subqueries descending from their parent subqueries.

Hierarchical datacube query plan generation

A query syntax analysis and postprocessing system and method receives a query request specifying values of a data type directed to a database without native support for the queried data type. A query engine or process for receiving a query request defined by a query syntax traverses the query request for a specification of an unsupported value or expression. In a large, distributed database environment, the query request may implicate multiple physical data stores, each having specific formats and recognized syntax. Values of unsupported or non-native types, or expressions that evaluate to an unsupported or non-native type, are identified and replaced with an expression that is recognized by the legacy database.

Query intake for legacy databases

The present disclosure provides a detailed description of techniques used in systems, methods, and in computer program products for dynamic aggregate generation and updating for high performance querying of large datasets. Certain embodiments are directed to technological solutions for determining at least one aggregate of selected virtual cube attributes (e.g., measures, dimensions, etc.) describing a virtual multidimensional data model of a subject database, and generating an aggregate table and a set of aggregate metadata for the aggregate. In some embodiments, an aggregate database statement configured to operate on the subject database can be issued to generate the aggregate table and/or aggregate metadata. Further, the aggregate can be dynamically determined responsive to receiving a database statement configured to operate on the virtual multidimensional data model representing the subject database. Also, the aggregate table can comprise one or more partitions in an aggregate view to facilitate aggregate management and/or quality.

Dynamic aggregate generation and updating for high performance querying of large datasets

Techniques are presented for applying fine-grained client-specific rules to divide (e.g., chunk) data statements to achieve cost reduction and/or failure rate reduction associated with executing the data statements over a subject dataset. Data statements for the subject dataset are received from a client. Statement attributes derived from the data statements are processed with respect to fine-grained rules and/or other client-specific data to determine whether a data statement chunking scheme is to be applied to the data statements. If a data statement chunking scheme is to be applied, further analysis is performed to select a data statement chunking scheme. A set of data operations are generated based at least in part on the selected data statement chunking scheme. The data operations are issued for execution over the subject dataset. The results from the data operations are consolidated in accordance with the selected data statement chunking scheme and returned to the client.

Data statement chunking

Techniques for determining user authorization to access data in dynamically generated database structures are presented. A method can commence upon receiving database statements from a user that is authorized to access a set of data in a database. A physical plan derived from the received database statements to operate on the database is inspected for references to dynamically generated data structures such as aggregates. A modified physical plan having no aggregates and/or other altered database structures is used to issue low latency database statements to operation on the database for verifying user access authorization to the underlying data. In some cases, the foregoing database statements are based on a virtual multidimensional data model. In other cases, a low latency directive is included in the modified physical plan to facilitate a low latency authorization response.

Data access authorization for dynamically generated database structures

Techniques are presented for rule-based selection of alternate data sources for multidimensional data statements. A virtual multidimensional data model is implemented to represent datasets that are accessed at various data sources. Derivative cubes generated from the virtual multidimensional data model are structured to have a respective data source metadata layer that is populated at data statement execution time to identify a target data source. Data source selection rules are established to map the attributes of data statements to target data sources. The data source selection rules are evaluated subject to data statement attributes derived from detected data statements to dynamically select target data sources for the data statements. The derivative cubes codify, in their respective data source metadata layers, a unique set of data source attributes identifying the target data sources. The derivative cubes are accessed to facilitate execution of the data statements on datasets at the target data sources.

Rule-based data source selection

Techniques are presented for performing enhanced real-time data statement monitoring and control without requiring modifications to data analysis client applications or data storage environments. A data statement processing layer is implemented between the data analysis client applications and the data storage environments. A data statement management agent interacts with an event bus in the data statement processing layer. The data statement management agent accesses event objects on the event bus that are generated during the processing of data statements. Rules are applied to select particular event objects. The selected event objects are transformed into event messages that are delivered to the client applications to monitor the processing of the data statements. Control commands received from the client application are transformed into event objects that are issued to the event bus to control the processing of the data statement.

Data statement monitoring and control

Techniques are presented for rule-based selection of query engines for performing data operations on a variety of datasets stored in heterogeneous data storage environments. Data statements for a subject dataset are mapped to a logical data structure representation of the subject dataset to formulate a logical plan for the data statements. The data statements are analyzed to derive a set of statement attributes associated with the data statements. Query engine attributes describing the query engines available to query various instances of the subject dataset are accessed. A set of query engine selection rules are evaluated subject to the statement attributes and/or the query engine attributes to dynamically select a target query engine for the data statements. A physical plan structured for the target query engine is generated from the logical plan. The physical plan is executed on the target query engine to perform data operations on the subject dataset.

Growing community of inventors

Santa Clara, CA, United States of America

Sarah Gerweck