International Business Machines Corporation

Paraccel LLC

Tableau Software, Inc.

Actian Corporation

Actian Sub Iii, Inc.

Red Brick Systems, Inc.

Richard Lee Cole

Yijou Chen

Eugene Szedenits, Jr

William Joseph McKenna

Sergei Perfilov

Aman Sinha

Latha S Colby

David G Wilhite, Jr

Galt Johnson

William McKenna

Edward P Haslam

Nasi Jazayeri

Heidi Lap Mun Lam

Jan Finis

Thomas Neumann

Tobias Muehlbauer

Daniel Ting

Manuel Then

Adrian Vogelsgesang

Lee E Schumacher

Craig J Bunger

Liqi Xu

Michael Haubenschild

Gary Ross Ricard

Mark John Anderson

Donovan Schneider

Marion Else Behnen

Larry William Youngren

Qi Jin

Wilson D Lee

Timo Pfahl

William S Davidson

William M Wagstaff

Holger Pirk

Peter B Passe

Ann M Koehler

Gopal B Mulagund

William J McKenna

William Mckenna

William J Mckenna

A method selects a federation engine to process a query. The system receives a query that specifies retrieval of data from multiple data sources at distinct physical locations. The system generates multiple federated query plans. Each federated query plan specifies execution of the query using a respective data source as the federation engine. For each federated query plan, the system forms a feature vector including two or more of: (1) a remote cost estimate for executing query operators on data sources distinct from the respective data source; (2) a local cost estimate for executing query operators on the respective data source; and (3) a data transfer cost estimate for transferring partial results from external data sources to the respective data source. The system predicts runtime for each federated query plan, selects a preferred federated query plan using the predicted runtimes, and the preferred federated query plan specifies the federation engine.

Federated query optimization

A method builds a regression model for predicting processing times for federated queries using a variety of data sources. The method includes obtaining federated queries (e.g., from benchmarks), and generates a plurality of federated query plans for each federated query. Each federated query plan corresponds to executing a respective federated query using a respective data source as the federation engine. The method includes forming feature vectors for each federated query plan based on cost estimations for executing the respective federated query plan and cost estimations for data transfer. The method further includes training a regression model, using the feature vectors for the plurality of federated query plans, to predict runtimes for executing federated queries using the variety of data sources as a federation engine. Some implementations use the trained regression model to determine a suitable federation engine for a given federated query.

A database engine includes one or more computing devices, each having one or more processors and memory. The memory stores programs configured for execution by the processors. The database engine receives a database query from a client, and parses the database query to build a query operator tree. The query operator tree includes a plurality of query operators. The database engine performs one or more optimization passes on the query operator tree, including a deduplication optimization pass, to form an optimized execution plan. The deduplication optimization pass includes determining that a first query operator is equivalent to a second query operator during a traversal of the query operator tree, and replacing the second query operator with a link to reuse results from the first query operator. The database engine executes the optimized execution plan to retrieve a result set from the database and returns the result set to the client.

Elimination of query fragment duplication in complex database queries

A method displays a user interface (UI) that includes a flow diagram having a plurality of nodes, receives user specification of a validation rule for a first node of the plurality of nodes in the flow diagram, and determines that an intermediate data set violates the validation rule. In response to determining that the first intermediate data set violates the validation rule, the method identifies errors corresponding to rows in the intermediate data set, and displays an error resolution UI that provides information about the errors. The error resolution UI includes a data flow trace region providing lineage of the errors in the flow. When a user selects an error in the data flow trace region, the data flow trace region is updated to provide the lineage of the selected error, including an updated graphic depiction for the respective error at each visually represented node.

Resolution of data flow errors using the lineage of detected error conditions

A parallel database system processes database queries invoking a user defined function (UDF). The database query is specified in a database query language, for example, the structured query language (SQL) and the UDF is specified in a programming language different from the database query language, for example, C/C++. The UDF may include multiple phases. The parallel database system generates optimized code for the UDF. The parallel database system includes a leader node and a plurality of compute nodes. The database query processes one or more database tables such that each compute node processes a slice of each database table. The leader node sets up a global context structure and provides it to each compute node. The global context may be shared across multiple executions of the query. The UDF may be used for efficient execution of extraction, transformation, and loading of data from external systems.

User defined functions for database query languages based on call-back functions

A database engine receives a database query from a client. The database engine parses the database query to build a query operator tree that includes a plurality of query operators. The database engine performs one or more optimization passes on the query operator tree, including a deduplication optimization pass, to form an optimized execution plan. The deduplication optimization pass includes: creating a list of query operators via a first traversal of the query operator tree, determining a first query operator that is equivalent to a second query operator, based on a hash map, via a second traversal of the query operator tree, and substituting, via a third traversal of the query operator tree, the second query operator with a tree node that links to the first query operator. The database engine executes the optimized execution plan to retrieve a result set from the database, and returns the result set.

A database system determines execution plans for database queries by evaluating a number of partial solutions for each database query. The database system determines a partial solutions limit on the number of partial solutions to be evaluated for determining the execution plan of the query. The database system determines a plurality of partial solutions, each partial solution corresponding to a portion of the execution plan for processing the database query. The database system evaluates a number of candidate partial solutions for determining a partial solution. The number of candidate partial solutions evaluated is determined based on the partial solutions limit. The database system combines the plurality of partial solutions to obtain an execution plan for the database query. The database system executes the database query by executing the execution plan.

Physical planning of database queries using partial solutions

A method enables users to resolve errors in a data flow. The method displays a user interface (UI) that includes a flow diagram having a plurality of nodes, receives user specification of a validation rule for a first node of the plurality of nodes in the flow diagram, and determines that an intermediate data set violates the validation rule. In response to determining that the first intermediate data set violates the validation rule, the method identifies errors corresponding to rows in the intermediate data set, and displays an error resolution UI that provides information about the errors. The error resolution UI includes a natural language summary region providing a synopsis of the errors, an error profile region graphically depicting the errors, a data flow trace region providing lineage of the errors in the flow, and a data region displaying data for a subset of columns from the intermediate data set.

A database system rewrites database queries based on rewrite constraints specification. The rewrite constraints specification provides directives to the database system to generate temporary tables for storing intermediate results of database queries and rewrite the corresponding database queries in terms of the temporary tables. The subqueries for matching the rewrite constraints specification may be specified in various ways. For example, the rewrite constraints specification may identify the subqueries by using one or more aliases. The rewrite constraints specification may specify a list of tables such that all database queries that join the database tables specified in the rewrite constraints specification are determined to match. The optimizer is able to optimize the rewritten database queries better by getting a better estimate for the statistics describing the temporary tables.

Rewrite constraints for database queries

A database system allows users to specify plan constraint specification that limit choices of execution plans considered by an optimizer of the database system for optimizing database queries. The plan constraint specification specifies attributes of execution plans. The plan constraint specification may identify tables/intermediate results that the plan constraint is applicable to using table names or aliases. The database system applies the plan constraint specification to database queries that match the tables/aliases specified in the plan constraint specification. The database system limits the choice of execution plans evaluated for a database query to execution plans satisfying the attributes specified in an applicable plan constraint specification. The plan constraint specification may specify distribution of a table, a join order for tables, the join mechanism, cardinality of a table or intermediate result, and so on.

Limiting plan choice for database queries using plan constraints

A database system identifies missing statistics that is not available for processing database queries and determines the missing statistics. The database system generates execution plans for database queries. The database system requests certain statistical information for generating a database query. If the database system determines that the requested statistical information is not available, the database system stores information describing the requested statistical information as missing statistics. The missing statistics may identify one or more columns associated with tables processed by the database query. The database system performs statistical analysis of database tables to generate the missing statistics so that the statistical information is available for generating execution plans for subsequent queries. The database system may rank the missing statistics based on the number of times the missing statistics was identified by the database system while processing database queries.

Estimating statistics for generating execution plans for database queries

A system and/or method are implemented to process queries to a database. In particular, the processing of queries to the database is enhanced by enhancing the determination of join orders of tables implicated in queries. Join orders between relatively large numbers of tables are determined by dividing the set of tables to be ordered into a plurality of subsets of tables, and ordering the individual subsets of tables.

System and method for processing database queries

A database system optimizes database queries that comprise subqueries. The database system factors out computation from two or more subqueries to a reusable subquery. The reusable subquery returns a result set. The subqueries are rewritten to use the result set and the database query modified to use the rewritten subqueries. The modified database query performs efficiently since the result set is computed only once for each subquery that uses the reusable query. The database system also composes nested subqueries that compute aggregate values. The database system identifies an inner subquery nested within an outer subquery, each subquery computing an aggregate value. The database system determines a new aggregate function corresponding to a composition of aggregate functions of the inner and the outer queries. The database system determines a composed query using the inner query with the outer query. The composed query uses the new aggregate function.

Optimizing database queries using subquery composition

Optimizing database queries using reusable subqueries

Growing community of inventors

Los Gatos, CA, United States of America

Richard Lee Cole