Microsoft Technology Licensing, LLC

Columbia University

At+t Corp.

At&t Intellectual Property I, L.p.

Nicolas Bruno

Surajit J Chaudhuri

Jingren Zhou

Divesh Srivastava

Rimma V Nehme

Nikolaos Koudas

Ravishankar Ramamurthy

Ming-Chuan Wu

Vivek Ravindranath Narasayya

Srikanth Kandula

Luis Gravano

Wei Lin

Murali M Krishna

Sameer Agarwal

Nikolacs Koudas

Dilys Thomas

Nilcolaos Koudas

Runtime statistics from the actual performance of operations on a set of data are collected and utilized to dynamically modify the execution plan for processing a set of data. The operations performed are modified to include statistics collection operations, the statistics being tailored to the specific operations being quantified. Optimization policy defines how often optimization is attempted and how much more efficient an execution plan should be to justify transitioning from the current one. Optimization is based on the collected runtime statistics but also takes into account already materialized intermediate data to gain further optimization by avoiding reprocessing.

Continuous cloud-scale query optimization and processing

A relational database server may concurrently execute many relational queries, but a complex relational query may cause performance delays in the fulfillment of other relational queries. Instead, the relational database server may generate a query plan for the relational query, and may endeavor to partition the relational query between a spool operator and a scan operator into two or more query slices, where each query slice may be executed within a query slice threshold. Many alternative candidate query plans may be considered, such as inserting spool and scan operators after various operators and parameterizing operators in order to partition the records of a relation into two or more ranges based on an attribute of the relation. A large search space of candidate query plans may be reviewed in order to select a query plan that respects the query slice threshold while efficiently executing the logic of the relational query.

Slicing relational queries using spool operators

The use of statistics collected during the parallel distributed execution of the tasks of a job may be used to optimize the performance of the task or similar recurring tasks. An execution plan for a job is initially generated, in which the execution plan includes tasks. Statistics regarding operations performed in the tasks are collected while the tasks are executed via parallel distributed execution. Another execution plan is then generated for another recurring job, in which the additional execution plan has at least one task in common with the execution plan for the job. The additional execution plan is subsequently optimized based at least on the statistics to produce an optimized execution plan.

Parallel computing execution plan optimization

A repartitioning optimizer identifies alternative repartitioning strategies and selects optimal ones, accounting for network transfer utilization and partition sizes in addition to traditional metrics. If prior partitioning was hash-based, the repartitioning optimizer can determine whether a hash-based repartitioning can result in not every computing device providing data to every other computing device. If prior partitioning was range-based, the repartitioning optimizer can determine whether a range-based repartitioning can generate similarly sized output partitions while aligning input and output partition boundaries, increasing the number of computing devices that do not provide data to every other computing device. Individual computing devices, as they are performing a repartitioning, assign a repartitioning index to each individual data element, which represents the computing device to which such a data element is destined. The indexed data is sorted by such repartitioning indices, thereby grouping together all like data, and then stored in a sequential manner.

Efficient partitioning techniques for massively distributed computation

Described is automatically processing an initial database repro (text representing a bug when corresponding script is executed in a database engine) into a min-repro (a subset of the text) that is simplified version of the initial repro yet still contains the bug. A parse tree representative of the initial database repro is processed into simplified parse trees based on language grammar rules, e.g., by replacing higher level nodes with descendant nodes. Repros of the simplified parse trees are executed to determine which simplified repros still fail execution because of the bug (that is, the simplified repros were not oversimplified). A minimum simplified parse tree with respect to a desired level of minimality is found from among those failing repros, with the simplified repro that corresponds to the minimum simplified parse tree output as the min-repro.

Minimizing database repros using language grammars

The min-repro finding technique described herein is designed to ease and speed-up the task of finding a min-repro, a minimum configuration that reproduces a problem in database-related products. Specifically, in one embodiment, the technique simplifies transformations in order to find one or more min-repros. One embodiment provides a high-level script language to automate some sub-tasks and to guide the search for a simpler the configuration that reproduces the problem. Yet another embodiment provides record-and-replay functionality, and provides an intuitive representation of results and the search space. These tools can save hours of time for both customers and testers to isolate the problem and can result in faster fixes and large cost savings to organizations.

Min-repro framework for database systems

Embodiments are directed to determining optimal partition configurations for distributed database data and to implementing parallel query optimization memo data structure to improve partition configuration cost estimation efficiency. In an embodiment, a computer system accesses a portion of database data and various database queries for a given database. The computer system determines, based on the accessed database data and database queries, a partition configuration search space which includes multiple feasible partition configurations for the database data and a workload of queries expected to be executed on that data. The computer system performs a branch and bound search in the partition configuration search space to determine which data partitioning path has the lowest partitioning cost. The branch and bound search is performed according to branch and bound search policies. The computer system also outputs the partition configuration with the determined lowest partitioning cost.

Automated partitioning in parallel database systems

An architecture for providing interactive sessions for physical database design is described, allowing users to readily try different options, identify problems, and obtain physical designs in a flexible way. Embodiments based on a .NET assembly and modifications to a database management system (DBMS) are also described.

Interactive physical design tuning

A flexible query hints system and method for discovering and expressing query hints in a database management system. Embodiments of the flexible query hints system and method include a power hints (Phints) language that enables the specification of constraints to influence a query optimizer. Phints expressions are defined as tree patterns annotated with constraints. Embodiments of the flexible query hints system and method also include techniques to incorporate the power hints language expressions into an extended query optimizer. Theses techniques include computing a directed acyclic graph for Phints expression, deriving candidate matches using the Phints expression and the graph, computing candidate matches, and extracting a revised execution plan having a lowest cost and satisfying constraints of the Phints expression. Embodiments of the flexible query hints system and method include a flexible query hint user interface that allow users to interactively adjust query hints.

Flexible query hints in a relational database

A lightweight physical design alerter can analyze a workload and determine whether a comprehensive tuning session would result in a configuration improvement over the current configuration. The alerter provides a low-overhead procedure that can run during normal operation of a database management system and produce a notification if a current configuration is less than optimal. The alerter can report lower and upper bounds on the improvements that could be obtained if a comprehensive tuning tool is launched. A lower bound can be justified by generating feasible configurations. The disclosed embodiments can be extended to query updates, materialized views, and other physical design features (e.g., partitioning).

Lightweight physical design alerter

Described is a constraint language and related technology by which complex constraints may be used in selecting configurations for use in physical database design tuning. The complex constraint (or constraints) is processed, e.g., in a search framework, to determine and output at least one configuration that meets the constraint, e.g., a best configuration found before a stopping condition is met. The search framework processes a current configuration into candidate configurations, including by searching for candidate configurations from a current configuration based upon a complex constraint, iteratively evaluating a search space until a stopping condition is satisfied, using transformation rules to generate new candidate configurations, and selecting a best candidate configuration. Transformation rules and pruning rules are applied to efficiently perform the search. Constraints may be specified as assertions that need to be satisfied, or as soft assertions that come close to satisfying the constraint.

Constrained physical design tuning

A method of query pattern matching uses a chain of linked stacks to compactly represent partial results to root-to-leaf query paths, which are then composed to obtain matches for the twig pattern.

Growing community of inventors

Redmond, WA, United States of America

Nicolas Bruno