Microsoft Technology Licensing, LLC

Hewlett-packard Company

Other

Surajit J Chaudhuri

Vivek Ravindranath Narasayya

Kris Kuppuswamy Ganjam

Kaushik Chakrabarti

Yeye He

Sanjay Agrawal

Venkatesh Ganti

Nicolas Bruno

Rajeev Motwani

Zhimin Chen

James P Finnigan

Kanstantsyn Zoryn

Ravishankar Ramamurthy

Chi Wang

Mayur D Datar

Wentao Wu

Tarique Ashraf Siddiqui

Eric J Horvitz

Lev Novik

Gary William Flake

Joshua T Goodman

Oliver Hurst-Hiller

Sudipto Das

Yue Wang

Henricus Johannes Maria Meijer

John Carlton Platt

Bolin Ding

Kaushik Shriraghav

Tao Cheng

William Henry Gates, Iii

Arvind Arasu

Trenholme J Griffin

Feng Li

Kenneth Alan Moss

Shriraghav Kaushik

Luis Gravano

Xu Chu

Bailu Ding

Senjuti Basu Roy

Anshuman Dutt

Ashraf I Aboulnaga

Anoop Gupta

Christopher A Meek

Eric David Brill

Efim Hudis

Stephen L Lawler

Kunal Mukerjee

Badrish Chandramouli

Christopher Alan Hays

Ramez Naam

Ashish Gupta

Usama M Fayyad

Manoj A Syamala

Yaron Y Goland

Arnd Christian Konig

Dong Xin

Lengning Liu

Kyuseok Shim

Zhongyuan Wang

Vishal Kumar Seshagirirao Anil

Gautam Das

Lucas Pritz

Aditya Kuppa

Meiyalagan Balasubramanian

Jie Song

Patrick Clinton Little

Kirk Hartmann Freiheit

Rohit Ananthakrishna

Gaurav Malhotra

Gantam Das

Kalen Wong

Guatam Das

Silu Huang

Gaurav Ghosh

Paula Budig Greve

Umeshwar Dayal

Lili Cheng

Lin Ma

Atul Adya

Rahul Kapoor

Marius Dumitru

Srikanth Kandula

Prasanna Ganesan

Cezary Antoni Marcjan

Shobana Balakrishnan

Ming-Chuan Wu

Raghav Kaushik

Ryszard K Kott

Murali M Krishna

Gerhard Weikum

Cong Yan

Ravi Ramamurthy

Prasana Ganesan

Zhiyuan Chen

Robert Grandl

Er Kang Zhu

Mohamed A Yakout

Anish Das Sarma

Abhinandan Sujit Das

Nilesh N Dalvi

This disclosure describes a workload reduction system that reduces the complexity of workloads sent to a database system. For instance, the workload reduction system pre-processes workloads sent to a database system by generating reduced workloads that include less complex queries that reference fewer tables or columns than the original workloads. In various implementations, the workload reduction system uses table reduction functions and query re-writing functions to generate the reduced workloads. As a result, the workload reduction system improves computational efficiency by rewriting complex queries from workloads into simpler ones that speed up index tuning and decrease individual what-if call times.

Using workload reduction to improve index tuning

This document relates to budget-constrained index tuning in database systems. A method for index tuning within a database system includes receiving a budget constraint, generating a set of candidate indexes for a workload of queries, and determining a derived cost for each query-configuration pair (QCP). The method further involves identifying a subset of QCPs where the derived cost and a what-if cost have a delta value below a predefined difference threshold and allocating the budget constraint to these identified QCPs to avoid what-if calls. Additionally, the method reallocates a saved portion of the budget constraint to other QCPs where the delta value exceeds the difference threshold, evaluates these other QCPs using what-if calls to select an optimal index configuration, and applies the selected index configuration to the database system.

Budget-constrained index tuning in database systems

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for facilitating data transformations, according to embodiments of the present invention. In one embodiment, a transformation function is executed using an example input value to obtain an initial output value. Thereafter, a plurality of supplemental transformation tools is applied to the initial output value to generate a plurality of intermediary output values. Based on a comparison of each of the intermediary output values to an example output value, the supplemental transformation tool that generated an intermediary output value having a greatest extent of similarity to the example output values is identified. The identified supplemental transformation tool and the transformation function are used to generate a transformation program that transforms the example input values to the desired form in which to transform data.

Efficient transformation program generation

A data processing system implements a cache-conscious aggregation framework for cache-efficient top-k aggregation over high cardinality large datasets. The framework leverages skew in the distribution of data in the datasets to minimize data movements within the local caches of the cores of the multicore processors of the data processing system. The framework performs representative sampling on the dataset and utilizes these samples to identify candidate groups in the dataset for the top-k results. The system performs exact aggregations for the candidate groups and performs hashing and pruning on the non-candidate groups in the dataset to identify top-k results included in the non-candidate groups without having to calculate the exact aggregations for the non-candidate groups.

Cache-efficient top-k aggregation over high cardinality large datasets

Systems and methods relate to auto-tagging of data in a data lake or a data storage. Generating a statistical summary of the data lake and interactively receiving data in a selected column of an exemplar data addresses an issue of efficiently and accurately auto-tagging data in a data lake. The present disclosure automatically generates a statistical summary of the data lake using a lightweight off-line processing. A graphical user interface interactively receives an exemplar data file with a selection of a column in the exemplar data file. A list of candidate data-tagging patterns is generated based on the statistical summary and updates the list by removing candidate data-tagging patterns that under-generalize the data. The present disclosure determines a data-tagging pattern by selecting a candidate data-tagging profile from the list based on having the least number of matching columns in the data lake.

Method and system for automatically tagging data

This document relates to relational databases and corresponding data tables. Non-conforming data tables can be automatically transformed into conforming relational data tables. One example can obtain conforming relational data tables and can generate training data without human labelling by identifying a transformational operator that will transform an individual conforming relational data table to a non-conforming data table and an inverse transformational operator that will transform the non-conforming data table back to the individual conforming relational data table. The example can train a model with the training data. The trained model can synthesize programs to transform other non-conforming data tables to conforming relational data tables.

Synthesizing transformations to relationalize data tables

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for facilitating data transformations, according to embodiments of the present invention. In one embodiment, a set of example values including example input values that indicate data values to be transformed and example output values that indicate a desired form in which to transform data. Based on the set of example values, a data transformation function that is relevant to the set of example values is identified. The data transformation function is used to generate a transformation program to transform the example input values to the desired form in which to transform data. A suggestion of the transformation program can be provided to a user device, wherein selection of the transformation program suggestion results in a data transformation.

Facilitating data transformations

Solutions for data unification include: receiving a data record, the data record comprising a plurality of data fields; selecting, from among the plurality of data fields, a subset of the data fields, the subset of the data fields being fewer in number than the plurality of data fields, wherein selecting the subset of the data fields comprises: applying a first rule to select at least a first one of the data fields within the data record for inclusion in the subset of the data fields; using content of the subset of the data fields, generating a stable identifier (stableID) for the data record; and inserting the stableID into a primary key data field of the data record.

Data unification

A method of training an index filter for an index tuning system includes receiving a plurality of different workloads and a plurality of different databases, each database including different tables and each workload including a plurality of queries; generating labeled training by making optimizer calls to a query optimizer using query and index configuration pairs from the plurality of databases and the plurality of workloads; training an index filter model to identify signals in the labeled training data, the signals being indicative of a potential performance improvement associated with using an index configuration for a given query; training the index filter model to learn rules over the signals for identifying spurious indexes; and storing the index filter model in a memory.

Scalable index tuning with index filtering and index cost models

In automated machine learning, an approximate best configuration can be selected among multiple candidate machine-learning configurations by progressively sampling training and test datasets for the iterative training and testing of the configurations while progressively pruning the set of candidate configurations based on associated estimated confidence intervals for their respective performance.

Efficient configuration selection for automated machine learning

A system and method for executing SQL statements includes receiving an SQL statement for comparing two trendsets over a relation using a scoring function, each of the trendsets including one or more trends, each of the trends being designated by a constraint and a grouping-measure combination, wherein comparing the trendsets includes identifying trend pairs for comparison, each of the trend pairs including a trend from the each of the trendsets having a common grouping-measure combination. The SQL statement is transformed into a basic plan of existing logical operators for performing the SQL statement. A set of sub-plans is determined based on the basic plan. Pairs of sub-plans are merged to generate a set of merged sub-plans. A cost for each of the merged sub-plans is determined. The merged sub-plan having the lowest cost is used to execute the SQL statement.

Systems and methods for accelerating and optimizing groupwise comparison in relational databases

The present disclosure relates to methods and systems for compressing workloads for use with index tuning. The methods and systems receive a workload with a plurality of queries. The methods and systems represent each query using query features and a utility. The methods and systems select a query for a query subset based on a benefit of the query determined using the query features and the utility. The methods and systems update the features and the utility of the remaining queries in the workload and select another query to add to the query subset based on an updated benefit determined using the updated features and utilities. The methods and systems select queries for the query subset equal to a received query subset size. The methods and systems use the query subset in index tuning to provide one or more indexes to recommendations.

Compressing workloads for scalable index tuning

The present disclosure relates to systems, methods, and computer-readable media for determining optimal index configurations for processing workloads in a database management system. For instance, an index configuration system can efficiently determine a subset of indexes for processing a workload utilizing one or more reinforcement learning models. For example, in various implementations, the index configuration system utilizes a Markov decision process and/or a Monte Carlo tree search model to determine an optimal subset of indexes for processing a workload in a manner that effectively utilizes computing device resources while also avoiding significant interference with customer workloads.

Constraint-based index tuning in database management systems utilizing reinforcement learning

The present disclosure relates to systems, methods, and computer-readable media for optimizing selection of a cached execution plan to use in processing a parametric query. For example, systems described herein involve training a plan selection model that makes use of machine learning to identify an execution plan from a set of pre-selected execution plans based on predicted cost of executing a query instance in accordance with the selected execution plan (e.g., relative to predicted costs of executing the query instance using other pre-selected execution plans). This application describes features related to lowering costs associated with selecting the execution plan in a way that will continue to be more accurate overtime based on training and refining the plan selection model.

Using query logs to optimize execution of parametric queries

A transform-by-pattern (TBP) system is configured to proactively suggest relevant TBP programs based on inputted source dataset and target dataset without requiring users typing in examples. The TBP system has access to multiple TBP programs, each of which includes a combination of a source pattern, a target pattern, and a transformation program that is configured to transform data that fits into the target pattern into data that fits into the source pattern. When a source dataset and a target dataset are received from a user, the TBP system identifies a subset of the source dataset and a subset of the target dataset as related data. The TBP system then identifies one or more applicable TBP programs amongst the multiple TBP programs, and suggest or apply at least one of the one or more applicable TBP programs.

Automatic transformation of data by patterns

Growing community of inventors

Remond, WA, United States of America

Surajit J Chaudhuri