Microsoft Technology Licensing, LLC

Wangchao Le

Marc Todd Friedman

Hiren Shantilal Patel

Yongchul Kwon

Evgueni Zabokritski

Xiangnan Li

A click-to-script service enables developers of big-data job scripts to quickly see the underlying script operations from optimized execution plans. Once a big-data job is received, the disclosed examples compile it and generate tokens that are associated with each operation of the big-data job. These tokens include may include the file name of the job, the line number of the operation, and/or an Abstract Syntax Tree (AST) node for the given operations. An original execution plan is optimized into an optimized execution plan, and the tokens for the original operations of the job script are assigned to the optimized operations of the optimized execution plan. The optimized execution plan is graphically displayed in an interactive manner such that users may view the optimized execution plan and click on its optimized operations to find the original operations of the job script.

Click-to-script reflection

Example aspects include techniques for clustering delete targets for vectorized deletion including retrieving, from a set of delete targets in a distributed database system, a file to be deleted, scanning existing clusters of files marked for deletion to identify at least one existing cluster of files having constraints corresponding to the file, based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files, based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file, and generating, for each file in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.

Methods and systems for performing a vectorized delete in a distributed database system

Example aspects include techniques for performing vectorized delete operations in a distributed database system including clustering multiple files stored in the distributed database system, and generating, for each of the multiple files and based on a deletion signal, a delta array including multiple bits representing the data items in the file and indicating, based on bit value, the target data items to be deleted from the file. Generating, for each of the multiple files, the delta array can include reading at least one second file shard of multiple second file shards before performing a join operation on at least one first file shard of multiple first file shards is completed.

Constructing a heavy hitter summary for query optimization. The heavy hitter summary is constructed by sampling each of multiple partitions of a dataset using a uniformed sampling rate. For each partition, performing a two-stage heavy hitter estimation process to determine whether an estimated frequency of a key of the sampled data units may be included in a partition-level heavy hitter summary. Constructing a partition-level heavy hitter summary for each partition of the dataset based on the keys determined via the two-stage process, and constructing a dataset-level heavy hitter summary based on the partition-level heavy hitter summary. The dataset-level heavy hitter summary may be used to optimize query trees.

Building heavy hitter summary for query optimization

Query optimization using of a query that is compiled into a query tree. The optimization is efficiently performed by using a distinct value estimation data structure (e.g., a KMV synopsis) that represents within an interval distinctness of values that are generated based on data within an interval, even if the resultant data from a subinterval is considered. Various candidate query trees are evaluated, with distinct value generation data structures being propagated for parent nodes based on the distinct value generation data structures of its child node(s). Propagation operations correlate to the operation represented by the parent node in the query tree. The optimizer uses the propagated distinct value estimation structure in order to evaluate the number of distinct values of data that would result from execution of the candidate query tree at least at the corresponding operations (and not just based on the distinct values of the input data).

Growing community of inventors

Redmond, WA, United States of America

Wangchao Le