Microsoft Technology Licensing, LLC

Other

Emc Corporation

University of Washington

Dantz Development Corporation

Marc Todd Friedman

Hiren Shantilal Patel

Shi Qiao

Wangchao Le

Alekh Jindal

Lev Novik

Jason Duane Clark

Ajay Kalhan

Andrew Eyring Kimball

Joseph P Mullally

Joachim Hammer

Jared E Bobbitt

Patrick Wing Sang Lau

Yongchul Kwon

Cihangir Biyikoglu

Istvan Cseri

Clemens Alden Szyperski

Tomas Talius

Stephan Doll

Santeri Olavi Voutilainen

Jaliya Nishantha Ekanayake

Robert Chris Hickman

Tony Petrossian

Irene Rogan Shaffer

Markus Weimer

Jin Sun

Abhishek Roy

Gilbert Antonius

Lucas Rosenblatt

Jeffrey C Tanner

Vijay Kumar Ramani

Remmelt Herbert Lieve Ammerlaan

H M Sajjad Hossain

Peter Orenberg

Soundararajan Srinivasan

Cesar Alejandro Galindo-Legaria

Milind Madhukar Joshi

Naveen Prakash

Haibo Lin

Erik Ismert

Rathijit Sen

Raghunath Ramakrishnan

Zhicheng Yin

Evgueni Zabokritski

Tao Guan

Daniel S Weld

Jyoti Leeka

John Wesley Fawcett

José A Blakeley

Brandon Barry Haynes

Ming Li

Anish Yatin Pimpley

Tobias Joakim Bertil Ternström

Jiong (Justin) Chen

Edward P Johnson

Ty Lam

Robert J Smith

Chung T Kwok

Sunny Gakhar

Anubha Srivastava

Vishal Lalchand Rohra

Yi Zhu

Shuo Li

Xiangnan Li

Solutions for optimizing job runtimes via prediction-based token allocation includes receiving training data comprising historical run data, the historical run data comprising job characteristics, runtime results, and a token count for each of a plurality of prior jobs, and the job characteristics comprising an intermediate representation and job graph data; based at least on the training data, training a token estimator, the token estimator comprising a machine learning (ML) model; receiving job characteristics for a user-submitted job; based at least on the received job characteristics, generating, with the token estimator, token prediction data for the user-submitted job; selecting a token count for the user-submitted job, based at least on the token prediction data; identifying the selected token count to an execution environment; and executing, with the execution environment, the user-submitted job in accordance with the selected token count.

Optimizing job runtimes via prediction-based token allocation

A click-to-script service enables developers of big-data job scripts to quickly see the underlying script operations from optimized execution plans. Once a big-data job is received, the disclosed examples compile it and generate tokens that are associated with each operation of the big-data job. These tokens include may include the file name of the job, the line number of the operation, and/or an Abstract Syntax Tree (AST) node for the given operations. An original execution plan is optimized into an optimized execution plan, and the tokens for the original operations of the job script are assigned to the optimized operations of the optimized execution plan. The optimized execution plan is graphically displayed in an interactive manner such that users may view the optimized execution plan and click on its optimized operations to find the original operations of the job script.

Click-to-script reflection

Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

System and method for machine learning for system deployments without performance regressions

Example aspects include techniques for clustering delete targets for vectorized deletion including retrieving, from a set of delete targets in a distributed database system, a file to be deleted, scanning existing clusters of files marked for deletion to identify at least one existing cluster of files having constraints corresponding to the file, based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files, based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file, and generating, for each file in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.

Methods and systems for performing a vectorized delete in a distributed database system

In a set of data analytics queries, at least a one of the queries comprising more than one operator, and each query being at least one of i) a producer of data for an other query in the set, and ii) a consumer of data from an other query in the set. In such examples, one or more computing devices identify each producer/consumer relationship between the queries. The one or more computing devices identify one or more optimizations among the queries based on the identified relationships. The one or more computing devices then apply at least one identified optimization to at least one of the queries.

Query set optimization in a data analytics pipeline

Example aspects include techniques for performing vectorized delete operations in a distributed database system including clustering multiple files stored in the distributed database system, and generating, for each of the multiple files and based on a deletion signal, a delta array including multiple bits representing the data items in the file and indicating, based on bit value, the target data items to be deleted from the file. Generating, for each of the multiple files, the delta array can include reading at least one second file shard of multiple second file shards before performing a join operation on at least one first file shard of multiple first file shards is completed.

Shuffling of into partitions by first grouping input vertices of a limited number. Each group of input vertices may then be simply shuffled into a corresponding group of intermediate vertices, such as by broadcasting. A second grouping occurs in which the intermediate vertices are grouped by partition. The intermediate vertices then shuffle into corresponding output vertices for the respective partitions of that group. If the intermediate vertices are still too large, then this shuffling may involve recursively performing the shuffling just described, until ultimately the number of intermediate vertices shuffling into the output vertices is likewise limited. Thus, the final shuffling into the output vertices might also be simply performed by broadcasting.

Grouped shuffling of partition vertices

Constructing a heavy hitter summary for query optimization. The heavy hitter summary is constructed by sampling each of multiple partitions of a dataset using a uniformed sampling rate. For each partition, performing a two-stage heavy hitter estimation process to determine whether an estimated frequency of a key of the sampled data units may be included in a partition-level heavy hitter summary. Constructing a partition-level heavy hitter summary for each partition of the dataset based on the keys determined via the two-stage process, and constructing a dataset-level heavy hitter summary based on the partition-level heavy hitter summary. The dataset-level heavy hitter summary may be used to optimize query trees.

Building heavy hitter summary for query optimization

Query optimization using of a query that is compiled into a query tree. The optimization is efficiently performed by using a distinct value estimation data structure (e.g., a KMV synopsis) that represents within an interval distinctness of values that are generated based on data within an interval, even if the resultant data from a subinterval is considered. Various candidate query trees are evaluated, with distinct value generation data structures being propagated for parent nodes based on the distinct value generation data structures of its child node(s). Propagation operations correlate to the operation represented by the parent node in the query tree. The optimizer uses the propagated distinct value estimation structure in order to evaluate the number of distinct values of data that would result from execution of the candidate query tree at least at the corresponding operations (and not just based on the distinct values of the input data).

Query optimization using propagated data distinctness

Aspects extend to methods, systems, and computer program products for optimally pipelining result sets with fault tolerance in distributed query execution. Distributed computing jobs are optimized by dividing the distributed computing jobs into one or more bubbles for execution. Each bubble can be independently executed, potentially in parallel with other bubbles, when resources to handle the bubble are available. Intra-bubble communication can be streamed between vertices within a bubble. Inter-bubble communication can be stored to durable storage. Bubbles provide a failure boundary for a job graph and re-executing a bubble along with storage of intermediate results in durable storage can be used to recover from failures. When a vertex inside a bubble fails, computation can resume by rescheduling the execution of the failed bubble from the durable inputs for that bubble. Durable storage provides a light-weight failover to handle non-deterministic behavior. Jobs can also leverage streaming to increase performance.

Optimizing pipelining result sets with fault tolerance in distributed query execution

Methods, systems, and computer program products are provided for partitioning online databases. Online database operations, such as, for example, SPLIT, MERGE, and DROP, are used to alter the arrangement of partitions in a federated database. A SPLIT operation splits rows at one partition across a plurality of other partitions. A MERGE operation merges rows at a plurality of partitions in to one partition. A DROP operation shifts responsibility for rows of data from one partition to another partition and then drops the rows from the one partition.

Partitioning online databases

The present invention extends to methods, systems, and computer program products for scoping the context used to access a database partition. Embodiments of the invention enable data isolation using partitions in multi-tenant databases, while relieving client applications from dealing with the partitions. For example, a computer system that includes a distributed database system comprising a plurality of database partitions in a federation receives a context to use when performing database access operations within the distributed database system. The context identifies specified relevant portion of the federation. The computer system also receives a database access operation that is associated with the context. The computer system modifies the semantics of the database access operation in accordance with the associated context, to direct application of the database access operation to the specified relevant portion of the federation.

Scoped database connections

The present invention extends to methods, systems, and computer program products for partitioning online databases. Online database operations, such as, for example, SPLIT, MERGE, and DROP, are used to alter the arrangement of partitions in a federated database. A SPLIT operation splits rows at one partition across a plurality of other partitions. A MERGE operation merges rows at a plurality of partitions in to one partition. A DROP operation shifts responsibility for rows of data from one partition to another partition and then drops the rows from the one partition.

Systems and methods that generate specialized plans for compiling SQL queries. A plan generator component scans the query representation for parameter sensitive predicates and evaluates each predicate individually based on the parameter values. Accordingly, queries can be identified not only based on their structures, but also based on their parameter conditions. The specialized plans are more efficient for particular values, wherein queries that employ such values are optimally executed.

Parameter-sensitive plans for structural scenarios

Growing community of inventors

Seattle, WA, United States of America

Marc Todd Friedman