Microsoft Technology Licensing, LLC

Hiren Shantilal Patel

Alekh Jindal

Shi Qiao

Marc Todd Friedman

Rathijit Sen

Malay Kumar Bag

Carlo Aldo Curino

Abhishek Roy

Zhicheng Yin

Wangchao Le

Irene Rogan Shaffer

Markus Weimer

Gilbert Antonius

Lucas Rosenblatt

Remmelt Herbert Lieve Ammerlaan

Peter Orenberg

Clemens Alden Szyperski

Jingren Zhou

Sriram S Rao

Subramaniam Venkatraman Krishnan

Konstantinos Karanasos

Fei Xu

Tarique Ashraf Siddiqui

Vijay Kumar Ramani

H M Sajjad Hossain

Soundararajan Srinivasan

Yiwen Zhu

Edward John Triou, Jr

Saeed Amizadeh

Brandon Barry Haynes

Jyoti Leeka

Anish Yatin Pimpley

Anubha Srivastava

Vishal Lalchand Rohra

Yi Zhu

Shuo Li

Chenggang Wu

Jieming Di

Sunny Gakhar

Solutions for optimizing job runtimes via prediction-based token allocation includes receiving training data comprising historical run data, the historical run data comprising job characteristics, runtime results, and a token count for each of a plurality of prior jobs, and the job characteristics comprising an intermediate representation and job graph data; based at least on the training data, training a token estimator, the token estimator comprising a machine learning (ML) model; receiving job characteristics for a user-submitted job; based at least on the received job characteristics, generating, with the token estimator, token prediction data for the user-submitted job; selecting a token count for the user-submitted job, based at least on the token prediction data; identifying the selected token count to an execution environment; and executing, with the execution environment, the user-submitted job in accordance with the selected token count.

Optimizing job runtimes via prediction-based token allocation

Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

System and method for machine learning for system deployments without performance regressions

Example aspects include techniques for clustering delete targets for vectorized deletion including retrieving, from a set of delete targets in a distributed database system, a file to be deleted, scanning existing clusters of files marked for deletion to identify at least one existing cluster of files having constraints corresponding to the file, based on identifying the at least one existing cluster of files, adding the file to the at least one existing cluster of files to create a new cluster of files, based on failing to identify the at least one existing cluster of files, creating the new cluster of files including the file, and generating, for each file in the new cluster of files and based on a deletion signal, a delta array including multiple bits representing data items in each file and indicating, based on bit value, target data items to be deleted from each file.

Methods and systems for performing a vectorized delete in a distributed database system

The cloud-based query workload optimization system disclosed herein the cloud-based query workloads optimization system receives query logs from various query engines to a cloud data service, extracts various query entities from the query logs, parses query entities to generate a set of common workload features, generates intermediate representations of the query workloads, wherein the intermediate representations are agnostic to the language of the plurality of the queries, identifies a plurality of workload patterns based on the intermediate representations of the query workloads, categorizes the workloads in one or more workload type categories based on the workload patterns and the workload features, and selects an optimization scheme based on the category of workload pattern.

Cloud based query workload optimization

A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Resource optimization for serverless query processing

In a set of data analytics queries, at least a one of the queries comprising more than one operator, and each query being at least one of i) a producer of data for an other query in the set, and ii) a consumer of data from an other query in the set. In such examples, one or more computing devices identify each producer/consumer relationship between the queries. The one or more computing devices identify one or more optimizations among the queries based on the identified relationships. The one or more computing devices then apply at least one identified optimization to at least one of the queries.

Query set optimization in a data analytics pipeline

Example aspects include techniques for performing vectorized delete operations in a distributed database system including clustering multiple files stored in the distributed database system, and generating, for each of the multiple files and based on a deletion signal, a delta array including multiple bits representing the data items in the file and indicating, based on bit value, the target data items to be deleted from the file. Generating, for each of the multiple files, the delta array can include reading at least one second file shard of multiple second file shards before performing a join operation on at least one first file shard of multiple first file shards is completed.

Techniques are described herein that are capable of selecting checkpoints of a database job. For instance, at compile time, temporal indicators associated with the query plans of the database job are determined. Each temporal indicator indicates first and second subsets of stages of the respective query plan. Values of attributes of each stage in at least each first subset are predicted using a machine learning technique. At the compile time, candidate stage(s) for each query plan are identified based on the respective candidate stage being a child of stage(s) in the corresponding second subset or not being a child of another stage in the respective query plan. The candidate stage(s) for each query plan are selectively chosen as respective checkpoint(s) based on whether the values of the attributes of each stage in at least the first subset of the stages of the respective query plan satisfy one or more criteria.

Data-driven checkpoint selector

Described herein is a system and method for training cardinality models in which workload data is analyzed to extract and compute features of subgraphs of queries. Using a machine learning algorithm, the cardinality models are trained based on the features and actual runtime statistics included in the workload data. The trained cardinality models are stored. Further described herein is a system and method of predicting cardinality of subgraphs of a query. Features for the subgraphs of the query are extracted and computed. Cardinality models are retrieved based on the features of the subgraphs of the query. Cardinalities of the subgraphs of the query are predicted using the retrieved cardinality models. One of the subgraphs of the query is selected to be utilized for execution of the query based on the predicted cardinalities.

Learning optimizer for shared cloud

Described herein is a system and method for detecting and reusing overlapping computations. Overlapping subgraphs of the query are determined using a normalized signature for a particular subgraph that identifies a particular subgraph across recurring instances of data. A normalized signature for each overlapping subgraph for the determined overlapping subgraphs of the query is provided. For each overlapping subgraph determined to be materialized: whether or not the particular subgraph has been materialized is determined using a precise signature corresponding to a normalized signature of the particular overlapping subgraph. The precise signature identifies a particular subgraph corresponding to the normalized signature within a particular recurring instance of data. When the particular subgraph has not been materialized, the subgraph is materialized and used to respond to the query. When the particular subgraph has been materialized, the materialized subgraph is used to respond to the query.

Computation reuse in analytics job service

Described herein is a system and method for selecting subexpressions to be materialized. For a predefined storage budget, subexpressions of a set of candidate subexpressions to be materialized to minimize query evaluation cost are selected based upon a calculated utility of the set of candidate subexpressions, interactions of the candidate subexpressions, and, a cost of evaluating the candidate subexpressions. Based upon the subexpressions selected to be materialized, subexpression(s) of the set of candidate subexpressions to use when evaluating particular queries of the set of queries to minimize query evaluation cost are determined.

Selection of subexpressions to materialize for datacenter scale

Embodiments of the present invention allow multiple data streams to be analyzed as a single data set. The single data set may be described as a stream set herein. The multiple streams that are included in the stream set may be specified through a user script or query. For example, a query may be used to gather all streams created within a date range. The query could include one or more filters to gather certain information from the data streams or to exclude certain data streams that otherwise are in the query's range. A stream may be an unstructured byte stream of data. The stream may be created by append-only writing to the end of the stream. The stream could also be a structured stream that includes metadata that defines column structure and affinity/clustering information.

Analyzing multiple data streams as a single data object

Growing community of inventors

Bothell, WA, United States of America

Hiren Shantilal Patel