Microsoft Technology Licensing, LLC

Keebo Inc.

Alekh Jindal

Hiren Shantilal Patel

Shi Qiao

Carlo Aldo Curino

Marc Todd Friedman

Rathijit Sen

Malay Kumar Bag

Abhishek Roy

Avrilia Floratou

Zhicheng Yin

Markus Weimer

Irene Rogan Shaffer

Karthik Saligrama Ramachandra

Jyoti Leeka

Brandon Barry Haynes

Gilbert Antonius

Lucas Rosenblatt

Peter Orenberg

Soundararajan Srinivasan

Kameswara Venkatesh Emani

Barzan Mozafari

Yongjoo Park

Brian Westphal

Vijay Kumar Ramani

Remmelt Herbert Lieve Ammerlaan

H M Sajjad Hossain

Clemens Alden Szyperski

Yuanyuan Tian

Advait Dixit

Sriram S Rao

Subramaniam Venkatraman Krishnan

Tarique Ashraf Siddiqui

Konstantinos Karanasos

Yiwen Zhu

Saeed Amizadeh

Anish Yatin Pimpley

Jieming Di

Sunny Gakhar

Anna Pavlenko

Anubha Srivastava

Vishal Lalchand Rohra

Yi Zhu

Shuo Li

Matthew Larsen

Rana Bijad M Alotaibi

Alan D Cabrera

David Wolfgang Grömling

Chenggang Wu

Examples detect equivalent subexpressions within a computational workload. Examples include converting a query plan tree associated with a first subexpression into a matrix. The first subexpression is a portion of a database query from the computational workload. Each node in the query plan tree is represented as a row of the matrix. The matrix is converted into a first vector. The first subexpression is determined to be equivalent to a second subexpression by comparing the first vector to a second vector associated with the second subexpression. The comparison includes computing a distance between the first and second vectors that is lower than a distance threshold. The computational workload is modified, based on the determining, to perform the first subexpression and exclude performance of the second subexpression as duplicative.

Machine learning accelerated semantic equivalence detection

Solutions for optimizing job runtimes via prediction-based token allocation includes receiving training data comprising historical run data, the historical run data comprising job characteristics, runtime results, and a token count for each of a plurality of prior jobs, and the job characteristics comprising an intermediate representation and job graph data; based at least on the training data, training a token estimator, the token estimator comprising a machine learning (ML) model; receiving job characteristics for a user-submitted job; based at least on the received job characteristics, generating, with the token estimator, token prediction data for the user-submitted job; selecting a token count for the user-submitted job, based at least on the token prediction data; identifying the selected token count to an execution environment; and executing, with the execution environment, the user-submitted job in accordance with the selected token count.

Optimizing job runtimes via prediction-based token allocation

Systems, methods, and devices are described for performing scalable data processing operations. A queue that includes a translatable portion comprising indications of data processing operations translatable to data queries and a non-translatable portion comprising indications of non-translatable data processing operations is maintained. A determination that a first data processing operation of a first code block statement is translatable to a database query is made. An indication of the first data processing operation is included in the translatable portion of the queue. Responsive to a determination that a second data processing operation of a second code block statement is undeferrable, the translatable portion of the queue is compiled into a database query. An execution of the database query to be executed by a database engine to generate a query result is caused. A result dataset corresponding to the query result is transmitted to an application configured to analyze the result dataset.

System and method for scalable data processing operations

Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

System and method for machine learning for system deployments without performance regressions

The cloud-based query workload optimization system disclosed herein the cloud-based query workloads optimization system receives query logs from various query engines to a cloud data service, extracts various query entities from the query logs, parses query entities to generate a set of common workload features, generates intermediate representations of the query workloads, wherein the intermediate representations are agnostic to the language of the plurality of the queries, identifies a plurality of workload patterns based on the intermediate representations of the query workloads, categorizes the workloads in one or more workload type categories based on the workload patterns and the workload features, and selects an optimization scheme based on the category of workload pattern.

Cloud based query workload optimization

A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Resource optimization for serverless query processing

In a set of data analytics queries, at least a one of the queries comprising more than one operator, and each query being at least one of i) a producer of data for an other query in the set, and ii) a consumer of data from an other query in the set. In such examples, one or more computing devices identify each producer/consumer relationship between the queries. The one or more computing devices identify one or more optimizations among the queries based on the identified relationships. The one or more computing devices then apply at least one identified optimization to at least one of the queries.

Query set optimization in a data analytics pipeline

Implementations described herein relate to systems and methods to configure a data warehouse system. In some implementations, a method includes obtaining, by a configuration management system, historical query workload metadata associated with a data warehouse from the data warehouse system, determining, a first configuration setting associated with a configurable parameter for a first time period, wherein the first configuration setting is associated with a computing resource utilization at the data warehouse system different from a previous configuration setting, transmitting, to the data warehouse system, the first configuration setting for the configurable parameter, receiving, from the data warehouse system, during the first time period, query workload metadata, determining, whether the query workload metadata meets a threshold performance, and based on a determination that the query workload metadata does not meet the threshold performance, transmitting a backoff configuration setting for the configurable parameter to the data warehouse system.

Managed tuning for data clouds

Implementations described herein relate to systems and methods to provide platform agnostic query acceleration. In some implementations, a method includes receiving, at a processor associated with a query acceleration service, a request from an client/application, wherein the request conforms to a particular wire protocol of a plurality of supported wire protocols, and wherein the request includes header data and body content data, analyzing the request to identify at least one of a query and a command in the body content data, determining an optimal matched model of the one or more query acceleration models, rewriting the query based on the optimal matched model, transmitting the rewritten query to the query processing platform, receiving a response to the rewritten query or the query from the query processing platform, and transmitting the received response to the application, wherein the transmission is configured based on the particular wire protocol.

Platform agnostic query acceleration

Techniques are described herein that are capable of selecting checkpoints of a database job. For instance, at compile time, temporal indicators associated with the query plans of the database job are determined. Each temporal indicator indicates first and second subsets of stages of the respective query plan. Values of attributes of each stage in at least each first subset are predicted using a machine learning technique. At the compile time, candidate stage(s) for each query plan are identified based on the respective candidate stage being a child of stage(s) in the corresponding second subset or not being a child of another stage in the respective query plan. The candidate stage(s) for each query plan are selectively chosen as respective checkpoint(s) based on whether the values of the attributes of each stage in at least the first subset of the stages of the respective query plan satisfy one or more criteria.

Data-driven checkpoint selector

Described herein is a system and method for training cardinality models in which workload data is analyzed to extract and compute features of subgraphs of queries. Using a machine learning algorithm, the cardinality models are trained based on the features and actual runtime statistics included in the workload data. The trained cardinality models are stored. Further described herein is a system and method of predicting cardinality of subgraphs of a query. Features for the subgraphs of the query are extracted and computed. Cardinality models are retrieved based on the features of the subgraphs of the query. Cardinalities of the subgraphs of the query are predicted using the retrieved cardinality models. One of the subgraphs of the query is selected to be utilized for execution of the query based on the predicted cardinalities.

Learning optimizer for shared cloud

Described herein is a system and method for detecting and reusing overlapping computations. Overlapping subgraphs of the query are determined using a normalized signature for a particular subgraph that identifies a particular subgraph across recurring instances of data. A normalized signature for each overlapping subgraph for the determined overlapping subgraphs of the query is provided. For each overlapping subgraph determined to be materialized: whether or not the particular subgraph has been materialized is determined using a precise signature corresponding to a normalized signature of the particular overlapping subgraph. The precise signature identifies a particular subgraph corresponding to the normalized signature within a particular recurring instance of data. When the particular subgraph has not been materialized, the subgraph is materialized and used to respond to the query. When the particular subgraph has been materialized, the materialized subgraph is used to respond to the query.

Computation reuse in analytics job service

Growing community of inventors

Kirkland, WA, United States of America

Alekh Jindal