Microsoft Technology Licensing, LLC

Carlo Aldo Curino

Rathijit Sen

Subramaniam Venkatraman Krishnan

Alekh Jindal

Konstantinos Karanasos

Hiren Shantilal Patel

Sriram S Rao

Shi Qiao

Christopher William Douglas

Matteo Interlandi

Marius Dumitru

Avrilia Floratou

Mostafa Elhemali

Kishore Chaliparambil

Karthik Saligrama Ramachandra

Malay Kumar Bag

Kameswara Venkatesh Emani

Kannababu Karanam

Hans C Lehnert Merino

Chuan Liu

Jitendra Dattatraya Padhye

Yuan Yu

Yuanyuan Tian

George Varghese

Raghunath Ramakrishnan

Tarique Ashraf Siddiqui

Kwanghyun Park

Zhicheng Yin

Yiwen Zhu

Markus Weimer

Jyoti Leeka

Abhishek Roy

Brian Paul Kroth

Jesus Camacho Rodriguez

Fotios Psallidas

Andreas Christian Mueller

Sudhir Darbha

Isha Tarte

Ivan Popivanov

Doris Suiyi Xin

Karla Jean Saur

Rana Alotaibi

Hanxian Huang

Krystian Sakowski

Zezhou Huang

Ashish Vulimiri

Joana Matos Fonseca Da Trindade

Andrew F Chung

A method, computer program product, and computing system for optimizing query operations on run length encoding (RLE) data in a parallel processing computing system. Data is received in a plurality of columns of an input table of a parallel processing computing system for query execution; the system determines that at least a portion of the received data in a first number of columns is compressed according to run length encoding (RLE), thereby comprising RLE data columns including RLE data and that the received data in a second number of columns is not compressed according to run length encoding (RLE), thereby comprising non-RLE data columns including non-RLE data. A query operation is executed on the RLE data and the non-RLE data by prioritizing processing of the RLE data columns over processing of the non-RLE data columns.

System and method for accelerating query execution

A method, computer program product, and computing system for processing query operations on run length encoding (RLE), data in a parallel processing computing system. Data for query execution is received at a parallel processing computing system, at least a portion of the data being compressed according to RLE, thereby forming RLE data; and a query operation is executed on the RLE data without performing a decompression operation on the RLE data.

System and method for performing query operations on run length encoded data

The description relates to executing an inference query relative to a database management system, such as a relational database management system. In one example a trained machine learning model can be stored within the database management system. An inference query can be received that applies the trained machine learning model on data local to the database management system. Analysis can be performed on the inference query and the trained machine learning model to generate a unified intermediate representation of the inference query and the trained model. Cross optimization can be performed on the unified intermediate representation. Based upon the cross-optimization, a first portion of the unified intermediate representation to be executed by a database engine of the database management system can be determined, and, a second portion of the unified intermediate representation to be executed by a machine learning runtime can be determined.

Query processing with machine learning

Systems and methods for characterizing and forecasting evolving query workloads. The method includes receiving a query, the received query including a parameter value and an arrival time; identifying the query as a recurrent query; extracting a query template from the received query by parsing the received query; based at least on the identifying, generating a feature vector for the received query, the feature vector generated based on the extracted template and the parameter value; and forecasting a future query based on the generated feature vector by applying a neural network.

Characterizing and forecasting evolving query workloads

Systems, methods, and devices are described for performing scalable data processing operations. A queue that includes a translatable portion comprising indications of data processing operations translatable to data queries and a non-translatable portion comprising indications of non-translatable data processing operations is maintained. A determination that a first data processing operation of a first code block statement is translatable to a database query is made. An indication of the first data processing operation is included in the translatable portion of the queue. Responsive to a determination that a second data processing operation of a second code block statement is undeferrable, the translatable portion of the queue is compiled into a database query. An execution of the database query to be executed by a database engine to generate a query result is caused. A result dataset corresponding to the query result is transmitted to an application configured to analyze the result dataset.

System and method for scalable data processing operations

The cloud-based query workload optimization system disclosed herein the cloud-based query workloads optimization system receives query logs from various query engines to a cloud data service, extracts various query entities from the query logs, parses query entities to generate a set of common workload features, generates intermediate representations of the query workloads, wherein the intermediate representations are agnostic to the language of the plurality of the queries, identifies a plurality of workload patterns based on the intermediate representations of the query workloads, categorizes the workloads in one or more workload type categories based on the workload patterns and the workload features, and selects an optimization scheme based on the category of workload pattern.

Cloud based query workload optimization

A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Resource optimization for serverless query processing

An automated tuning service is used to automatically tune, or modify, the operational parameters of a large-scale cloud infrastructure. The tuning service performs automated and fully data/model-driven configuration based from learning various real-time performance of the cloud infrastructure. Such performance is identified through monitoring various telemetric data of the cloud infrastructure. The tuning service leverages a mix of domain knowledge and principled data-science to capture the essence of our cluster dynamic behavior in a collection of descriptive machine learning (ML) models. The ML models power automated optimization procedures for parameter tuning, and inform administrators in most tactical and strategical engineering/capacity decisions (such as hardware and datacenter design, software investments, etc.). Rich 'observational' models (models collected without modifying the system) are combined with judicious use of “fighting” (testing in production), allowing the tuning service to automatically configure operational parameters of a large cloud infrastructure for a broad range of applications.

Tuning large data infrastructures

Performance degradation of an application that is caused by another computing process that shares infrastructure with the application is detected. The application and the other computing device may execute via different virtual machines hosted on the same computing device. To detect the performance degradation that is attributable to the other computing process, certain storage segments of a data storage (e.g., a cache) shared by the virtual machines is written with data. A pattern of read operations are then performed on the segments to determine whether an increase in read access time has occurred. Such a performance degradation is attributable to another computing process. After detecting the degradation, a metric that quantifies the detected degradation attributable to the other computing process is provided to an ML model, which determines the actual performance of the application absent the degradation attributable to the other computing process.

Performance evaluation of an application based on detecting degradation caused by other computing processes

Various technologies described herein pertain to controlling geo-scale analytics with bandwidth and regulatory constraints. An analytical query (e.g., a recurrent analytical query, a non-recurrent analytical query, etc.) to be executed over distributed data in data partitions stored in a plurality of data centers can be received. Moreover, a query execution plan for the analytical query can be generated, where the query execution plan includes tasks. Further, replication strategies for the data partitions can be determined. A replication strategy for a particular data partition can specify one or more data centers to which the particular data partition is to be replicated if the particular data partition is to be replicated. The tasks of the query execution plan for the analytical query can further be scheduled to the data centers based on the replication strategies for the data partitions. The analytical query can be part of a workload of analytical queries.

Geo-scale analytics with bandwidth and regulatory constraints

Methods, systems, and computer program products are provided for generating and utilizing materialized graph views. A system according to one embodiment includes a graph database including a graph and schema, a workload analyzer, a view enumerator, a query rewriter and an execution engine. The workload analyzer is configured to receive and analyze queries in a query workload. The view enumerator is configured to use an inference engine to operate on facts derived from the graph and a query, and view templates comprising inference rules to enumerate candidate views. The workload analyzer is further configured to selects a candidate view to materialize, provide the selected view to the execution engine that is configured to generate the materialized view. The workload analyzer may select the at least one candidate view based on factors such as query evaluation cost estimates, candidate view performance improvement estimates, view size estimates and view creation cost estimates.

Materialized graph views for efficient graph analysis

Described herein is a system and method for ranking and/or taking an action regarding execution of jobs of a shared computing cluster based upon predicted user impact. Information regarding previous executions of a plurality of jobs is obtained, for example, from job execution log(s). Data dependencies of the plurality of jobs are determined. Job impact of each of the plurality of jobs as a function of the determined data dependencies is calculated. User impact of each of the plurality of jobs as a function of the determined data dependencies, the calculated job impact, and time is calculated. The plurality of jobs are ranked in accordance with the calculated user impact. An action is taken in accordance with the ranking of the plurality of jobs. The action can include automatic scheduling of the jobs and/or providing information regarding the rankings to a user.

Provenance driven job relevance assessment

Embodiments are directed to progressively migrating source computer nodes where the source computer nodes perform a computer-implemented service. In one embodiment, a computer system determines that execution of the performed service is to be migrated from the source computer nodes to target computer nodes. The computer system groups the source computer nodes into multiple source subgroups, where each source subgroup includes at least one source computer node. The computer system then schedules creation of target subgroups of target nodes. These target subgroups include at least one source computer node and, themselves, correspond to a source subgroup. The computer system activates a first target subgroup corresponding to a first source subgroup, and deactivates the first source subgroup. In this manner, the first target subgroup replaces the first source subgroup. Still further, the target subgroups are scheduled to be created only after the first source subgroup has been deactivated.

Seamless cluster servicing

Growing community of inventors

Woodinville, WA, United States of America

Carlo Aldo Curino