Microsoft Technology Licensing, LLC

Raghunath Ramakrishnan

Baskar Sridharan

Ramarathnam R Venkatesan

Sriram Kottarakurichi Rajamani

Mitica Manu

Sri Sai Kameswara Pavan Kumar Kasturi

Victor V Boyko

Pushkar Vijay Chitnis

Sumeet Priyadarshee Dash

Shastry M C Shankara

Peng T Li

Yuan Yu

Marc Todd Friedman

Carlo Aldo Curino

Haibo Lin

Rathijit Sen

Sriram S Rao

Karthick Krishnamoorthy

Subramaniam Venkatraman Krishnan

Kwanghyun Park

Konstantinos Karanasos

Jaliya Nishantha Ekanayake

Christopher William Douglas

Zhicheng Yin

Matteo Interlandi

Virajith Jalaparti

Markus Weimer

Tao Guan

Brandon Barry Haynes

José A Blakeley

Jin Sun

Yongchul Kwon

Fotios Psallidas

Ming Li

Krishna Pradeep Reddy Tamma

Spiro Michaylov

Chakrapani Bhat Talapady

Aditya Telidevara

Joseph M Gagne

Ivan Popivanov

Doris Suiyi Xin

Karla Jean Saur

Milan Potocnik

Mohammad Shafiei Khadem

Kevin Bocksrocker

A hypergraph workload manager in a server is configured for failure tolerant and explainable state machine driven hypergraph execution. The hypergraph executor comprises a query optimizer, a hypergraph enlister, a pipeline analyzer, and a state machine generator. The query optimizer translates a user query into a query operator graph. The hypergraph enlister enlists the query operator graph into a hypergraph containing a set of query operator graphs representative of already submitted user queries. The enlistment is configured to join query operator graphs where it makes sense to optimize query executions. Updates to the hypergraph based on the enlistment results in a set of disconnected graphs. The pipeline analyzer performs an analysis of all operators of all queries in the hypergraph to find an optimal sequencing of execution. The state machine generator is configured to generate a hierarchical state machine for all operators of a disconnected graph of the hypergraph.

Failure tolerant and explainable state machine driven hypergraph execution

Systems and methods are provided that introduce an approach for executing a multi-query workload that leverages live execution feedback from nodes to detect resourcing issues and anomalies, and deploy real-time corrective measures for the multi-query workload. Leveraging live execution feedback from the nodes as the queries are executing make it possible to detect various resourcing issues and anomalies, and enable the system to perform corrective actions 'live' or in “real-time” during an execution of a query, and more specifically during execution of the tasks within a query.

Dynamic workload management optimizations using real-time execution feedback

The description relates to executing an inference query relative to a database management system, such as a relational database management system. In one example a trained machine learning model can be stored within the database management system. An inference query can be received that applies the trained machine learning model on data local to the database management system. Analysis can be performed on the inference query and the trained machine learning model to generate a unified intermediate representation of the inference query and the trained model. Cross optimization can be performed on the unified intermediate representation. Based upon the cross-optimization, a first portion of the unified intermediate representation to be executed by a database engine of the database management system can be determined, and, a second portion of the unified intermediate representation to be executed by a machine learning runtime can be determined.

Query processing with machine learning

A system that maps files into multiple parts and initiates storage of each part within a respective storage provider. The mapping facilitates this by correlating each of the multiple parts to a corresponding storage provider which may be different than the storage provider that stores a different part of the same file. The system then initiates storage of the multiple parts of the file in each of their corresponding storage provider in accordance with the mapping, and using a storage-provider-facing API. Similarly, retrieval and/or editing of a partial file may be accomplished by referencing the mapping, and using the same API issued to the appropriate storage provider. Each storage provider is capable of interfacing with the system using this same API.

Mapping storage across storage providers

Operating upon encrypted data with a particular data scope. A base encryption key is established and associated with the particular data scope, and then stored in a base encryption key store. That base encryption key store might be managed by an application or service that stores base encryption keys for multiple data scopes. A proxy encryption key acts as a kind of proxy for the base encryption key. The proxy encryption key may be used for frequent operations on encrypted data within the particular data scope. Thus, the principles described herein act as a frequency amplifier that allows key-based operations upon the particular data scope to be performed at much higher frequencies than otherwise would be possible by operating directly using the base encryption key.

Data operations using a proxy encryption key

Sharding a data unit across storage locations (e.g., data centers, clusters, accounts, and so forth). For each location, the multi-link traffic constraints in data channeling with the storage location are identified. For instance, for a given data channel leading to and/or from a storage location, there may be multiple links. Each link may have its own constraints (e.g., in throughput and/or storage). Policy may consider the sharding of files in a manner that honors the constraints. The policy may also be set so that all portions of the shards may be timely accessed in parallel. When sharding, the policy is applied in order to identify initial placement of the computing data unit across the multiple storage locations. Furthermore, the policy may be consulted to determine which storage location to access the shard from and/or which channel to use to access the shard from a given location.

Sharding over multi-link data channels

Aspects extend to methods, systems, and computer program products for optimally pipelining result sets with fault tolerance in distributed query execution. Distributed computing jobs are optimized by dividing the distributed computing jobs into one or more bubbles for execution. Each bubble can be independently executed, potentially in parallel with other bubbles, when resources to handle the bubble are available. Intra-bubble communication can be streamed between vertices within a bubble. Inter-bubble communication can be stored to durable storage. Bubbles provide a failure boundary for a job graph and re-executing a bubble along with storage of intermediate results in durable storage can be used to recover from failures. When a vertex inside a bubble fails, computation can resume by rescheduling the execution of the failed bubble from the durable inputs for that bubble. Durable storage provides a light-weight failover to handle non-deterministic behavior. Jobs can also leverage streaming to increase performance.

Optimizing pipelining result sets with fault tolerance in distributed query execution

Growing community of inventors

Bellevue, WA, United States of America

Raghunath Ramakrishnan