Microsoft Technology Licensing, LLC

Peter Zabback

Jonathan David Goldstein

Badrish Chandramouli

Jingren Zhou

Wei Lin

Zhengping Qian

Ian Jose

Campbell Bryce Fraser

David Maier

Di Wang

Mariyan Dimitrov Fransazov

Beysim Sezgin

Tihomir Tsvetkov Tarnavski

Aleksandras Surna

Romit Chattopadhyay

Tomer Verona

Sreenivas Gukal

A low-latency cloud-scale computation environment includes a query language, optimization, scheduling, fault tolerance and fault recovery. An event model can be used to extend a declarative query language so that temporal analysis of event of an event stream can be performed. Extractors and outputters can be used to define and implement functions that extend the capabilities of the event-based query language. A script written in the extended query language can be translated into an optimal parallel continuous execution plan. Execution of the plan can be orchestrated by a streaming job manager which schedules vertices on available computing machines. The streaming job manager can monitor overall job execution. Fault tolerance can be provided by tracking execution progress and data dependencies in each vertex. In the event of a failure, another instance of the failed vertex can be scheduled. An optimal recovery point can be determined based on checkpoints and data dependencies.

Distributed stream processing in the cloud

According to examples, an apparatus may include a processor and a memory on which is stored machine readable instructions that are to cause the processor to receive, from a first node, a client request and a response corresponding to the processed client request, in which the first node is configured to generate the response by processing the client request on a service and processed client request is configured to change a state of the service. The instructions are also to cause the processor to receive, from the first node, a response corresponding to the received request, generate an entry including the received request and response, commit the generated entry into a journal, and send an acknowledgement of the received request and response to the first node after the generated entry is committed into the journal.

Service state preservation across nodes

Distributed stream processing in the Cloud

A logical merge module is described herein for producing an output stream which is logically compatible with two or more physically divergent input streams. Representative applications of the logical merge module are also set forth herein.

Efficient logical merging over physically divergent streams

A checkpoint marker can be received at a first operator. The first operator can process the checkpoint marker by sending the checkpoint marker to a second operator and sending state checkpoint information representing a state of the first operator to a checkpoint writer. The checkpoint information can be used to rehydrate the state of one or more operators. For example, after a system failure, system shutdown, etc., checkpoint information can be received from a reader unit at a checkpoint information input queue of the first operator. A state of the first operator can be rehydrated using the checkpoint information. Processing of information in a data input queue of the first operator can be suspended while the checkpoint information is used to rehydrate the state of the first operator. Other operators in a system with the first operator (e.g., the second operator) may be checkpointed and rehydrated in the same manner as the first operator.

Operator state checkpoint markers and rehydration

Described herein are technologies pertaining to migrating state information of operators in a first continuous query plan to a second continuous query plan in an asynchronous manner, such that the first continuous query plan need not cease executing during the migrating of the state information. State information pertaining to stateful operators, such as join operators, is migrated from the first continuous query plan to the second continuous query plan by way of a transformation plan. State matching is utilized to generate the transformation plan.

Asynchronous transfer of state information between continuous query plans

Histograms formed based on samples of a population, such as histograms created from random page-level samples of a data store, are intelligently scaled to histograms estimating distribution of the entire population of the data store. As an optional optimization, where a threshold number of duplicate samples are observed during page-level sampling, the number of distinct values in the overall population data is presumed to be the number of distinct values in the sample data. Also, during estimation of distinct values of an overall population, a 'Chao' estimator can optionally be utilized as a lower bound of the estimate. The resulting estimate is then used when scaling, which can take domain knowledge of the data being scaled into account in order to prevent scaled estimates from exceeding the limits of the domain. Also, a 'sum of the parts' mathematical relationship can be taken into account during scaling that the sum of the scaled distinct values for each bin of an estimate histogram should total an estimate for the total distinct values of the entire population.

Generating histograms of population data by scaling from sample data

Methods, systems, and computer-readable media are disclosed for partitioned query execution in event processing systems. A particular method includes receiving a plurality of events via an input stream. The plurality of events is partitioned into one or more groups, and a query application module is instantiated for each of the one or more groups based on a compiled query application plan. Each particular query application module for a particular group is configured to apply a query to events of the particular group to generate partial results. The method includes merging the partial results of each of the query application modules to generate merged output results and providing the output results to an output stream.

Partitioned query execution in event processing systems

Systems and methods that eliminate non-qualifying data for queries against data warehouses and improve execution time, via a dynamic filter component(s). In general, such dynamic filter components are derived from data during processing of the query and without being explicitly defined by the users within a query forwarded to the data warehouse. Moreover, an evaluation component can monitor efficiency of filter components (e.g., number of rows that can be eliminated), and dynamically change and/or update the evaluation order of such filters.

Dynamic filters for relational query processing

A query execution system is provided. The system includes a monitor component that detects data value changes in a database. An adjustment component initiates an intermediate query in view of detected data value changes, the intermediate query employed to adjust statistics related to a query plan optimization.

Growing community of inventors

Kirkland, WA, United States of America

Peter Zabback