Microsoft Technology Licensing, LLC

Yinan Li

Badrish Chandramouli

Jonathan David Goldstein

Donald Kossmann

Nikolaos Romanos Katsipoulakis

Kapil Laxmikant Surlaker

Lin Qiao

Min Tu

Shirshanka Das

Chavdar Botev

Issac Buenrostro

Narasimha R Veeramreddy

Jianan Lu

Ying Dai

Sahil Takiar

Kenneth D Goodhope

Ziyang Liu

Systems and methods of processing queries for column-oriented database systems involve processing the queries to identify which of values of column data are to be accessed for the query. A select bitmap is then generated having k bits wherein each bit corresponds to one of the values of the column. The select bitmap is generated such that each bit representing a value of the column data that is to be accessed for the query has a first value and each bit representing a value of the column data that is not to be accessed for the query has a second value. Encoded values are then extracted from memory for each of the values represented in the select bitmap by a bit having the first value. The extracted encoded values are decoded to generate decoded query data. The decoded query data is then processed to generate result data.

Selection pushdown in column stores using bit manipulation instructions

A method includes obtaining a query containing at least one field from which data is being queried, obtaining a dataset having a schema-free data exchange format having multiple fields of data at different physical positions in the dataset, and parsing the dataset by obtaining a structural index that maps logical locations of fields to physical locations of the fields of the dataset, accessing the structural index with logical locations of the fields that index to the physical locations, and providing data from the fields based on the physical locations responsive to the query.

Parser for schema-free data exchange format

The techniques and systems described herein are directed to incremental sorting in a stream processing engine. For example, because of network delays, streaming data can be received at a stream processing engine that is out-of-order with respect to the timestamp of the event generation. Streaming data can be incrementally sorted by partitioning events into ordered runs, which provide intermediate arrays of ordered elements. Upon an assertion of a punctuation event (e.g., a progress indicator), partial runs can be output and merged to form sorted streaming data. Further, the sorted streaming data can be used to provide speculative outputs based on speculation latencies, which improves throughput and completeness of processing on the streaming data.

Efficient sorting for a stream processing engine

In order to ingest data from an arbitrary source in a set of sources, a computer system accesses predefined configuration instructions. Then, the computer system generates a dynamic data-ingestion pipeline that is compatible with a Hadoop file system based on the predefined configuration instructions. This dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data. Moreover, the computer system receives the data from the source. Next, the computer system processes the data using the dynamic data-ingestion pipeline as the data is received without storing the data in memory for the purpose of subsequent ingestion processing.

Dynamic data-ingestion pipeline

Growing community of inventors

Redmond, WA, United States of America

Yinan Li