Microsoft Technology Licensing, LLC

Snowflake Inc.

Microsoft Corporation

Dinesh Chandrakant Kulkarni

Anders Hejlsberg

Matthew J Warren

Henricus Johannes Maria Meijer

Istvan Cseri

Luca Bolognese

Peter A Hallam

Tyler Jones

Gary Shon Katzenberger

Tyler Arthur Akidau

Mads Torgersen

Erik Meijer

Di Fei Zhang

Benoit Dageville

Thierry Cruanes

Daniel E Sotolongo

Daniel Mills

Donald F Box

Igor A Zinkovsky

Jomo Fisher

John Joseph Duffy

Christopher Wellington Brumme

Vance Palmer Morrison

Clemens Alden Szyperski

Lucas James Hoban

John Wesley Dyer

Terry A Adams

Daigo Hamura

Evgueni Zabokritski

Ion Vasilian

Mihir Dhananjay Kulkarni

Lanhao Wu

Peter A c/o Microsoft Corporation International Patents Hallam

Henricus Johannes Maria c/o Microsoft Corporation International Patents Meijer

In one aspect, a computer-implemented method includes detecting a committed version of recurrently executed tasks of a first data pipeline on a primary deployment that is hosted on a first cloud service, and replicating the committed version of the recurrently executed tasks to a second data pipeline on a secondary deployment that is hosted on a second cloud service.

Cross-cloud replication of recurrently executing data pipelines

Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Cross-organization and cross-cloud automated data pipelines

Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Optimized processing of data in different formats

A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Streams on shared database objects

The subject technology determines, using a stream object provided by a network-based database system separate from an external system, a set of offsets of each shard of a set of shards, the set of offsets being determined by executing a user defined function (UDF) included in the stream object, the stream object and the UDF being implemented in a particular programming language different than a database query language. The subject technology identifies an operation to perform on the set of shards. The subject technology, based on the set of shards and the set of offsets, performs, using the UDF included in the stream object, the operation on an external data source hosted by the external system. The subject technology increments a property indicating a different version of the stream object after performing the operation on the external data source, the property being included in the stream object.

Extensible streams for operations on external systems

The subject technology determines a set of shards corresponding to an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system. The subject technology determines, using a stream object, a set of offsets of each shard of the set of shards. The subject technology identifies an operation to perform on the set of shards, the operation comprising a read operation or a write operation. The subject technology, based on the set of shards and the set of offsets, performs the operation on the external data source.

Extensible streams on data sources

An operation (such as a relational query) may be processed on a processing engine (such as a relational database server) on behalf of a client. A conventional processing involves the delivery of the operation to the processing engine, which executes the entire operation to completion and returns a result data set. It may be more efficient to allocate part of the operation to be performed on the client, but a developer may be unable or unavailable to rewrite the operation in a distributed manner. Instead, the operation may be automatically partitioned into a pre-engine client portion, a processing engine portion, and a client portion, and the instructions of each portion may be automatically allocated respectively to the client, the server, and the client. The partitioning may be adjusted to conserve computing resources, such as bandwidth and storage, and the instructions may be reordered to improve the processing of the operation.

Automated client/server operation partitioning

Data source queries may be specified in various languages, such as SQL, XQuery, and object-oriented languages, and each language may have various linguistic features that enable different types of queries. It may be advantageous to formulate mixed language queries having portions specified in a first language and portions specified in a second language, and to provide mixed language query evaluation components that can parse portions of queries of different languages. It may also be advantageous to permit the translation of single language queries into multiple languages to be parsed by such mixed language query evaluation components. Such techniques may permit queries to utilize linguistic features and processing techniques of one language that are less efficient, less direct, or simply unavailable in another language.

Data query translating into mixed language data queries

A compiler supporting a language in which selected semantic objects are represented as data objects. The data objects may be used in multiple ways to expand the capabilities of the programming language. Data objects may be passed to applications and used to create executable instructions for that application. In this way, instructions written in the native language of the compiler may be used to control applications that accept programs in a language inconsistent with the native language of the compiler. The syntax checking and variable binding capabilities of the compiler may be used for those instructions that will be executed by an application separate from the object code generated by the compiler. The semantic objects represented as data objects may be selected based on express operations included in the source code or may be based on implicit type conversion.

Compiler supporting programs as data objects

Queries against data sources (such as language-integrated queries to be applied against relational databases) may be prepared for processing by a query processing pipeline. This pipeline performs services including transforming the query into a form more easily applied by the data source, e.g., into SQL, by matching query terms with query operators configured to handle various query operations. Many query processing pipelines include single-item query operators that perform individual operations (e.g., an updating query operator that specifies an updating of individual records of a table.) An integrated query pipeline may be devised that includes both single-item query operators and multiple-item query operators that specify operations on sets of data items. An integrated query processing pipeline may analyze the query to determine whether a single-item or multiple-item operation is specified in order to select an appropriate query operator, which may improve the efficiency and performance of the data source querying.

Query processing pipelines with single-item and multiple-item query operators

Growing community of inventors

Sammamish, WA, United States of America

Dinesh Chandrakant Kulkarni