Snowflake Inc.

Google Inc.

Daniel Mills

Daniel E Sotolongo

Tyler Jones

Tyler Arthur Akidau

Istvan Cseri

Leon Papke

Subramanian Muralidhar

Dinesh Chandrakant Kulkarni

Craig D Chambers

Ganeshan Ramachandran Iyer

Robert Bradshaw

Reuven Lax

Jiaxing Liang

Abdullah Al Mahmood

Xin Huang

Di Fei Zhang

Prasanna Rajaperumal

Frances J Perry

Fabian Hueske

Ben Chambers

Manuel Alfred Fahndrich

Mark Brian Shields

Ahmet Altay

Samuel Green McVeety

Andrei Maksimenka

Harris Samuel Nover

Rafael De Jesús Fernández Moctezuma

Mikhail Shmulyan

Samuel Carl Whittle

Cosmin Ionel Arad

Jeffrey Paul Gardner

Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Error tables for use with recovery actions

Provided herein are systems and methods for a stream object configuration, including query processing of stream objects using stream expansion. For example, a method includes decoding a query to obtain a first data processing operation and a first stream object. The first stream object is associated with a view on a base table. A first stream expansion on the first stream object is performed. The first stream expansion is based on generating a second stream object on the base table. A second stream expansion of the second stream object is performed. The second stream expansion is based on replacing the second stream object with at least a second data processing operation. The query is executed based on completing the first data processing operation and the at least a second data processing operation.

Query processing of stream objects using stream expansion

Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Cross-organization and cross-cloud automated data pipelines

A system for a materialized table (MT) refresh using multiple processing pipelines includes at least one hardware processor coupled to memory storing instructions. The instructions cause the at least one hardware processor to perform operations including determining dependencies among a plurality of intermediate MTs generated from a source MT. The source MT uses a table definition with a query on one or more base tables and a lag duration value. A graph snapshot of dependencies among the plurality of intermediate MTs is generated. Processing pipelines are configured. Each of the processing pipelines corresponds to a subset of the plurality of intermediate MTs indicated by the graph snapshot. Responsive to detecting an instruction for a refresh operation on the source MT, refreshes on corresponding intermediate MTs of the plurality of intermediate MTs in each processing pipeline of the processing pipelines are performed to complete the refresh operation on the source MT.

Materialized table refresh using multiple processing pipelines

A method includes retrieving a plurality of materialized tables (MTs). Each of the plurality of MTs includes a lag duration and refers to a corresponding base table of a plurality of base tables. The lag duration indicates a maximum time period that a result of a prior refresh of a query on the corresponding base table can lag behind a current time instance. A plurality of time instances for the MT is determined based on the lag duration and a number of prior refreshes of the corresponding base table. A plurality of aligned time instances for the plurality of MTs is determined based on the plurality of time instances for each of the plurality of MTs. Refresh operations are scheduled for the plurality of MTs at one or more of the plurality of aligned time instances that are within the maximum time period.

Query execution using materialized tables

Error tables to track errors associated with a base table

Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views and streams on tables. For example, a method includes parsing, by at least one hardware processor, a received query to determine a plurality of data processing operations associated with a base table. A stream object on the base table is generated. The stream object includes a plurality of query results from executing the plurality of data processing operations and a corresponding plurality of time stamps of times the plurality of query results are obtained. A stream expansion of the stream object is performed. The stream expansion is based on computing changes to the base table from the plurality of data processing operations.

Stream object type for query processing

Provided herein are systems and methods for a database object (e.g., materialized table) configuration including using the database object for incremental maintenance of query results. For example, a method includes creating a materialized table using a table definition. The table definition includes a query on one or more base tables and a lag duration value. The lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance. A delta set for the one or more base tables is determined. The delta set includes at least one change applied to the one or more base tables since the prior refresh of the query. The delta set is merged with the materialized table to perform an incremental update of the materialized table.

Incremental maintenance of query results

Provided herein are systems and methods for a database object (e.g., materialized table) configuration including scheduling refreshes of the materialized table. For example, a method includes determining a dependency graph for a first MT. The dependency graph comprises a second MT from which the first MT depends. The first MT includes a query on one or more base tables and a lag duration value. The lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance. A tick period is selected for a set of ticks based on the lag duration value. The set of ticks corresponds to a set of aligned time instances. Refresh operations are scheduled for the first and second MTs at corresponding time instances from the set of aligned time instances. The corresponding time instances are separated by the tick period.

Execution and consistency model for materialized tables

Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views. For example, a method includes configuring by at least one hardware processor at a first computing node, a view on a source table for shared access with at least a second computing node. The view includes a query associated with the source table. The query associated with the source table is executed to obtain a plurality of query results. A stream object is generated on the view. The stream object includes the plurality of query results of executing the query and a corresponding plurality of time stamps associated with times the plurality of query results are obtained. The generated stream object is configured for the shared access with at least the second computing node.

Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views. For example, a method includes detecting a first stream object on a view. The view includes a query associated with a source table. A syntax tree of the query is determined based on a definition of the view. The view is expanded based on replacing the first stream object with the syntax tree. The syntax tree includes a second stream object on the source table. Stream expansion of the second stream object is performed based on computing changes on the source table.

Database object type for querying and transactional consumption of changes in queries results

A method includes receiving data corresponding one of streaming data or batch data and a content of the received data for computation. The method also includes determining an event time of the data for slicing the data, determining a processing time to output results of the received data, and emitting at least a portion of the results of the received data based on the processing time and the event time.

Data flow windowing and triggering

Growing community of inventors

Seattle, WA, United States of America

Daniel Mills