Snowflake Inc.

Tyler Jones

Istvan Cseri

Daniel E Sotolongo

Tyler Arthur Akidau

Daniel Mills

Dinesh Chandrakant Kulkarni

Zhuo Zhang

Isaac Kunen

Peter Povinec

Derek Denny-Brown

Fabian Hueske

Leon Papke

Benoit Dageville

Thierry Cruanes

Subramanian Muralidhar

Ganeshan Ramachandran Iyer

Jiaxing Liang

Abdullah Al Mahmood

Xin Huang

Di Fei Zhang

Igor A Zinkovsky

Prasanna Rajaperumal

Yevgeniy Kogan

Dzmitry Pauliukevich

Boyuan Zhang

Justin A Jones

Vincent Pangallo

Matthew Kaminecki

Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Error tables for use with recovery actions

A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Real-time streaming data ingestion into database tables

Provided herein are systems and methods for a stream object configuration, including query processing of stream objects using stream expansion. For example, a method includes decoding a query to obtain a first data processing operation and a first stream object. The first stream object is associated with a view on a base table. A first stream expansion on the first stream object is performed. The first stream expansion is based on generating a second stream object on the base table. A second stream expansion of the second stream object is performed. The second stream expansion is based on replacing the second stream object with at least a second data processing operation. The query is executed based on completing the first data processing operation and the at least a second data processing operation.

Query processing of stream objects using stream expansion

Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Cross-organization and cross-cloud automated data pipelines

Techniques described herein can enable stream replication. A first deployment can store a table including one or more streams. The techniques described herein can be used to replicate the table at a second deployment while replicating the one or more streams associated with the table. Select prior table versions and partitions in the table are copied to the second deployment to enable stream replication.

Selective table replication to enable stream replication

Techniques described herein can accelerate change data capture determinations such as stream reads, which show changes made to a table between two points in time. Three distinct row bitests that mark deleted, updated, inserted, rows in micro-partitions can be added as metadata for the table. These bitsets can be generated during DML operations and then stored as metadata of the new partition generated by the DML operations. The bitsets can then be used to generate streams showing the changes in the table between two points in time (changes interval).

Accelerating change data capture determination using row bitsets preliminary

A system for a materialized table (MT) refresh using multiple processing pipelines includes at least one hardware processor coupled to memory storing instructions. The instructions cause the at least one hardware processor to perform operations including determining dependencies among a plurality of intermediate MTs generated from a source MT. The source MT uses a table definition with a query on one or more base tables and a lag duration value. A graph snapshot of dependencies among the plurality of intermediate MTs is generated. Processing pipelines are configured. Each of the processing pipelines corresponds to a subset of the plurality of intermediate MTs indicated by the graph snapshot. Responsive to detecting an instruction for a refresh operation on the source MT, refreshes on corresponding intermediate MTs of the plurality of intermediate MTs in each processing pipeline of the processing pipelines are performed to complete the refresh operation on the source MT.

Materialized table refresh using multiple processing pipelines

A method includes retrieving a plurality of materialized tables (MTs). Each of the plurality of MTs includes a lag duration and refers to a corresponding base table of a plurality of base tables. The lag duration indicates a maximum time period that a result of a prior refresh of a query on the corresponding base table can lag behind a current time instance. A plurality of time instances for the MT is determined based on the lag duration and a number of prior refreshes of the corresponding base table. A plurality of aligned time instances for the plurality of MTs is determined based on the plurality of time instances for each of the plurality of MTs. Refresh operations are scheduled for the plurality of MTs at one or more of the plurality of aligned time instances that are within the maximum time period.

Query execution using materialized tables

Error tables to track errors associated with a base table

Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Optimized processing of data in different formats

A system or persistent table may be generated storing changelog information of a primary base table. The system table may then be used to create streams of relevant information. In some examples, the streams may read from the system table for information past a retention period of the primary table while reading from the primary table information in the retention period.

Streams using persistent tables

A credentials store definition identifying a remote credential store is received. The credential store definition includes access information to enable access to the remote credentials store. A credentials object is created in an internal database based on a credentials object definition. The credentials object identifies a security credential to retrieve from the remote credentials store to access an external resource. At runtime, a request to access the external resource is received, and based on receiving the request, the security credentials identified by the credentials object are retrieved from the remote credential store using the access information. The retrieved security credential is provided to a processing component to access the external resource.

Facilitating access to remotely stored credentials for accessing external resources

A firearm recoil mechanism is provided that includes a front spring plate integrally coupled with a Bolt Carrier Group. The front spring plate is slidably coupled with first and Second Guide Rods, where the first and Second Guide Rods extend through apertures in the front spring plate, first and second Recoil Springs engaging with the front spring plate, where the first Recoil Spring is disposed around the first Guide Rod, and where second Recoil Spring is disposed around the Second Guide Rod. A piston integrally coupled with the front spring plate and the Bolt Carrier Group. The piston and the front spring plate move with the Bolt Carrier Group in opposing directions while the first and Second Guide Rods remain stationary to coil and recoil the first Recoil Spring and second Recoil Spring on the first and Second Guide Rods, respectively.

Growing community of inventors

San Francisco, CA, United States of America

Tyler Jones