Snowflake Inc.

Ganeshan Ramachandran Iyer

Subramanian Muralidhar

Benoit Dageville

Jiaxing Liang

Abdullah Al Mahmood

Raghav Ramachandran

Xin Huang

Thierry Cruanes

Istvan Cseri

Jiaqi Yan

Nithin Mahesh

Daniel E Sotolongo

Unmesh Jagtap

Daniel Mills

Yucan Liu

Isaac Kunen

Saurin Shah

Tyler Jones

Ruta Dhaneshwar

Purav B Saraiya

Yanyi Zhang

Daria Wieteska

Jan Tomasz Topinski

Pawel Pollak

Mikolaj Niedbala

Tomasz Malisz

Michal Gdak

David Schultz

Tyler Arthur Akidau

Raja Suresh Krishna Balakrishnan

Chong Han

Jian Xu

Yanrui Zhang

Yang Wang

Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Error tables for use with recovery actions

Techniques for deduplicating files to be ingested by a database are disclosed. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein each of the set of bloom filters is built with a number of hash functions that is based on a desired false positive rate. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested, identifying using the set of bloom filters, candidate files from the set of candidate files that are duplicative of a file in the first set of files and pruning from the set of candidate files, each candidate file identified as being duplicative of a file in the first set of files using the set of bloom filters.

Efficient deduplication of randomized file paths

Techniques for continuous ingestion of files using custom file formats are described. A custom file format may include formats not natively supported by a data system. Unstructured files (e.g., images) may also be considered custom file formats. A custom file format may be set using a user defined table function and scanner options.

Continuous ingestion of custom file formats

Provided herein are systems and methods for configuring integrity constraints (including a check constraint) and row violation logging using error tables. An example method includes decoding a query received at a network-based database system. The query includes a command to perform an operation on a base table. An integrity constraint associated with the base table is retrieved. The integrity constraint specifies a desired configuration for the base table. A verification of the integrity constraint is performed to detect erroneous data of the base table that violates the desired configuration. The erroneous data is input into an error table that is configured as a nested object of the base table. A notification that the erroneous data is available in the error table is generated and output.

Configuring check constraint and row violation logging using error tables

Systems and methods for generating a machine-learning (ML) model for extracting information from one or more electronic documents, where the ML model can be used as a data object, which can be part of a database command or as part of a document information extraction process that is continuously running (e.g., document information extraction pipeline).

Generating machine-learning model for document extraction

Techniques for configuring event tables include retrieving, by at least one hardware processor of a computing node, log data at a first process of the computing node. The log data is associated with a function executing at a second process of the computing node. The log data is revised using a table stage to generate revised log data. The table stage is configured as a dedicated storage location of the first process. The revising includes a data enrichment process based on metadata associated with execution of the function at the second process. The revised log data is ingested into an event table.

Configuring an event table using computing node processes

Error tables to track errors associated with a base table

A system and method of dynamic task allocation and warehouse scaling. The method includes receiving a request to process a task. The method includes monitoring a plurality of execution nodes of a datastore to determine a plurality of central processing unit (CPU) utilizations. Each CPU utilization of the plurality of CPU utilizations is associated with a respective execution node of the plurality of execution nodes. The method includes identifying, by a processing device based on the plurality of CPU utilizations, a particular execution node associated with a maximum CPU utilization to process the task. The method includes allocating the task to the particular execution node.

Dynamic task allocation and datastore scaling

A data transfer request can be received by each of multiple deployments; however, only a primary deployment can process the request. The data transferred by the primary deployment may be replicated in the secondary deployments. In response to a failover event, one of the secondary deployments can be designated as the new primary development and continue the data transfer based on the data transfer request and the replication information received from the old primary deployment prior to the failover.

Data ingestion replication and disaster recovery

The logging techniques described herein can enable using logging tools without having to use different methods for sandbox implementations and push out the log data to storage without problems. The log data is treated as sensitive data and is protected according to the defined security policies. Further, the results may be compressed and encrypted.

Logging from user-defined functions

Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.

Schema evolution

Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.

Load history calculation in internal stage replication

Disclosed are techniques for deduplicating files to be ingested by a database. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein the data exchange includes a metadata storage where metadata including a list of files ingested is stored. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested into the data exchange, the set of bloom filters may be used to identify from within the set of candidate files, each candidate file that is duplicative of a file in the first set of files.

Growing community of inventors

Redmond, WA, United States of America

Ganeshan Ramachandran Iyer