Snowflake Inc.

Saurin Shah

Polita Paulus

Nileema Shingte

Subramanian Muralidhar

Istvan Cseri

Benoit Dageville

Thierry Cruanes

Torsten Walther Grabs

Sukruth Komarla Sukumar

Aravind Ramarathinam

Srinidhi Karthik Bisthavalli Srinivasa

Isaac Kunen

Igor A Zinkovsky

Edward Ma

Elliott Brossard

Ju-yi Kuo

Jonathan Lee Leang

Schuyler James Manchester

Peter Povinec

Abhiram Arikapudi

Vasile Paraschiv

Marianne Shaw

Sahaj Saini

Ganeshan Ramachandran Iyer

Jan Tomasz Topinski

Pawel Pollak

Mikolaj Niedbala

Tomasz Malisz

Michal Gdak

Daria Wieteska

Nithin Mahesh

Robert Bengt Benedikt Gernhardt

Chong Han

Yanrui Zhang

Lucas A Reber

Joshua W McKibben

Mayank Gupta

Kishore Udayashankar Kasi

Sujay C Yantrapragada

Cameron Tekiyeh

Brian Jack Lee

A method to access data includes generating, by at least one hardware processor, a user-defined function (UDF). The UDF includes a file reference object corresponding to at least one file and a file operation configuration. The at least one file is retrieved based on a UDF call into the UDF. The retrieving is further based on the file reference object. A compression status of the at least one file is determined. The UDF call is processed to generate result data using the at least one file. The processing of the UDF call is based on the compression status and the file operation configuration.

Accessing data using a user-defined function (UDF)

Systems and methods are provided for controlling the deletion of data in a database system. The system receives input comprising a deletion criterion for a database system. The system applies the deletion criterion to a set of tables of the database system. The system determines that an individual portion of the set of tables satisfies the deletion criterion. In response to determining that the individual portion of the set of tables satisfies the deletion criterion, the system transfers the individual portion of the set of tables to a temporary storage system.

Storage lifecycle control in a database system

Questionnaire completion systems and methodologies for a data platform. The data platform receives from a consumer an unstructured questionnaire to be completed based on structured database objects, semi-structured database objects, and unstructured database objects stored on the data platform by a provider. The data platform generates a secured completion of the unstructured questionnaire based on a questionnaire completion model and the unstructured questionnaire. The data platform determines a confidence score for the completion and in response to determining the confidence score does not exceed a threshold value, the data platform generates a structured query based on the unstructured questionnaire and a structured query model, and generates the secured completion based on querying the structured database objects using the structured query. The data platform applies a security function to the secured completion to generate a completion of the unstructured questionnaire and provides the completion to the consumer.

Automated completion of questionnaires based on unstructured database objects

Disclosed herein are embodiments of systems and methods for synchronizing file-catalog table with a file stage. In an embodiment, a data platform receives a notification of a modification to one or more files in a file stage. The file stage includes data storage having a storage location. The data platform updates, based on receiving the notification of the modification, a first file-catalog table for the file stage based on the modification. The first file-catalog table includes a row for each file in the file stage, as well as a column for each of one or more metadata properties of the one or more files in the file stage.

Synchronizing file-catalog table with file stage

Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.

Accessing data objects using user defined function (UDF) streams

The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.

Replication of unstructured staged data between database deployments

Systems and methods for generating a machine-learning (ML) model for extracting information from one or more electronic documents, where the ML model can be used as a data object, which can be part of a database command or as part of a document information extraction process that is continuously running (e.g., document information extraction pipeline).

Generating machine-learning model for document extraction

System and method for efficiently querying external tables are described herein. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Systems and methods for efficiently querying external tables

Techniques for secure sharing of stage data include generating a listing in a first data exchange of a data provider. The listing includes stage data stored in a stage of the data provider and metadata associated with the stage data. A secure view of the listing is configured in a second data exchange based on posting the listing from the first data exchange to the second data exchange. The stage data is retrieved in response to a request from a client device to view the stage data received in the second data exchange. A security function is applied to the stage data to generate modified stage data. The modified stage data is stored at a second location in the stage. The metadata is updated to reference the second location in the stage.

Secure sharing of stage data of a data exchange listing

A method includes decoding, by at least one hardware processor, a request for a user-defined function (UDF). The request includes a reference to one or more files. The method further includes generating, by the at least one hardware processor, the UDF based on the request. The UDF includes a file reference object with file path information corresponding to the reference. The file path information identifies a file path to the one or more files. A UDF call into the UDF is detected. The UDF call specifies the file path information. The UDF call is processed to generate result data using the one or more files.

Accessing data using a file reference-based user defined function

In one aspect, a method includes forming, at a database platform, a file stage in a user account on the database platform, the user account being associated with a client, forming, at the database platform, a hidden stage directory table and a data pipeline for the file stage, and binding the data pipeline to the file stage. The method also includes receiving a notification of an event of the file stage. The method also includes identifying the data pipeline corresponding to the file stage. The method also includes updating the hidden stage directory table of the identified data pipeline.

Auto refresh of directory tables for stages

Described herein are techniques for replicating external stages between deployments of e.g., a cloud data lake using a modified storage integration. The modified storage integration may be defined with a set of storage locations. The storage integration comprises a base URL for each of the set of storage locations and each storage location identifies a remote deployment where a cloud platform is hosted and a geographic region of the remote deployment. An external stage object may be bound to the storage integration, where the external stage object facilitates a data loading operation that is currently in progress on the first storage location. In response to detecting an outage at the first storage location, the data loading operation that was in progress on the first storage location may be continued at the second storage location using the storage integration.

Stage replication in a cloud data lake

A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.

Export data from tables into partitioned folders on an external data lake

Growing community of inventors

Kirkland, WA, United States of America

Saurin Shah