Snowflake Inc.

Amazon Technologies, Inc.

Aravind Ramarathinam

Saurin Shah

Polita Paulus

Sukruth Komarla Sukumar

Amit Purohit

Santosh Chandrachood

Ankit Kamboj

Vinayak Thapliyal

Linchi Zhang

Ramkumar Kamalapuram Sugavanam

Min Zhou

Gayatri Ramesh Deo

Lukasz Misiuda

Anoop Gupta

Nithin Mahesh

Tanmoy Dutta

Robert Bengt Benedikt Gernhardt

Brian L Welcker

Sandor Loren Maurice

Alexey Kuznetsov

Chong Han

Lawrence Douglas Webley

Yuhui Yuan

Jagdeep Singh Sidhu

Ramakrishna Dwivedula

Yanrui Zhang

Sachin Honnudike

Sundaresan Krishnamurthy

Serhii Poliakov

Parijatham Santosh Kumar Vodela

James H Mlodgenski

Disclosed herein are embodiments of systems and methods for synchronizing file-catalog table with a file stage. In an embodiment, a data platform receives a notification of a modification to one or more files in a file stage. The file stage includes data storage having a storage location. The data platform updates, based on receiving the notification of the modification, a first file-catalog table for the file stage based on the modification. The first file-catalog table includes a row for each file in the file stage, as well as a column for each of one or more metadata properties of the one or more files in the file stage.

Synchronizing file-catalog table with file stage

The distributed database can implement unstructured data replication using an internal or external storage location. Metadata, such as a directory table that lists the unstructured files, can be replicated across different deployments, followed by replication of the staged data. Replicating the staged data can be implemented by replication of only the stage metadata or replication of the database files between the deployments.

Replication of unstructured staged data between database deployments

In one aspect, a method includes forming, at a database platform, a file stage in a user account on the database platform, the user account being associated with a client, forming, at the database platform, a hidden stage directory table and a data pipeline for the file stage, and binding the data pipeline to the file stage. The method also includes receiving a notification of an event of the file stage. The method also includes identifying the data pipeline corresponding to the file stage. The method also includes updating the hidden stage directory table of the identified data pipeline.

Auto refresh of directory tables for stages

Techniques for query execution against relational databases using connection pooling are described. According to some embodiments, a query processing service is disclosed that enables users to execute queries against target relational databases implemented by a relational database service. The service receives query requests originated by a client application at a web service endpoint and identifies a connection to a particular target database. In some examples, the query request is a Hyper Text Transfer Protocol (HTTP) message comprising a query to be executed by a target database instance within the provider network. The service transmits the query for execution at the target database via the connection and obtains a query result based on the execution of the query. The service transforms the query result into a format suitable for the client and transmits a query response to the client.

SQL execution over HTTP for relational databases using connection pooling

In response to receiving a parallel processing job from a customer, a system operated by a computing resource service provider allocates and configures a cluster of computer systems capable of executing the job. In an embodiment, each computer system is configured with a first network stack that allows access to resources of the computing resource service provider and a second network stack that allows access to resources of the customer. In an embodiment, the state of the cluster is monitored by the system via the first network stack. In an embodiment, the system deploys a set of tasks on the cluster for fulfilling the processing job. In an embodiment, the tasks have access to the second network stack so that each task can access resources of the customer.

Cluster management

Disclosed herein are systems and methods for population of a file-catalog table for a file stage in a user account on a data platform. In an embodiment, a data platform receives, from a client associated with a user account, a request to populate a file-catalog table of the user account based on a plurality of files stored in a file stage of the user account. The data platform responsively executes a list-files table function with respect to the file stage to generate a database-table object having a row for each file stored in the file stage. The data platform populates the file-catalog table of the user account based on the database-table object generated by the list-files table function.

Population of file-catalog table for file stage

Disclosed herein are systems and methods for implementing a file-catalog table for a file stage. In an embodiment, a data platform receives, from a client, a request that the data platform create a file-catalog table for a file stage of a user account associated with the client, where the file stage contains one or more files at a storage location. The user account includes a file-stage object that corresponds to the file stage and that includes an indication of the storage location. In response to receiving the request, the data platform creates the file-catalog table for the file stage. The file-catalog table lists the one or more files in the file stage, and includes at least two different metadata properties of the files listed in the file-catalog table.

File-catalog table for file stage

A system operated by a computing resource service provider maintains a pool of computing resources for use in processing job requests submitted by customers. The system allocates computing resources to the pool in anticipation of future resource needs. In various embodiments, future resource needs can be estimated based on scheduled jobs, or historical job information. In an embodiment, the computing resources are virtual computer systems which may be arranged in a cluster. In response to receiving a parallel processing job from a customer, the system reserves computing resources from the pool for performing the job. In an embodiment, the reserved resources are configured with a network namespace that is able to access to a customer's resources.

Resource management

A data platform provides support for unstructured files, semi-structured files, and structured files. In various different embodiments, the data platform provides functions for obtaining relative paths, absolute paths, and presigned URLs for files in internal and external stages; application programming interfaces (APIs) for integrating with external functions; built-in file-catalog tables; integration with cloud-storage platforms; customized views for flexible file-sharing options, stage-encryption options; notification integrations; table functions for generating file-catalog tables; streams on stages; streams on tables; and other features.

Supporting unstructured, semi-structured, and structured files

Growing community of inventors

Sammamish, WA, United States of America

Aravind Ramarathinam