Snowflake Inc.

Haowei Yu

Andong Zhan

Benoit Dageville

Varun Ganesh

Jiaxing Liang

Jiansheng Huang

Scott Ziegler

Derek Denny-Brown

Isaac Kunen

Srinath Shankar

Edward Ma

Brandon S Baker

Elliott Brossard

Bing Li

Mingli Rui

Niranjan Kumar Sharma

Soumitr Rajiv Pandey

Jacob Salassi

Nitya Kumar Sharma

Sven Tenzing Choden Konigsmark

Michael A Halcrow

Xuguang Yang

Systems and methods are disclosed for securely executing user-defined functions within a cloud data platform. A method involves receiving, via hardware processors, a request to execute a user-defined function (UDF) contained within a sandbox process. The UDF comprises code for performing specified operations that necessitate access to external resources. To facilitate this access, a secure egress path is established using an overlay network designed to isolate the UDF's network traffic from other processes. Authentication and authorization details for the UDF are managed externally to the sandbox process, ensuring that the UDF's functionality remains orthogonal to the cloud data platform's operations. This approach enables the secure and controlled execution of UDFs, allowing them to interact with external systems while maintaining the integrity and security of the cloud data platform environment.

Secure execution of user-defined functions

Systems, methods, and devices for batch ingestion of data into a table of a database. A method includes determining a notification indicating a presence of a user file received from a client account to be ingested into a database. The method includes identifying data in the user file and identifying a target table of the database to receive the data in the user file. The method includes generating an ingest task indicating the data and the target table. The method includes assigning the ingest task to an execution node of an execution platform, wherein the execution platform comprises a plurality of execution nodes operating independent of a plurality of shared storage devices collectively storing database data. The method includes registering metadata concerning the target table in a metadata store after the data has been fully committed to the target table by the execution node.

Batch data ingestion in database systems

This disclosure provides methods and techniques of performing source network address translation (SNAT) at a packet sender (e.g., a client device) instead of a gateway device (e.g., a proxy device). The present disclosure performs a SNAT operation at the packet sender, relieving the gateway device from the SNAT operation to perform other duties (e.g., policy enforcement). An example method of network address translation includes modifying, by a processing device at a data packet origination device (e.g., a client device), network address information in an internet protocol (IP) header of a packet using a public IP address. The method further includes sending the packet to a public network based on the public IP address.

Client-operated and port-range lease based source network address translation

Methods, systems, and computer programs are presented for enabling any sandboxed user-defined function code to securely access the Internet via a cloud data platform. A remote procedure call is received by a cloud data platform from a user-defined function (UDF) executing within a sandbox process. The UDF includes code related to at least one operation to be performed. The cloud data platform provides an overlay network to establish a secure egress path for UDF external access. The cloud data platform enables the UDF executing in the sandbox process to initiate a network call.

Secure network access from sandboxed applications

A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the cluster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.

Distributed metadata-based cluster computing

The subject technology receives, in a computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology determines by a security manager whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy. The subject technology performs the at least one operation. The subject technology sends a result of the at least one operation to the computing process, where sending the result of the at least one operation utilizes a data transport mechanism that supports a network transfer of columnar data.

Managing execution of a user defined function

The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based at least in part on the at least one operation to a second computing process to perform. The subject technology determines, by a security manager executing within the second computing process, whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy, on operations executing within a sandbox environment provided by the second computing process. The subject technology performs, in the second computing process, the at least one operation, the security manager executing within the second computing process.

Secure execution of a user defined function

The subject technology at a data system, an ingest request to ingest one or more files into a table. The subject technology, after obtaining the ingest request and prior to the ingesting of the one or more files, persists the one or more files in a first file queue that corresponds to the table, the first file queue further corresponding to a client account, and the data system further comprising a second file queue that corresponds to both a second client account and a second table. The subject technology ingests, by one or more execution nodes, the one or more files into one or more micro-partitions of the table, each of the one or more micro-partitions comprising contiguous units of storage of a storage device.

Data ingestion using file queues

The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based on the at least one operation to a second computing process to perform, the second computing process being different than the first computing process and comprising a sandbox for executing the at least one operation. The subject technology receives, by the second computing process, the request. The subject technology determines, using at least a security policy, whether performing the at least one operation is permitted. The subject technology performs, in the second computing process, the least one operation. The subject technology sends, by the second computing process, a result of the at least one operation to the first computing process.

Secure code execution in a database system

The subject technology obtains, at a database system, an ingest request to ingest one or more files into a table of a database. The subject technology, after obtaining the ingest request and prior to the ingesting of the one or more files, persists the one or more files in a file queue that corresponds to the table. The subject technology assigns the one or more files to one or more execution nodes to be ingested into the table. The subject technology operates an ingest puller to poll the file queue. The subject technology ingests, by the one or more execution nodes, the one or more files into one or more micro-partitions of the table via one or more pipes.

Batch data ingestion

A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the duster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.

Growing community of inventors

Newark, CA, United States of America

Haowei Yu