Snowflake Inc.

Nitya Kumar Sharma

Igor A Zinkovsky

Srilakshmi Chintala

Istvan Cseri

Albert L Hu

Isaac Kunen

Elliott Brossard

Rodger N Kline

Brandon S Baker

Derek Denny-Brown

Haowei Yu

Andong Zhan

Chong Han

Niranjan Kumar Sharma

Sven Tenzing Choden Konigsmark

Michael A Halcrow

Selcuk Aya

Ju-yi Kuo

Jonathan Lee Leang

Multiple execution environments are established for processing database statements with diverse package dependencies. A database system receives a user application and identifies, via hardware processors, multiple package dependencies required by the application, including at least a first and a second package dependency. Upon receiving a database statement from a user that invokes these dependencies, the system retrieves the necessary packages from a repository. It then creates a first execution environment incorporating the first package dependency and a second execution environment with the second package dependency. Utilizing the multiple execution environments, the system generates results data in response to the database statement, effectively managing and executing user-defined functions that rely on different versions or types of package dependencies within a unified database framework.

Multiple environments and functions in a database system

Systems and methods are disclosed for securely executing user-defined functions within a cloud data platform. A method involves receiving, via hardware processors, a request to execute a user-defined function (UDF) contained within a sandbox process. The UDF comprises code for performing specified operations that necessitate access to external resources. To facilitate this access, a secure egress path is established using an overlay network designed to isolate the UDF's network traffic from other processes. Authentication and authorization details for the UDF are managed externally to the sandbox process, ensuring that the UDF's functionality remains orthogonal to the cloud data platform's operations. This approach enables the secure and controlled execution of UDFs, allowing them to interact with external systems while maintaining the integrity and security of the cloud data platform environment.

Secure execution of user-defined functions

External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.

Autoscaling external function requests

An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Processing external functions using user-defined functions (UDFs)

Systems and methods for caching file system collections atomically. The systems and methods perform operations comprising: receiving a request to access a collection of data comprising a plurality of files stored on a storage device; in response to receiving the request, transferring the collection of data from the storage device to a first subfolder in a cache associated with the storage device; generating a lock file comprising a reference to the first subfolder in the cache; and atomically controlling access to the collection of data in the first subfolder and removal of the collection of data in the first subfolder from the cache via the lock file.

Atomic cache management of file collections

A database dependency resolver system can identify different dependencies of a user application and integrate the identified dependencies in different execution environments of a distributed database system. The different execution environments can manage different versions of a given programming language, or other types of computational architectures (e.g., different CPU types). A database user can provide a database statement (e.g., query) that activates the different dependencies in the different environments to generate results data.

Multiple environments and functions in a distributed database

Methods, systems, and computer programs are presented for enabling any sandboxed user-defined function code to securely access the Internet via a cloud data platform. A remote procedure call is received by a cloud data platform from a user-defined function (UDF) executing within a sandbox process. The UDF includes code related to at least one operation to be performed. The cloud data platform provides an overlay network to establish a secure egress path for UDF external access. The cloud data platform enables the UDF executing in the sandbox process to initiate a network call.

Secure network access from sandboxed applications

Extending database external functions with user-defined functions

A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Configuring parallelism parameters for invocation of external table functions

Partitioning to support invocation of external table functions on multiple batches of input rows

Handshake protocol to facilitate invocation of external table functions

Invoking external table functions from a database system

Growing community of inventors

Bellevue, WA, United States of America

Nitya Kumar Sharma