Snowflake Inc.

Andong Zhan

Derek Denny-Brown

Haowei Yu

Brandon S Baker

Srinath Shankar

Mark M Manning

Isaac Kunen

Edward Ma

Elliott Brossard

Bing Li

Mingli Rui

Soumitr Rajiv Pandey

Jacob Salassi

Nitya Kumar Sharma

Srilakshmi Chintala

Naresh Kumar

Jianzhun Du

Niranjan Kumar Sharma

Sven Tenzing Choden Konigsmark

Michael A Halcrow

Eric Shawn Vandenberg

Leonhard Franz Spiegelberg

Yun Zou

Systems and methods are disclosed for securely executing user-defined functions within a cloud data platform. A method involves receiving, via hardware processors, a request to execute a user-defined function (UDF) contained within a sandbox process. The UDF comprises code for performing specified operations that necessitate access to external resources. To facilitate this access, a secure egress path is established using an overlay network designed to isolate the UDF's network traffic from other processes. Authentication and authorization details for the UDF are managed externally to the sandbox process, ensuring that the UDF's functionality remains orthogonal to the cloud data platform's operations. This approach enables the secure and controlled execution of UDFs, allowing them to interact with external systems while maintaining the integrity and security of the cloud data platform environment.

Secure execution of user-defined functions

A method provides deterministic row ordering for query result sets in a database system. A framework receives a query referencing a database table and generating a row index column in response. The framework generates the row index column that involves determining an order associated with source files underlying the referenced table and assigns each table row a unique index value based on the file order and row position within the files. The framework executes the received query to produce a result set ordered by the generated row index column. The framework determines row order based on analyzing underlying data file metadata and facilitates returning the result set with consistent, reliable ordering, enabling stable positional-based access semantics when querying distributed database tables.

Dataframe workloads using read-only data snapshots

A method includes receiving instructions to perform order-dependent DataFrame operation on data. In response to receiving the instructions, the framework analyzes the instructions to identify the order-dependent DataFrame operation, and generates an executable query corresponding to the identified order-dependent DataFrame operation. The framework executes the generated executable query on the data stored in the first database in the cloud data platform, creates a row position column that generates for each row a unique integer identifier, reproducible across multiple sessions and/or queries for the same underlying data to make the data accessible via positional indexing, and assigns each row of the first database a unique row index value based on the row position column and the row position column order. The framework orders a result from performing the order-dependent DataFrame operation based on the unique row index value assigned to each row of the first database, and returns the ordered result to the user.

Row-order dependent dataframe workloads

A method for tracing function execution includes instantiating, by at least one hardware processor of a computing node, a user-defined function (UDF) server associated with a plurality of configurations. A plurality of child processes of the UDF server are instantiated using the plurality of configurations. A filtering process is configured at an operating system (OS) kernel of the computing node using a child process of the plurality of child processes. The filtering process includes a set of system call categories and a corresponding set of filtering policies. A system call received at the OS kernel and associated with a system call category of the set of system call categories is detected to violate a corresponding filtering policy of the set of filtering policies. A tracing event of the system call is initiated based on the detecting.

Tracing function execution in a database system

Methods, systems, and computer programs are presented for enabling any sandboxed user-defined function code to securely access the Internet via a cloud data platform. A remote procedure call is received by a cloud data platform from a user-defined function (UDF) executing within a sandbox process. The UDF includes code related to at least one operation to be performed. The cloud data platform provides an overlay network to establish a secure egress path for UDF external access. The cloud data platform enables the UDF executing in the sandbox process to initiate a network call.

Secure network access from sandboxed applications

A method for tracing function execution includes instantiating, by at least one hardware processor of a computing node, a user code runtime configured with access to an operating system (OS) kernel of the computing node. The user code runtime is configured with a first set of filtering policies associated with a first set of allowed system calls. The OS kernel is configured with a second set of filtering policies associated with a second set of allowed system calls. A system call initiated by the user code runtime is detected to violate one or both of the first set of allowed system calls and the second set of allowed system calls. A trace of the system call is initiated based on the detecting.

A system includes at least one hardware processor of a computing node and at least one memory storing instructions that cause the at least one hardware processor to perform operations. The operations include instantiating a user code runtime to execute within a sandbox process. The sandbox process configures access by the user code runtime to an operating system (OS) kernel of the computing node. The OS kernel is configured with one or more filtering policies. A determination is performed of whether a system call received by the OS kernel violates the one or more filtering policies. The system call is triggered by at least one operation of the user code runtime. A tracing event is instantiated to trace execution of the system call based on the determination.

Tracing user-defined functions in a database system

A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the cluster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.

Distributed metadata-based cluster computing

Provided herein are systems and methods for tracing and tracing supervision of UDFs in a database system. For example, a method includes receiving a user-defined function (UDF), the UDF including code related to at least one operation to be performed. A user code runtime is instantiated to execute the code of the UDF as a child process. The user code runtime includes a filtering process configured with a plurality of filtering policies. A system call of the at least one operation is detected based on a notification from an operating system (OS) manager, the notification identifying the system call. A determination is made on whether performing the system call is permitted based on the plurality of filtering policies. A report is generated based on the determining.

Tracing supervisor for UDFs in a database system

The subject technology receives, in a computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology determines by a security manager whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy. The subject technology performs the at least one operation. The subject technology sends a result of the at least one operation to the computing process, where sending the result of the at least one operation utilizes a data transport mechanism that supports a network transfer of columnar data.

Managing execution of a user defined function

The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based at least in part on the at least one operation to a second computing process to perform. The subject technology determines, by a security manager executing within the second computing process, whether performing the at least one operation is permitted, the security manager determines restrictions, based at least in part on a security policy, on operations executing within a sandbox environment provided by the second computing process. The subject technology performs, in the second computing process, the at least one operation, the security manager executing within the second computing process.

Secure execution of a user defined function

The subject technology receives, in a first computing process, a user defined function, the user defined function including code related to at least one operation to be performed. The subject technology sends a request based on the at least one operation to a second computing process to perform, the second computing process being different than the first computing process and comprising a sandbox for executing the at least one operation. The subject technology receives, by the second computing process, the request. The subject technology determines, using at least a security policy, whether performing the at least one operation is permitted. The subject technology performs, in the second computing process, the least one operation. The subject technology sends, by the second computing process, a result of the at least one operation to the first computing process.

Secure code execution in a database system

A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the duster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.

Growing community of inventors

San Mateo, CA, United States of America

Andong Zhan