Snowflake Inc.

Edward Ma

Isaac Kunen

Srinath Shankar

Zihan Li

Elliott Brossard

Bing Li

Saurin Shah

Igor A Zinkovsky

Polita Paulus

Sukruth Komarla Sukumar

Ju-yi Kuo

Jonathan Lee Leang

Mingli Rui

Schuyler James Manchester

Khushboo Bhatia

Istvan Cseri

Haowei Yu

Andong Zhan

Derek Denny-Brown

Filip Drozdowski

A method to access data includes generating, by at least one hardware processor, a user-defined function (UDF). The UDF includes a file reference object corresponding to at least one file and a file operation configuration. The at least one file is retrieved based on a UDF call into the UDF. The retrieving is further based on the file reference object. A compression status of the at least one file is determined. The UDF call is processed to generate result data using the at least one file. The processing of the UDF call is based on the compression status and the file operation configuration.

Accessing data using a user-defined function (UDF)

Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.

Accessing data objects using user defined function (UDF) streams

A database system configured to manage and execute stored procedures within a secure sandbox process. The system receives a response to a database query and, through an Application Programming Interface (API) executing within the sandbox process, converts the response into a remote procedure call. The sandbox process is modified to restrict communication with external networks while enabling communication with a designated execution node, which facilitates interaction between the stored procedure and database system components. The stored procedure, executing within the confines of the sandbox process, directs the API to communicate with the execution node. The execution node, in turn, submits the database query to the database system component.

Invoking specified functionality within a sandbox process

A method includes decoding, by at least one hardware processor, a request for a user-defined function (UDF). The request includes a reference to one or more files. The method further includes generating, by the at least one hardware processor, the UDF based on the request. The UDF includes a file reference object with file path information corresponding to the reference. The file path information identifies a file path to the one or more files. A UDF call into the UDF is detected. The UDF call specifies the file path information. The UDF call is processed to generate result data using the one or more files.

Accessing data using a file reference-based user defined function

Methods, systems, and computer programs are presented for providing stored procedures stored within a database to invoke specified functionality in a cloud data platform. A response to a database query is received at a compute node of a cloud data platform. A java database connectivity (JDBC) instance executing within a sandbox process converts the response into a remote procedure call. The remote procedure call is provided to a stored procedure executing within the sandbox process as the response to the database query.

Stored procedures executing within a sandbox process

Disclosed are systems, methods, and non-transitory computer-readable media for receiving a remote procedure call from a stored procedure executing within a sandbox process. A database system converts the remote procedure call into a database query, and a sandbox process separates the stored procedure from the other services and processes the database query on a database.

Stored procedures in a database system

A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Cluster diagnostics data for distributed job execution

A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF signatures and interfaces (e.g., classes, sub-classes) that can be called by other users. Upon a UDF being called, one or more interface objects (e.g., InputStream) can be created and requests transferred to a execution node via a network channel. The execution node can implement multiple threads that are authorized and download file data from a staging location (e.g., internal stage, external stage) concurrently.

Accessing files in a database stage using a user defined function

A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF interfaces and file reference objects that can be called by other users. Upon a UDF being called, files on a stage, one or more interface objects (e.g., InputStream), and file reference objects can be implemented by execution nodes of the distributed database system. The execution nodes can implement multiple threads that are authenticated and can download file data from a staging location concurrently.

A shared database platform can interface with a cluster computing platform over a network through a connector. The data transferred over the network can include metadata result packages that can be distributed to worker nodes of the cluster computing platform, which receive the metadata objects and access the result data for further processing on a staging platform, such as a scalable storage platform.

Distributed metadata-based cluster computing

Disclosed are systems, methods, and non-transitory computer-readable media for a providing stored procedures in a network-based database system. A database system executes a stored procedure within a sandbox process. The sandbox process separates the stored procedure from the other services and processes of the database system and may also limit the resources (e.g., storage, memory, etc.) and functionality available to the stored procedure. Stored procedures are commonly designed to execute database queries that are processed by other components of the database system. To provide for secure communication between the stored procedure and the other components of the database system, the sandbox process is loaded with java database connectivity (JDBC) libraries (e.g., JDBC instance) that is modified to limit the communication abilities of the stored procedure.

Stored procedures in a network based database system

The subject technology receives first code in a first programming language, the first code including a reference to a first dataset. The subject technology converting the first code to: second code in a second programming language different than the first programming language, and at least one data query in a second format different than the first programming language. Further the subject technology sends the converted first code and the at least one data query to a data system for execution.

Providing code translation related to resilient distributed datasets in database systems

A system comprises at least one hardware processor and a memory storing instructions. When executed, the instructions cause the at least one hardware processor to perform operations comprising receiving, in a computing process, a Java user-defined table function (Java UDTF), the Java UDTF including code related to a process method to be performed that includes receiving one or more input tables and transforming the one or more input tables to an output table; determining, using at least a security policy, whether performing one or more portions of the process method are permitted; and performing portions of the process method determined to be permitted.

Growing community of inventors

San Jose, CA, United States of America

Edward Ma