Snowflake Inc.

Elliott Brossard

Isaac Kunen

Igor A Zinkovsky

Edward Ma

Saurin Shah

Polita Paulus

Sukruth Komarla Sukumar

Ju-yi Kuo

Jonathan Lee Leang

Schuyler James Manchester

Istvan Cseri

Derek Denny-Brown

Nitya Kumar Sharma

Srinath Shankar

Haowei Yu

Rodger N Kline

Andong Zhan

Srilakshmi Chintala

Jacob Salassi

Siddharth Kulkarni

Soumitr Rajiv Pandey

Chong Han

Filip Drozdowski

A method to access data includes generating, by at least one hardware processor, a user-defined function (UDF). The UDF includes a file reference object corresponding to at least one file and a file operation configuration. The at least one file is retrieved based on a UDF call into the UDF. The retrieving is further based on the file reference object. A compression status of the at least one file is determined. The UDF call is processed to generate result data using the at least one file. The processing of the UDF call is based on the compression status and the file operation configuration.

Accessing data using a user-defined function (UDF)

Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.

Accessing data objects using user defined function (UDF) streams

Embodiments described herein provide techniques for in-line compiling of UDFs in other programming languages. These techniques enable faster programming iterations because it allows users to compile directly in the cloud processing system. Moreover, it allows the UDFs to tie into existing libraries. The compiled results are treated as sensitive and handled with appropriate security policies, as with any other user data in the system.

Inline compilation of user defined functions

External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.

Autoscaling external function requests

An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Processing external functions using user-defined functions (UDFs)

A method includes detecting, by at least one hardware processor, an upload of a user application within a database system. The user application includes user-defined function (UDF) code. A plurality of dependencies of the user application is determined by the at least one hardware processor. A plurality of execution environments corresponding to the plurality of dependencies is generated by the at least one hardware processor. The plurality of execution environments is associated with a corresponding plurality of data types of the UDF. A database query is decoded. The database query specifies database data of a data type of the plurality of data types of the UDF. The database query is processed using at least one of the plurality of execution environments to generate results data. The at least one of the plurality of execution environments corresponds to the data type of the plurality of data types of the UDF.

Processing user-defined functions (UDFs) using multiple execution environments

A method includes decoding, by at least one hardware processor, a request for a user-defined function (UDF). The request includes a reference to one or more files. The method further includes generating, by the at least one hardware processor, the UDF based on the request. The UDF includes a file reference object with file path information corresponding to the reference. The file path information identifies a file path to the one or more files. A UDF call into the UDF is detected. The UDF call specifies the file path information. The UDF call is processed to generate result data using the one or more files.

Accessing data using a file reference-based user defined function

Extending database external functions with user-defined functions

A database query specifies a user-defined function (UDF) to apply to one or more tables in a distributed database. The one or more tables are identified in the distributed database. The one or more tables are associated with an initial set of data types native to the distributed database. The one or more tables are converted to database data comprising data types of the UDF. The UDF is executed on the database data to generate results data. The results data is converted to table results data. The table results data is associated with the initial set of data types. The table results data is stored in the distributed database.

Database encoding and decoding for database functions

A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF signatures and interfaces (e.g., classes, sub-classes) that can be called by other users. Upon a UDF being called, one or more interface objects (e.g., InputStream) can be created and requests transferred to a execution node via a network channel. The execution node can implement multiple threads that are authorized and download file data from a staging location (e.g., internal stage, external stage) concurrently.

Accessing files in a database stage using a user defined function

A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF interfaces and file reference objects that can be called by other users. Upon a UDF being called, files on a stage, one or more interface objects (e.g., InputStream), and file reference objects can be implemented by execution nodes of the distributed database system. The execution nodes can implement multiple threads that are authenticated and can download file data from a staging location concurrently.

Growing community of inventors

Kirkland, WA, United States of America

Elliott Brossard