Snowflake Inc.

Microsoft Technology Licensing, LLC

Facebook, Inc.

Igor A Zinkovsky

Isaac Kunen

Istvan Cseri

Elliott Brossard

Nitya Kumar Sharma

Srilakshmi Chintala

Albert L Hu

Ju-yi Kuo

Jonathan Lee Leang

Saurin Shah

Polita Paulus

Edward Ma

Sukruth Komarla Sukumar

Schuyler James Manchester

Jonathon Michael Stall

Anthony Crider

Chong Han

Rodger N Kline

Tyler Jones

Dinesh Chandrakant Kulkarni

Muthukaruppan Annamalai

Thomas Edward Quinn, Jr

James J Hugunin

Kaushik Ravichandran

Harish Srinivas

Ziliang Zhang

Marcin Sawicki

Selcuk Aya

David Andrew Whitechapel

Michael Shneerson

Harish Venkata Kantamneni

Apurva A Sinha

Luning Pan

Anthony L Crider

A method to access data includes generating, by at least one hardware processor, a user-defined function (UDF). The UDF includes a file reference object corresponding to at least one file and a file operation configuration. The at least one file is retrieved based on a UDF call into the UDF. The retrieving is further based on the file reference object. A compression status of the at least one file is determined. The UDF call is processed to generate result data using the at least one file. The processing of the UDF call is based on the compression status and the file operation configuration.

Accessing data using a user-defined function (UDF)

Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.

Accessing data objects using user defined function (UDF) streams

Methods, systems, and computer programs are presented for a policy to enable customers to create and apply allowed items and restricted items provided by third-party managers. User code is received by a cloud data platform, the code for performing an operation. The cloud data platform receives a policy including allowed items and restricted items. The cloud data platform compares the allowed and restricted items to a specification, and determines a difference based on the comparison. The cloud data platform determines whether to permit the operation based at least in part on the differences.

Providing access control for policies

Multiple execution environments are established for processing database statements with diverse package dependencies. A database system receives a user application and identifies, via hardware processors, multiple package dependencies required by the application, including at least a first and a second package dependency. Upon receiving a database statement from a user that invokes these dependencies, the system retrieves the necessary packages from a repository. It then creates a first execution environment incorporating the first package dependency and a second execution environment with the second package dependency. Utilizing the multiple execution environments, the system generates results data in response to the database statement, effectively managing and executing user-defined functions that rely on different versions or types of package dependencies within a unified database framework.

Multiple environments and functions in a database system

External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.

Autoscaling external function requests

An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Processing external functions using user-defined functions (UDFs)

A method includes detecting, by at least one hardware processor, an upload of a user application within a database system. The user application includes user-defined function (UDF) code. A plurality of dependencies of the user application is determined by the at least one hardware processor. A plurality of execution environments corresponding to the plurality of dependencies is generated by the at least one hardware processor. The plurality of execution environments is associated with a corresponding plurality of data types of the UDF. A database query is decoded. The database query specifies database data of a data type of the plurality of data types of the UDF. The database query is processed using at least one of the plurality of execution environments to generate results data. The at least one of the plurality of execution environments corresponds to the data type of the plurality of data types of the UDF.

Processing user-defined functions (UDFs) using multiple execution environments

A method includes decoding, by at least one hardware processor, a request for a user-defined function (UDF). The request includes a reference to one or more files. The method further includes generating, by the at least one hardware processor, the UDF based on the request. The UDF includes a file reference object with file path information corresponding to the reference. The file path information identifies a file path to the one or more files. A UDF call into the UDF is detected. The UDF call specifies the file path information. The UDF call is processed to generate result data using the one or more files.

Accessing data using a file reference-based user defined function

Systems and methods for caching file system collections atomically. The systems and methods perform operations comprising: receiving a request to access a collection of data comprising a plurality of files stored on a storage device; in response to receiving the request, transferring the collection of data from the storage device to a first subfolder in a cache associated with the storage device; generating a lock file comprising a reference to the first subfolder in the cache; and atomically controlling access to the collection of data in the first subfolder and removal of the collection of data in the first subfolder from the cache via the lock file.

Atomic cache management of file collections

A database dependency resolver system can identify different dependencies of a user application and integrate the identified dependencies in different execution environments of a distributed database system. The different execution environments can manage different versions of a given programming language, or other types of computational architectures (e.g., different CPU types). A database user can provide a database statement (e.g., query) that activates the different dependencies in the different environments to generate results data.

Multiple environments and functions in a distributed database

Extending database external functions with user-defined functions

Methods, systems, and computer programs are presented for a packages policy object to enable customers to create and apply packages policies provided by third-party package managers. A user-defined function (UDF) is received by a cloud data platform. The UDF includes code related to at least one operation to be performed. The cloud data platform receives a package policy including at least one allowlist and at least one blocklist and compares the at least one allowlist and the at least one blocklist to the at least one package specification. The cloud data platform computes a difference set based on the comparison. The difference set includes metadata associated with one or more packages to remove. The cloud data platform determines whether the UDF is permitted based at least in part on the difference set.

Granular access control of packages

A database query specifies a user-defined function (UDF) to apply to one or more tables in a distributed database. The one or more tables are identified in the distributed database. The one or more tables are associated with an initial set of data types native to the distributed database. The one or more tables are converted to database data comprising data types of the UDF. The UDF is executed on the database data to generate results data. The results data is converted to table results data. The table results data is associated with the initial set of data types. The table results data is stored in the distributed database.

Database encoding and decoding for database functions

A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Configuring parallelism parameters for invocation of external table functions

A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Growing community of inventors

Bellevue, WA, United States of America

Igor A Zinkovsky