Snowflake Inc.

Tableau Software, Inc.

Microsoft Technology Licensing, LLC

Isaac Kunen

Igor A Zinkovsky

Istvan Cseri

Edward Ma

Elliott Brossard

Srinath Shankar

Derek Denny-Brown

Jun Ho Kim

Will Pugh

Saurin Shah

Polita Paulus

Sukruth Komarla Sukumar

Ju-yi Kuo

Jonathan Lee Leang

Schuyler James Manchester

Zihan Li

Khushboo Bhatia

Nitya Kumar Sharma

Tyler Jones

Srilakshmi Chintala

Albert L Hu

Romualdo T Impas

Donald Allen Barnett

Daniel R Dole

Haowei Yu

David Ross Buerer

Andong Zhan

Juan Pablo Candelas Gonzalez

Brenton Gunning

Siddharth Kulkarni

Soumitr Rajiv Pandey

Jacob Salassi

Thierry Cruanes

Ganeshan Ramachandran Iyer

William Pugh

Mengxi Chen

Filip Drozdowski

Michael Kallay

Bing Li

A method to access data includes generating, by at least one hardware processor, a user-defined function (UDF). The UDF includes a file reference object corresponding to at least one file and a file operation configuration. The at least one file is retrieved based on a UDF call into the UDF. The retrieving is further based on the file reference object. A compression status of the at least one file is determined. The UDF call is processed to generate result data using the at least one file. The processing of the UDF call is based on the compression status and the file operation configuration.

Accessing data using a user-defined function (UDF)

Techniques for accessing data objects using UDF streams include decoding by at least one hardware processor of a first computing node, a first request for a UDF. The first request is generated at a second computing node by one of a plurality of input streams of the UDF. A second request for one or more data objects associated with the UDF is generated at the first computing node using an execution thread assigned to the first request. An authorization of the execution thread is performed using data access rights assigned to the first request. The one or more data objects are retrieved based on the authorization in response to the second request. The one or more data objects are stored in a storage location accessible by the execution thread.

Accessing data objects using user defined function (UDF) streams

A computer system displays a user interface that includes a data flow pane, a profile pane, and a data pane. The data flow pane displays a flow diagram including a plurality of linked nodes. Each node specifies a respective operation and a respective intermediate data set generated upon execution of the respective operation. The profile pane displays schemas corresponding to selected nodes in the flow diagram, including (1) data elements showing information about data fields and (2) statistical information about data values for the data fields. User interaction to modify any of the data elements in the profile pane modifies the flow diagram displayed in the data flow pane. The data pane displays rows of intermediate data sets corresponding to selected nodes in the flow diagram. User interaction with individual data values from the displayed rows of data modifies the flow diagram displayed in the data flow pane.

User interface to prepare and curate data for subsequent analysis

Embodiments described herein provide techniques for in-line compiling of UDFs in other programming languages. These techniques enable faster programming iterations because it allows users to compile directly in the cloud processing system. Moreover, it allows the UDFs to tie into existing libraries. The compiled results are treated as sensitive and handled with appropriate security policies, as with any other user data in the system.

Inline compilation of user defined functions

External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.

Autoscaling external function requests

A database system configured to manage and execute stored procedures within a secure sandbox process. The system receives a response to a database query and, through an Application Programming Interface (API) executing within the sandbox process, converts the response into a remote procedure call. The sandbox process is modified to restrict communication with external networks while enabling communication with a designated execution node, which facilitates interaction between the stored procedure and database system components. The stored procedure, executing within the confines of the sandbox process, directs the API to communicate with the execution node. The execution node, in turn, submits the database query to the database system component.

Invoking specified functionality within a sandbox process

A method includes decoding, by at least one hardware processor, a request for a user-defined function (UDF). The request includes a reference to one or more files. The method further includes generating, by the at least one hardware processor, the UDF based on the request. The UDF includes a file reference object with file path information corresponding to the reference. The file path information identifies a file path to the one or more files. A UDF call into the UDF is detected. The UDF call specifies the file path information. The UDF call is processed to generate result data using the one or more files.

Accessing data using a file reference-based user defined function

A credentials store definition identifying a remote credential store is received. The credential store definition includes access information to enable access to the remote credentials store. A credentials object is created in an internal database based on a credentials object definition. The credentials object identifies a security credential to retrieve from the remote credentials store to access an external resource. At runtime, a request to access the external resource is received, and based on receiving the request, the security credentials identified by the credentials object are retrieved from the remote credential store using the access information. The retrieved security credential is provided to a processing component to access the external resource.

Facilitating access to remotely stored credentials for accessing external resources

The logging techniques described herein can enable using logging tools without having to use different methods for sandbox implementations and push out the log data to storage without problems. The log data is treated as sensitive data and is protected according to the defined security policies. Further, the results may be compressed and encrypted.

Logging from user-defined functions

Methods, systems, and computer programs are presented for providing stored procedures stored within a database to invoke specified functionality in a cloud data platform. A response to a database query is received at a compute node of a cloud data platform. A java database connectivity (JDBC) instance executing within a sandbox process converts the response into a remote procedure call. The remote procedure call is provided to a stored procedure executing within the sandbox process as the response to the database query.

Stored procedures executing within a sandbox process

A computer system displays a user interface that includes a flow pane, a tool pane, a profile pane, and a data pane. The flow pane displays a node/link flow diagram that identifies data sources, operations, and output datasets. The tool pane includes a data source selector that enables users to add data sources to the flow diagram, and includes an operation palette that enables users to insert nodes into the flow diagram for performing specific transformation operations. The profile pane displays schemas corresponding to selected nodes in the flow diagram, including information about data fields and statistical information about data values for the data fields and enables users to modify the flow diagram by interacting with individual data elements. The data pane displays rows of data corresponding to selected nodes in the flow diagram, and enables users to modify the flow diagram by interacting with individual data values.

Accessing external resources using remotely stored credentials

Disclosed are systems, methods, and non-transitory computer-readable media for receiving a remote procedure call from a stored procedure executing within a sandbox process. A database system converts the remote procedure call into a database query, and a sandbox process separates the stored procedure from the other services and processes the database query on a database.

Stored procedures in a database system

A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Configuring parallelism parameters for invocation of external table functions

Growing community of inventors

Seattle, WA, United States of America

Isaac Kunen