Snowflake Inc.

Databricks Inc.

Srinath Shankar

Edward Ma

Harsha S Kapre

Isaac Kunen

Zihan Li

Shi Xin

Khushboo Bhatia

Mark T Keller

Kushan A Zaveri

Andong Zhan

Bing Li

Eric Keng-Hao Liang

Harsh Chaturvedi

Mingli Rui

Alicja Luszczak

Elliott Brossard

Derek Denny-Brown

Haowei Yu

Soumitr Rajiv Pandey

Jacob Salassi

Gregory George Owen

Srilakshmi Chintala

Naresh Kumar

Jianzhun Du

Eric Shawn Vandenberg

Leonhard Franz Spiegelberg

Yun Zou

Istvan Cseri

Ali Ghodsi

Matei Zaharia

Sameer Paranjpye

A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

Query watchdog

A method provides deterministic row ordering for query result sets in a database system. A framework receives a query referencing a database table and generating a row index column in response. The framework generates the row index column that involves determining an order associated with source files underlying the referenced table and assigns each table row a unique index value based on the file order and row position within the files. The framework executes the received query to produce a result set ordered by the generated row index column. The framework determines row order based on analyzing underlying data file metadata and facilitates returning the result set with consistent, reliable ordering, enabling stable positional-based access semantics when querying distributed database tables.

Dataframe workloads using read-only data snapshots

Techniques for fetching query result data using result batches includes retrieving query result information from a first computing node of a database system. The query result information includes result data associated with a multi-stage execution of a query. A plurality of result batches are generated based on the query result information. Each result batch of the plurality of result batches includes a retrieval function for a portion of the result data. The portion of the result data for each result batch of the plurality of result batches is retrieved based on execution of the retrieval function by at least a second computing node of the database system. The result data associated with the multi-stage execution of the query is output based on completion of the retrieval.

Fetching query result data using result batches

A method includes receiving instructions to perform order-dependent DataFrame operation on data. In response to receiving the instructions, the framework analyzes the instructions to identify the order-dependent DataFrame operation, and generates an executable query corresponding to the identified order-dependent DataFrame operation. The framework executes the generated executable query on the data stored in the first database in the cloud data platform, creates a row position column that generates for each row a unique integer identifier, reproducible across multiple sessions and/or queries for the same underlying data to make the data accessible via positional indexing, and assigns each row of the first database a unique row index value based on the row position column and the row position column order. The framework orders a result from performing the order-dependent DataFrame operation based on the unique row index value assigned to each row of the first database, and returns the ordered result to the user.

Row-order dependent dataframe workloads

A database system configured to manage and execute stored procedures within a secure sandbox process. The system receives a response to a database query and, through an Application Programming Interface (API) executing within the sandbox process, converts the response into a remote procedure call. The sandbox process is modified to restrict communication with external networks while enabling communication with a designated execution node, which facilitates interaction between the stored procedure and database system components. The stored procedure, executing within the confines of the sandbox process, directs the API to communicate with the execution node. The execution node, in turn, submits the database query to the database system component.

Invoking specified functionality within a sandbox process

Techniques for configuring query result information include decoding a query received from a client device of a network-based database system. A multi-stage execution of the query is configured to obtain a plurality of query result subsets. A combined query result is generated using the plurality of query result subsets. The combined query result further includes storage location information for each of the plurality of query result subsets. The combined query result is encoded for transmission to the client device in response to the query.

Configuring query result information based on multi-stage query execution

A proxy revocation service provides a reliable service for performing revocation checks. The proxy revocation service queries public certificate authorities for the revocation status of a set of digital certificates and maintains a database of the revocation statuses. The proxy revocation service provides a singular endpoint that is Application Protocol Interface (API) accessible to web clients. Web clients communicate with the proxy revocation service through use of API message to perform revocation checks, rather than communicating with the public certificate authorities using an online certificate status protocol (OCSP). Use of the proxy revocation service provides both a reliable service for performing revocation checks as well as shifts the complexity away from the web clients.

Certificate revocation check proxy service

Methods, systems, and computer programs are presented for providing stored procedures stored within a database to invoke specified functionality in a cloud data platform. A response to a database query is received at a compute node of a cloud data platform. A java database connectivity (JDBC) instance executing within a sandbox process converts the response into a remote procedure call. The remote procedure call is provided to a stored procedure executing within the sandbox process as the response to the database query.

Stored procedures executing within a sandbox process

Techniques for fetching query result data using result batches includes generating a plurality of result batches based on query result information. The query result information is associated with query result data generated from execution of a query. Each result batch of the plurality of result batches includes a result data retrieval function for a corresponding data portion of a plurality of data portions of the query result data. The plurality of result batches are encoded for distribution to a corresponding plurality of computing nodes. The techniques further include causing retrieving of the plurality of data portions of the query result data by the corresponding plurality of computing nodes based on the result data retrieval function for each of the plurality of data portions.

Disclosed are systems, methods, and non-transitory computer-readable media for receiving a remote procedure call from a stored procedure executing within a sandbox process. A database system converts the remote procedure call into a database query, and a sandbox process separates the stored procedure from the other services and processes the database query on a database.

Stored procedures in a database system

A shared database platform can interface with a cluster computing platform over a network through a database connector and one or more cluster connectors. The data transferred over the network can include telemetry metadata that can be distributed to execution nodes of the cluster computing platform for generation and transmission of cluster data to the shared database platform.

Cluster diagnostics data for distributed job execution

Techniques for configuring query result information associated with query result data include configuring execution of a query using a plurality of execution stages. Query result data is generated based on the execution of the query. The query result data includes a plurality of data portions. Each data portion of the plurality of data portions is obtained at a corresponding execution stage of the plurality of execution stages. Query result information is generated based on the execution of the query. The query result information includes authentication information associated with accessing the plurality of data portions.

Growing community of inventors

Belmont, CA, United States of America

Srinath Shankar