Snowflake Inc.

Qiming Jiang

Orestis Kostakis

Boxin Jiang

John Reumann

Abdul Q Munir

Jeffrey Rosen

Prayag Chandran Nirmala

Istvan Cseri

Boyu Wang

Jiaxun Wu

Michel Adar

Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a data query referencing unredacted data of a database, wherein the data query that is received comprises a value identifying a type of sensitive data to be redacted from the unredacted data, responsive to the data query, executing, by a processing device, a redaction operation to identify sensitive data that matches the type within the unredacted data of the database, and returning a redacted data set in which the sensitive data that matches the type is replaced or removed to the data query.

Query-based database redaction

A system for improving task scheduling on a cloud data platform is provided. A task to be executed using resources of a computing cluster is received. A task execution plan is generated and information about data to be used for the ask is accessed. Resource requirements for executing the task are predicted by applying machine learning to the task execution plan and the information about the data. Assignment data is generated to execute the task on the resources by applying machine learning information about a current state of the resources and predicted resource requirements.

Task-execution planning using machine learning

Using an attributes model of a time series forecasting model, determine a set of features based on time series data, the set of features including periodic components. The time series data may be divided into a set of segments. For each segment of the set of segments, a weight may be assigned using an age of the segment, resulting in a set of weighted segments of time series data. Using a trend detection model of the time series forecasting model, trend data from the set of weighted segments of time series data may be determined. A time series forecast may be generated by combining the set of features and the trend data.

Enhanced time series forecasting

Systems and methods for managing input and output error of a machine learning (ML) model in a database system are presented herein. A set of test queries is executed on a first version of a database system to generate first test data, wherein the first version of the system comprises a ML model to generate an output corresponding to a function of the database system. An error model is trained based on the first test data and second test data generated based on a previous version of the system. The error model determines an error associated with the ML model between the first and previous versions of the system. The first version of the system is deployed with the error model, which corrects an output or an input of the ML model until sufficient data has been produced by the error model to retrain the ML model.

Handling system-characteristics drift in machine learning applications

The subject technology receives a query directed to a set of source tables, each source table organized into a set of micro-partitions. The subject technology determines a set of metadata, the set of metadata comprising table metadata, query metadata, and historical data related to the query. The subject technology predicts, using a machine learning model, an indicator of an amount of computing resources for executing the query based at least in part on the set of metadata. The subject technology generates a query plan for executing the query based at least in part on the predicted indicator of the amount of computing resources. The subject technology executes the query based at least in part on the query plan.

Predictive resource allocation for distributed query execution

An example method of tuning a machine learning operation can include receiving a data query comprising a reference to an input data set of a database, generating a plurality of unique sets of hyperparameters by varying a hyperparameter value of each set of hyperparameters of the plurality of unique sets of hyperparameters based on the input data set, in response to receiving the data query, training a plurality of machine learning models using the input data set of the data query, each of the plurality of machine learning models configured according to a respective one of a plurality of unique sets of hyperparameters, selecting a first machine learning model of the plurality of machine learning models based on an accuracy of an output of the first machine learning model, and returning the output of the first machine learning model in response to the data query.

Hyperparameter tuning in a database environment

A system for improving task scheduling on a cloud data platform is provided. A task is received, from a user of a cloud data platform, for execution on a dataset of a cloud data platform using a plurality of resources. A task graph is generated, and metadata related to the dataset is accessed for use in execution of the task. A predicted resource profile is generated by applying a first machine learning scheme to the task graph and the metadata of the dataset. Assignment data is generated to execute processes of the task on the plurality of resources. The assignment data generated by applying a second machine learning scheme to current state data of a current computational state of the plurality of resources and the predicted resource profile generated by the first machine learning scheme.

Data-driven task-execution scheduling using machine learning

Embodiments of the present disclosure may provide a database optimization system that can generate computational values through a database compiler and assignment data for execution of a query by a plurality of nodes of a database system. The computational values and assignment data can be generated by one or more machine learning schemes. The machine learning schemes can be trained on previous computational values and previous assignment data.

Data-driven query-execution scheduling

A processing device may analyze a set of time series data using a time series forecasting model comprising an attributes model and a trend detection model. The attributes model may comprise a modified gradient boosting decision tree (GBDT) based algorithm. Analyzing the set of time series data comprises determining a set of features of the set of time series data, the set of features including periodic components as well as arbitrary components. A trend of the set of time series data may be determined using the trend detection model and the set of features and the trend may be combined to generate a time series forecast.

Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a data query referencing unredacted data of a database, responsive to the data query, executing, by a processing device, a redaction operation to identify sensitive data within the unredacted data of the database, and returning a redacted data set in which the sensitive data is replaced or removed to the data query.

Embodiments of the present disclosure describe systems, methods, and computer program products for executing and tuning a machine learning operation within a database. An example method can include receiving a data query referencing an input data set of a database, executing a plurality of machine learning operations to generate, in view of the input data set, a plurality of output data sets each having a respective accuracy value, wherein each of the plurality of machine learning operations is executed by a processing device according to one of a plurality of unique sets of hyperparameters, selecting a first output data set of the plurality of output data sets in view of the accuracy values, and returning the first output data set in response to the data query.

The subject technology requests information related to usage history metadata from a metadata database. The subject technology receives the requested information from the metadata database, the requested information comprising information related to user demand. The subject technology predicts a size value indicating an amount of computing resources to request for executing a set of queries based on the usage history metadata. The subject technology determines, during a prefetch window of time within a first period of time, a current size of freepool of computing resources. The subject technology, in response to the current size of the freepool of computing resources being smaller than the predicted size value, sends a request for additional computing resources to include in the freepool of computing resources.

Providing resources using predicted size values

Managing freepool resources using predicted size values

The subject technology determines a number of execution nodes to execute a task. The subject technology determines that no virtual warehouse from a pool of virtual warehouses includes at least the number of execution nodes. The subject technology generates a new virtual warehouse including at least the number of execution nodes. The subject technology determines whether a previous execution of a prior task was completed in less than a threshold time period of a time interval, the time interval corresponding to a particular period of time for executing at least one task. The subject technology increments a score corresponding to a size of a particular virtual warehouse. The subject technology selects the new virtual warehouse from the pool of virtual warehouses based at least in part on the incremented score and the number of execution nodes included in the selected new virtual warehouse.

Growing community of inventors

Redmond, WA, United States of America

Qiming Jiang