Snowflake Inc.

Daniel Geoffrey Karp

Johan Harjono

Arthur Kelvin Shi

Rares Radut

Kunal Prafulla Nabar

Samir Rehmtulla

Thanakul Wattanawong

Ata E Husain Bohra

The subject technology selects a candidate compute service manager from a set of instances of compute service managers to issue a query restart by selecting an execution node, the execution node being included in a particular virtual warehouse associated with the candidate compute service manager, the selecting facilitating improving utilization of cluster resources and improving query execution on the selected candidate compute service manager. The subject technology receives a notification indicating that a particular compute service manager has been quiesced. The subject technology determines a set of jobs that are not yet scheduled for execution and eligible for query retry. The subject technology determines a second set of jobs from the set of jobs to send at least another compute service manager for execution. The subject technology sends the second set of jobs to at least another compute service manager for execution.

Query retries using quiesce notifications

Techniques described herein can optimize usage of computing resources in a data system. Dynamic throttling can be performed locally on a computing resource in the foreground and autoscaling can be performed in a centralized fashion in the background. Dynamic throttling can lower the load without overshooting while minimizing oscillation and reducing the throttle quickly. Autoscaling may involve scaling in or out the number of computing resources in a cluster as well as scaling up or down the type of computing resources to handle different types of situations.

Autoscaling and throttling in an elastic cloud service

The subject technology selects a particular zone among multiple zones based on a target skew to meet a global balancing of cluster instances. The subject technology deploys a particular type of cluster instance to the particular zone. The subject technology, for each zone from the multiple zones, determines a respective number of cluster instances. The subject technology identifies a second particular type of cluster instance to add based on a total number of the second particular type of cluster instance in the multiple zones and a second total number of the particular type of cluster instance in the multiple zones. The subject technology adds the second particular type of cluster instance to a second particular zone to meet the global balancing of cluster instances in the multiple zones.

Cluster balancing for zones of a database system

The subject technology retrieves information related to a set of instances of compute service managers, each instance of a particular compute service manager being associated with a set of virtual warehouses. The subject technology filters the information to determine a set of candidates from the set of instances of compute service managers. The subject technology sorts the set of candidates based at least in part on each workload of each of the set of candidates. The subject technology selects a candidate compute service manager from the set of instances of compute service managers to issue a query restart by randomly selecting an execution node, the execution node being included in a particular virtual warehouse associated with the candidate compute service manager, the selecting facilitating improving utilization of cluster resources and improving query execution on the selected candidate compute service manager.

Quiesce notifications for query retries

The subject technology selects a particular zone among multiple zones based on a target skew to meet a global balancing of cluster instances. The subject technology deploys a particular type of cluster instance to the particular zone. The subject technology, for each zone from the multiple zones, determines a respective number of cluster instances. The subject technology identifies a second particular type of cluster instance to remove based on a total number of the second particular type of cluster instance in the multiple zones and a second total number of the particular type of cluster instance in the multiple zones. The subject technology removes the second particular type of cluster instance from a second particular zone to meet the global balancing of cluster instances in the multiple zones.

Cluster instance balancing of a database system across zones

Autoscaling in an elastic cloud service

The subject technology determines an availability zone skew among multiple zones. The subject technology, based on the availability zone skew, determines a target skew to meet a global balancing of cluster instances. The subject technology, based on the target skew, selects a particular zone among multiple zones. The subject technology deploys a particular type of cluster instance to the particular zone. The subject technology, for each zone from the multiple zones, determining a respective number of cluster instances. The subject technology identifies a first zone that includes a highest number of cluster instances based on the respective number of cluster instances from each zone. The subject technology identifies a second zone that includes a lowest number of cluster instances based on the respective number of cluster instances from each zone.

Zone balancing in a multi cluster database system

The subject technology requests information regarding an instance identifier of a compute service manager instance to a particular job. The subject technology retrieves information related to a set of instances of compute service managers in a set of virtual warehouses. The subject technology filters the information to determine a set of candidates from the set of instances of compute service managers. The subject technology sorts the set of candidates based at least in part on a workload. The subject technology selects a candidate compute service manager to issue a query restart by randomly selecting an execution node.

Query retry using quiesce notification

The subject technology determines, after a period of time elapses over a periodic segment of time, an imbalance of cluster instances deployed in multiple zones based on a threshold value, the cluster instances including different types of clusters associated with compute service manager instances. The subject technology identifies a particular type of cluster instance to include in a particular zone from the multiple zones. The subject technology adds the particular type of cluster instance to the particular zone to meet a global balancing of cluster instances in the multiple zones. The subject technology determines, after a second period of time elapses over the periodic segment of time, that a number of cluster instances deployed in the multiple zones is below the threshold value indicating a current balance of cluster instances in the multiple zones.

Growing community of inventors

San Carlos, CA, United States of America

Daniel Geoffrey Karp