Amazon Technologies, Inc.

Induja Sreekanthan

Athanasios Papathanasiou

Sriram Subramanian

Vijayan Prabhakaran

Gokul Soundararajan

Ippokratis Pandis

Naresh Chainani

Krishna Chaitanya Gudipati

Fabian Oliver Nagel

Aditya Subrahmanyan

Sanjay Wangoo

Venu Gopal Nayar

Ankil Shah

Yao Xiao

Siyi Zhang

Yehan Zhang

Vaishali Ravindra Narkhede

Methods for modeling performance of tiered storage of a data processing service given a decrease in the storage capacity of a warm storage tier of the tiered storage are disclosed. Metadata of the warm storage tier is used to track hits due to incoming queries on data blocks that are stored in the warm storage tier. The metadata prioritizes data block identifiers that correspond to the data blocks stored in the warm storage tier by frequency of hits due to the incoming queries, or various other prioritization schemes. One or more partitions of the metadata may be set that correspond to respective downsized storage capacity scenarios of the warm storage tier. When an incoming query targets a data block within a given partition of the metadata, a hit counter is incremented to track the hit rate that would be made on the downsized warm storage tier corresponding to that partition.

Down-sized cluster performance modeling for a tiered data processing service

Methods for improving performance of tiered storage of a data processing service by detecting and reducing thrashing of data blocks between warm and cold storage tiers are disclosed. In order to understand the frequency of hits by incoming queries to data blocks that are not currently stored in the warm storage tier, the elapsed time between query hits to the respective data blocks may be tracked using timers. Times below a given amount of time may be used to indicate thrashing. For example, recently evicted data blocks that are subsequently hit by a query within a short amount of time since eviction may indicate thrashing. In scenarios in which thrashing may be occurring, a threshold corresponding to the number of times a given data block in the cold storage tier receives a query hit before being added to the warm storage tier may be turned on.

Thrashing detection and avoidance for a tiered data processing service

Methods for modeling performance of tiered storage of a data processing service given an increase in the storage capacity of a warm storage tier of the tiered storage are disclosed. Buffers in the warm storage tier are used to store data block identifiers corresponding to a set of data blocks that would be stored in the warm storage tier given the increase in storage capacity in addition to those already stored in the warm storage tier. When an incoming query targets a data block that has a corresponding data block identifier in one of the buffers, a hit counter is incremented in order to track the hit rate that would be made on the up-sized warm storage tier. In response to adding the data block targeted by the query to the warm storage tier, one or more evictions from the warm storage tier may additionally be triggered.

Up-sized cluster performance modeling for a tiered data processing service

Online restore may be performed between databases with different topologies while applying a custom data distribution. A request to restore a database into a different topology of nodes may be received. A plan to move different portions of the database from a current topology to the new topology made using a general distribution scheme. The plan may be performed to move the different portions of the database into the new topology and the database made available for access using the new topology. A background process may be applied to modify the distribution of the database at the new topology to match a custom distribution scheme that was implemented at the current topology.

Online restore to different topologies with custom data distribution

Growing community of inventors

Mountain View, CA, United States of America

Induja Sreekanthan