Amazon Technologies, Inc.

Athanasios Papathanasiou

Induja Sreekanthan

Sriram Subramanian

Vijayan Prabhakaran

Venu Gopal Nayar

Methods for modeling performance of tiered storage of a data processing service given a decrease in the storage capacity of a warm storage tier of the tiered storage are disclosed. Metadata of the warm storage tier is used to track hits due to incoming queries on data blocks that are stored in the warm storage tier. The metadata prioritizes data block identifiers that correspond to the data blocks stored in the warm storage tier by frequency of hits due to the incoming queries, or various other prioritization schemes. One or more partitions of the metadata may be set that correspond to respective downsized storage capacity scenarios of the warm storage tier. When an incoming query targets a data block within a given partition of the metadata, a hit counter is incremented to track the hit rate that would be made on the downsized warm storage tier corresponding to that partition.

Down-sized cluster performance modeling for a tiered data processing service

Methods for improving performance of tiered storage of a data processing service by detecting and reducing thrashing of data blocks between warm and cold storage tiers are disclosed. In order to understand the frequency of hits by incoming queries to data blocks that are not currently stored in the warm storage tier, the elapsed time between query hits to the respective data blocks may be tracked using timers. Times below a given amount of time may be used to indicate thrashing. For example, recently evicted data blocks that are subsequently hit by a query within a short amount of time since eviction may indicate thrashing. In scenarios in which thrashing may be occurring, a threshold corresponding to the number of times a given data block in the cold storage tier receives a query hit before being added to the warm storage tier may be turned on.

Thrashing detection and avoidance for a tiered data processing service

Methods for modeling performance of tiered storage of a data processing service given an increase in the storage capacity of a warm storage tier of the tiered storage are disclosed. Buffers in the warm storage tier are used to store data block identifiers corresponding to a set of data blocks that would be stored in the warm storage tier given the increase in storage capacity in addition to those already stored in the warm storage tier. When an incoming query targets a data block that has a corresponding data block identifier in one of the buffers, a hit counter is incremented in order to track the hit rate that would be made on the up-sized warm storage tier. In response to adding the data block targeted by the query to the warm storage tier, one or more evictions from the warm storage tier may additionally be triggered.

Growing community of inventors

Los Gatos, CA, United States of America

Athanasios Papathanasiou