Microsoft Technology Licensing, LLC

Amazon Technologies, Inc.

Vmware, Inc.

Datrium, Inc.

Vijayan Prabhakaran

Douglas Brian Terry

Mehul A Shah

Lidong Zhou

Sriram Subramanian

Benjamin Albert Sowell

Bohou Li

Induja Sreekanthan

Athanasios Papathanasiou

R Hugo Patterson, Iii

Gokul Soundararajan

Ganesh Venkitachalam

Lakshmi Narayanan Bairavasundaram

Thomas Lee Rodeheffer

Garrett Cale Smith

Sazzala Reddy

Mahesh Balakrishnan

John D Davis

Marcos Kawazoe Aguilera

Roy Levin

Michael Isard

Qifa Ke

Chandramohan Amyangot Thekkath

Yuan Yu

Yinglian Xie

Kunal Talwar

Krishna Chaitanya Gudipati

Jonathan James Currey

Ramakrishna Rao Kotla

Sanuj Basu

Ehud Wieder

JunFeng Yang

Venugopalan Ramasubramanian

Aditya Subrahmanyan

Jingyue Wu

Venu Gopal Nayar

Yiying Zhang

Vinay Agrawal

Vikas Malik

Tushar Poddar

Mukesh Punhani

Rajesh Parangi Sharabhalingappa

Hussam Abu-Libdeh

Methods for modeling performance of tiered storage of a data processing service given a decrease in the storage capacity of a warm storage tier of the tiered storage are disclosed. Metadata of the warm storage tier is used to track hits due to incoming queries on data blocks that are stored in the warm storage tier. The metadata prioritizes data block identifiers that correspond to the data blocks stored in the warm storage tier by frequency of hits due to the incoming queries, or various other prioritization schemes. One or more partitions of the metadata may be set that correspond to respective downsized storage capacity scenarios of the warm storage tier. When an incoming query targets a data block within a given partition of the metadata, a hit counter is incremented to track the hit rate that would be made on the downsized warm storage tier corresponding to that partition.

Down-sized cluster performance modeling for a tiered data processing service

Methods for improving performance of tiered storage of a data processing service by detecting and reducing thrashing of data blocks between warm and cold storage tiers are disclosed. In order to understand the frequency of hits by incoming queries to data blocks that are not currently stored in the warm storage tier, the elapsed time between query hits to the respective data blocks may be tracked using timers. Times below a given amount of time may be used to indicate thrashing. For example, recently evicted data blocks that are subsequently hit by a query within a short amount of time since eviction may indicate thrashing. In scenarios in which thrashing may be occurring, a threshold corresponding to the number of times a given data block in the cold storage tier receives a query hit before being added to the warm storage tier may be turned on.

Thrashing detection and avoidance for a tiered data processing service

A protocol for implementing ACID transactions that provides snapshot isolation in a distributed setting that does not require synchronized clocks is described. The protocol ensures at commit time that transactions touching common objects do not commit out of order. The protocol can be used in the context of a distributed data lake built on an object store in which clients can transactionally add or remove objects from logical tables.

Transactional protocol for snapshot isolation without synchronized clocks

Methods for modeling performance of tiered storage of a data processing service given an increase in the storage capacity of a warm storage tier of the tiered storage are disclosed. Buffers in the warm storage tier are used to store data block identifiers corresponding to a set of data blocks that would be stored in the warm storage tier given the increase in storage capacity in addition to those already stored in the warm storage tier. When an incoming query targets a data block that has a corresponding data block identifier in one of the buffers, a hit counter is incremented in order to track the hit rate that would be made on the up-sized warm storage tier. In response to adding the data block targeted by the query to the warm storage tier, one or more evictions from the warm storage tier may additionally be triggered.

Up-sized cluster performance modeling for a tiered data processing service

Techniques for using tree data structures to maintain a transactionally consistent set with support for time-travel queries are described. When a transaction commits, a new version of the tree data structure is created using a copy-on-write based method such that the tree shares internal nodes with previous trees to save space. This approach may be used in the implementation of a transactional data catalog in which the files that make up a table are stored in a transactional set.

Tree-based approach for transactionally consistent version sets

Techniques for implementing systems using transactional version sets are described. Transactional version sets or t-sets include a collection of elements, each having a collection of metadata. A t-set is transactional in that a sequence of updates to one or more t-sets are made within an atomic transaction. A t-set is versioned since each committed transaction that updates it produces a new timestamped version that can be accessed via time-travel queries.

Transactional version sets

Techniques for resizing a distributed database are described. A request to resize a distributed database is received. The distributed database stores data organized into one or more rows of one or more tables. Each node of the first plurality of nodes is assigned a portion of the data. A portion of the data assigned to a first node in the first plurality of nodes is selected to be assigned to a second node in a second plurality of nodes. The number of nodes in the first and second plurality of nodes is different, and the first and second plurality of nodes include at least one common node. Metadata of the selected portion of the data is transferred from the first node to the second node. The metadata that includes a location of the selected portion of the data within a provider network.

In place resize of a distributed database

Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

System and methods for storage data deduplication

A machine-implemented method includes automatically determining that a host device is restarting from a disruptive stoppage of operations and that in-process write transactions by the host device to respective pages of non-volatile storage may have been interrupted. The method includes, in response to the determination, automatically scanning the non-volatile storage for all metadata-containing storage pages with respective identifications S(i) and having corresponding metadata relating each respective storage page S(i) to a corresponding data page P(j) and a corresponding version number V(k). The method includes automatically identifying scanned storage pages S(i) that have for their corresponding data page P(j) a most recent version number HV(k) and, in some cases, a secondmost recent version number. The method includes designating for expungement scanned storage pages S(i) that are not both of committed and having the more recent of the most recent and secondmost recent version number for their corresponding data structure page P(j).

Cyclic commit transaction protocol

A cyclic commit protocol is used to store relationships between transactions and is used by the technology to determine whether a transaction is committed or not. The protocol allows creation of a cycle of transactions which can be used to recover the state of a storage device after a host failure by identifying the last committed version of intention records as committed or uncommitted based on the data stored in the physical pages.

A data partitioning plan is automatically generated that&#x2014;given a data-parallel program and a large input dataset, and without having to first run the program on the input dataset&#x2014;substantially optimizes performance of the distributed execution system that explicitly measures and infers various properties of both data and computation to perform cost estimation and optimization. Estimation may comprise inferring the cost of a candidate data partitioning plan, and optimization may comprise generating an optimal partitioning plan based on the estimated costs of computation and input/output.

Optimizing data partitioning for data-parallel computing

An application programming interface is provided that allows applications to assign multiple service-level agreements to their data transactions. The service-level agreements include latency bounds and consistency guarantees. The applications may assign utility values to each of the service-level agreements. A monitor component monitors the various replica nodes in a cloud storage system for latency and consistency, and when a transaction is received from an application, the monitor determines which of the replica nodes can likely fulfill the transaction in satisfaction of any of the service-level agreements. Where multiple service-level agreements can be satisfied, the replica node that can fulfill the transaction according to the service-level agreement with the greatest utility is selected. The application may be charged for the transaction based on the utility of the service-level agreement that was satisfied.

Consistency-based service-level agreements in cloud storage environments

A duplicate-aware disk array (DADA) leaves duplicated content on the disk array largely unmodified, instead of removing duplicated content, and then uses these duplicates to improve system performance, reliability, and availability of the disk array. Several implementations disclosed herein are directed to the selection of one duplicate from among a plurality of duplicates to act as the proxy for the other duplicates found in the disk array. Certain implementations disclosed herein are directed to scrubbing latent sector errors (LSEs) on duplicate-aware disk arrays. Other implementations are directed to disk reconstruction/recovery on duplicate-aware disk arrays. Yet other implementations are directed to load balancing on duplicate-aware disk arrays.

Duplicate-aware disk arrays

A hybrid storage device uses a write cache such as a hard disk drive, for example, to cache data to a solid state drive (SSD). Data is logged sequentially to the write cache and later migrated to the SSD. The SSD is a primary storage that stores data permanently. The write cache is a persistent durable cache that may store data of disk write operations temporarily in a log structured fashion. A migration policy may be used to determine how long to cache the data in the write cache before migrating the data to the SDD. The migration policy may be implemented using one or more migration triggers that cause the contents of the write cache to be flushed to the SSD. Migration triggers may include a timeout trigger, a read threshold trigger, and a migration size trigger, for example.

Extending SSD lifetime using hybrid storage

Growing community of inventors

Sunnyvale, CA, United States of America

Vijayan Prabhakaran