Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Other

Yan Valerie Leshinsky

Samuel James McKelvie

Pradeep Jnana Madhavarapu

Anurag Windlass Gupta

James McClellan Corey

Stefano Stefani

Benjamin Tobler

John D Mehr

Navjot Virk

Raman Mittal

Timothy Andrew Rath

Wei Xiao

Erik B Christensen

Anders Hejlsberg

Jakub Kulesza

David Alan Lutz

Kerry Quintin Lee

Jai Vasanth

Stefan Harrington Pharies

Keith W Ballinger

Elissa E S Murphy

Catherine Claire Marshall

Mark W Fussell

Michael David Marr

Henricus Johannes Maria Meijer

Darin Lee Frink

Danny Wei

Ian Paul Nowland

Maximiliano Maccanti

Christopher Nathan Watson

Christopher Lovett

Adam Douglas Morley

Elissa E Murphy

Chia-Hsun Chen

Lara Marie Sosnosky

Lakshmi N Pallikila

Mustafa I Abrar

Marc John Brooker

Gopala Krishna R Kakivaya

Christopher Wellington Brumme

Stephen James Millet

Giovanni Moises Della-Libera

Nikhil Kothari

Michael M Magruder

Christopher Richard Newcombe

Alexandre Olegovich Verbitski

Raja Krishnaswamy

David Bau

Brian LaMacchia

Gregory Darrell Fee

Lance Olson

Scott David Guthrie

Kamal Kant Gupta

Ippokratis Pandis

Henry L Sanders

David S Ebbo

Julie D Bennett

Tengiz Kharatishvili

Sean Edwin Trowbridge

Travis Muhlestein

James S Miller

Jonathan C Hawkins

Krzysztof J Cwalina

Seth Mitchell Demsey

Luca Bolognese

Brian M Grunkemeyer

Naresh Chainani

Shawn Patrick Burke

Andrew Eyring Kimball

Christopher L Anderson

Loren Kohnfelder

Evgeny Veselov

Yung-Shin Lin

Brian Keith Pepin

Anthony J Moore

Mark T Anders

Erik B Olson

Jun Fang

Yann Christensen

Fabio A Yeon

Jeffrey M Cooperstein

Susan M Warren

Brian Joseph Moore

James H Hogg

Nicholas M Kramer

Manu Vasandani

Craig T Sinclair

Robert Michael Howard

Corina Eva Feuerstein

Adam Wade Smith

David S Mortenson

Caleb L Doise

Alan Geoffrey Boshier

Shajan Dasan

Chad W Royal

Manish S Prabhu

Radu Rares Palanca

Tarun Anand

Joseph L Roxe

Eric Ray Hotinger

A distributed data store may implement passive distribution encryption keys to enable access to encrypted data stored in the distributed data store. Keys to encrypt a data volume stored in the distributed data store may be encrypted according to a distribution key and provided to a client of the distributed data store. Storage nodes that maintain portions of the data volume may receive the encrypted key from a client to enable access to the data volume. The storage nodes may decrypt the key according to the distribution key and enable access to the data volume at the storage nodes. In some embodiments, a key hierarchy may be implemented to encrypt the keys that provide access to the encrypted data. The key hierarchy may include a user key.

Passive distribution of encryption keys for distributed data stores

Database query processing may be scaled using additional processing clusters. A database query is received at a processing cluster. A determination is made as to whether additional processing clusters will be used to process the database query. Operations to cause compute nodes of the processing cluster to instruct operations at the additional processing clusters are included in a plan generated to perform database queries determined to use additional processing clusters. The plan is executed to be perform the database query causing compute nodes of the processing cluster to send instructions to corresponding additional processing clusters in order to generate and return a response to the database query.

Scaling database query processing using additional processing clusters

A log-structured data store implementing data backup may implement variable data replication. Write requests may be received at different storage nodes maintaining respective replicas of a portion of a log for data maintained in the log-structured data store. Log records indicating the write requests may be stored in the respective replicas of the log portions at the different storage nodes. The log records may be sent to a backup data store to be durability persisted as part of an archived version of the log. At some of the storage nodes, in response to determining that the log records have been durably persisted in the backup data store, storage space for the log records may be reclaimed. In other remaining storage nodes, the log records may be retained and made accessible for servicing read requests.

Variable data replication for storage implementing data backup

A system that implements a scalable data storage service may maintain tables in a data store on behalf of storage service clients. The service may maintain table data in multiple replicas of partitions that are stored on respective computing nodes in the system. In response to detecting an anomaly in the system, detecting a change in data volume on a partition or service request traffic directed to a partition, or receiving a service request from a client to split a partition, the data storage service may create additional copies of a partition replica using a physical copy mechanism. The data storage service may issue a split command defined in an API for the data store to divide the original and additional replicas into multiple replica groups, and to configure each replica group to maintain a respective portion of the table data that was stored in the partition before the split.

System and method for performing live partitioning in a data store

A distributed data store may provide continuous backup for data stored in the distributed data store. Updates to data may be replicated amongst storage nodes according to a peer-to-peer replication scheme. A backup node may participate in the peer-to-peer replication scheme to identify additional updates to be applied to a backup version of the data in a separate data store. The backup node may obtain the updates according to the peer-to-peer replication scheme and update the backup version of the data. In some embodiments, configuration changes to the data in the distributed data store may be detected via the peer-to-peer replication scheme such that a backup node can adapt performance of backup operations in conformity with the configuration change.

Continuous backup of data in a distributed data store

A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous to quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Dynamic quorum membership changes

A storage engine may implement dynamic splitting of contentious data pages. Data pages may store data for a table of a data store as part of an indexing structure for the table. Access to the table may be provided by locating the corresponding data pages via the indexing structure. Access contention for different data pages may be monitored. Data pages may be identified for splitting based on the monitoring. A split operation for an identified data page may be formed to store the data on the identified data page on two different data pages so that subsequent access requests for the data are divided between the two data pages. Monitoring of access contention may also be performed to identify data pages for merging in order to consolidate access requests to a single data page.

Dynamic splitting of contentious index data pages

A data store may implement variable data replication and quorum-based durability schemes. Write requests may be maintained in complete views of the data or incomplete views of the data at storage nodes in a quorum set in the data store. Write requests may be received at different storage nodes sufficient to establish a write quorum, and stored in the replicas of the data at the storage nodes where they are received. Storage nodes maintaining incomplete views of the data may determine that each of the storage nodes maintaining a complete view of the data in the quorum set stores certain write requests, and in response, may reclaim storage space for those write requests included in the incomplete views of the data in order to store other data at the storage node.

Variable data replication for storage systems implementing quorum-based durability schemes

A database system may receive a write request that specifies a modification to be made to a particular data record stored by the database system. A log record representing the modification to be made to the particular data record may be sent to a storage service of the database system. An indication (e.g., log record or other indication) that indicates a cached version of the particular data record stored in a read replica's cache is stale may be sent to a read replica. For a subsequent read of the particular data record received by the read replica, the read replica may request the particular data record from the storage service.

Efficient read replicas

A data store maintaining data may implement memory-based on-demand data page generation. Log records may be maintained in system memory according to a version threshold for data pages of data stored in persistent storage. When a request for a version of a particular data page is received that is within the version threshold, the data page may be obtained from persistent storage and log records associated with the data page may be applied to the data page to generate the requested version of the data page. The version threshold for the data pages may change, and log records in system memory may be identified for coalesce operations. The coalesce operations may be efficiently performed (e.g., in the background), updating the data pages in persistent storage to include the changes of the identified log records and space in system memory for the identified log records may be reclaimed.

Memory-based on-demand data page generation

Hybrid quorum policies may be implemented for access requests to state maintained in a distributed system. An update to state maintained in a distributed system may be sent to multiple nodes in the distributed system maintain the state in order to satisfy a hybrid quorum policy for updating the state in the distributed system. The hybrid quorum policy may include individual quorum policies for multiple quorum sets including nodes of the distributed system. Acknowledgments received from nodes in the distributed system indicating that the update is performed may be evaluated according to the individual quorum policies to determine whether the hybrid quorum policy is satisfied. If the hybrid quorum policy is satisfied, then the update to the state may be considered durable.

Hybrid quorum policies for durable consensus in distributed systems

A log-structured data store may implement optimized log storage for asynchronous log updates. In some embodiments, log records may be received indicating updates to data stored for a storage client and indicating positions in a log record sequence. The log records themselves may not be guaranteed to be received according to the log record sequence. Received log records may be stored in a hot log portion of a block-based storage device according to an order in which they are received. Log records in the hot log portion may then be identified to be moved to a cold log portion of the block-based storage device in order to complete a next portion of the log record sequence. Log records may be modified, such as compressed, or coalesced, before being stored together in a data block of the cold log portion according to the log record sequence.

Optimized log storage for asynchronous log updates

Growing community of inventors

Kirkland, WA, United States of America

Yan Valerie Leshinsky