Amazon Technologies, Inc.

Raman Mittal

Murali Brahmadesam

Ramesh Shankar

Yan Valerie Leshinsky

Changan Han

Seungmin Wei

Li-Che Hsiao

Anurag Windlass Gupta

Stefano Stefani

Samuel James McKelvie

Pradeep Jnana Madhavarapu

James McClellan Corey

Yuri L Volobuev

Benjamin Tobler

Saleem Mohideen

Punit Rajgaria

Xiaofeng Bao

Lon Lundgren

Minsoo Oh

Xiang Peng

Benjamin S Ellis

Konstantin Dubinets

Venkatesh Nandakumar

Alexandre Olegovich Verbitski

Sumit Kumar

Grant Alexander MacDonald McAlister

Tiffany Jianto

Shanpei Zhou

Peng Gu

Andrew Willis Callahan

Crosbie Matthew Smith

Kevin Liu

Yi Wen Wong

Aadithya Chandramalle Gowda

Alisdair William Owens

Interruptions in an ordered replication stream can be received from in a database. A database instance of the ordered replication stream may detect missing log records and then evaluate log records that are subsequently received to determine whether to update a portion of an in-memory data structure for performing database requests using a comparison of sequence numbers of the last update corresponding to the portion of the in-memory data structure indicated in the log record and the last update recorded for the portion of the in-memory data structure. For comparison indicating valid portions, the log record can be used and for comparison indicating invalid portions, the portion of the in-memory data structure can be made unavailable.

Recovering from interruptions in database replication streams

Backup and restore of client-managed tables and system-managed tables in a same database may be performed. A request to restore a database with both a client-managed table and system-managed table may be received. The request may specify a point in time to restore the database. A metadata volume for the database that describes both the client-managed table and the system-managed table may be restored and used to identify the volumes for both the client-managed table and shards of the system-managed table. Redo logs for each of the volumes may be used according to timestamps that are synchronized according to a time synchronization service to generate restored versions of the volumes for the client-managed table and the shards of the system-managed table from backup copies, which can then be made available for accessing the database.

Backup and restore of client-managed and system-managed database tables

Variable reclamation of data copies can be performed. Copies of data may be stored in a data store. The copies of the data may be available to restore or return the data to a state described in or based on one of the copies. One or more copies, or portions of a copy, may be selected for invalidation according to a variable rate. The variable rate may be determined based on accesses to the data. Selected copies, or portions of a copy, may then be invalidated.

Variable reclamation of data copies

Connection loss may be minimized for performing database query engine changes. A distributed database system may include different instances of the query engine that provide access to a database. When an event to change the version of the query engine is detected, a copy of the database may be created and a new instance of the query engine created. Read-only access to the database may be maintained using the different instances of the query engine while the new instance may be upgraded to the different version of the query engine. Upon successful installation of the different version of the query engine at the new instance, the new instance may be given read-write access to the database using the copy of the database and other database instances may be upgraded to the different version of the query engine.

Minimizing connection loss when changing database query engine versions

Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Direct storage loading for adding data to a database

A point-in-time clone may be created for a database. A request to create the point-in-time clone may be received. The clone may be provided with access to a storage for the database that stores a history of modifications to the database applicable to return data of the database according to a state of the data at the specified point in time. The clone may then be updated so that the updates made to the clone are stored for subsequent access by the clone.

Creating database clones at a specified point-in-time

Data loss thresholds are enforced when performing updates to mirrored data sets. When an update is received for a data set that is mirrored to other copies of the data set, an evaluation of the difference between the data set and the other copies of the data set may be performed. If the evaluation determines that the difference fails to satisfy a threshold for acceptable data loss at the other copies upon a failure of the data set, a data loss mitigation action may be applied to performance of the update until the difference satisfies the threshold.

Enforcing data loss thresholds for performing updates to mirrored data sets

Online restore operations for a database engine can be performed. A request to restore a database to a previous state can be received. Previously stored content of the database, such as snapshot stored prior to the previous state, can be identified along with log records describing changes to be made to the content prior to the previous state. State information in a query engine can be updated based on the previously stored content and log records so that queries can be processed based on the state information at the database restore to the previous state.

Online restore to a selectable prior state for database engines

A distributed data store may provide continuous backup for data stored in the distributed data store. Updates to data may be replicated amongst storage nodes according to a peer-to-peer replication scheme. A backup node may participate in the peer-to-peer replication scheme to identify additional updates to be applied to a backup version of the data in a separate data store. The backup node may obtain the updates according to the peer-to-peer replication scheme and update the backup version of the data. In some embodiments, configuration changes to the data in the distributed data store may be detected via the peer-to-peer replication scheme such that a backup node can adapt performance of backup operations in conformity with the configuration change.

Continuous backup of data in a distributed data store

Systems and methods are disclosed to implement a synchronous database mirroring technique using delayed visibility write operations. In embodiments, a durable log of write operations in a primary database is replicated to a secondary database. The two databases do not make corresponding operations durable in a synchronous manner. However, the two databases coordinate to provide a visible view of the two durable logs that is guaranteed be durable in both databases. In embodiments, the primary database may monitor the state of the durable log of the secondary database, and periodically update the visible range of the logs to include operations that have been made durable in both databases. Advantageously, the visible range of the logs is permitted to grow asynchronously from the logs themselves, so that the two databases can provide a synchronized visible view of the log data without placing timing constrains on the data's durability.

Synchronous database geo-mirroring using delayed visibility write operations

Adaptive replication of changes may be performed for copies of a database. Log records may be generated and stored that correspond to changes to a database while a database is being copied. If the changes to be applied to a copy of the database is less than or equal to a threshold number of changes, then the copy of the database may be updated using the stored log records. If the changes to be applied to the copy of the database are greater than the threshold number of changes, then the copy of the database may be updated using data stored in the database.

Adaptive database replication for database copies

Data pages of a database may be shared to create a copy of the database. A request to create a copy of the database may be received. Pointers to the data pages in the data store may be generated and stored as part of the copy of the database. When a request to read a data page in the copy of the database is received, the pointer to the data page may be accessed and the data page read from the database. When a request to modify a data page in the copy of the database is received, a modified version of the data page may be stored as part of the copy of the database and the pointer updated to point to the modified version of the data page.

Shared pages for database copies

A log-structured data store may implement optimized log storage for asynchronous log updates. In some embodiments, log records may be received indicating updates to data stored for a storage client and indicating positions in a log record sequence. The log records themselves may not be guaranteed to be received according to the log record sequence. Received log records may be stored in a hot log portion of a block-based storage device according to an order in which they are received. Log records in the hot log portion may then be identified to be moved to a cold log portion of the block-based storage device in order to complete a next portion of the log record sequence. Log records may be modified, such as compressed, or coalesced, before being stored together in a data block of the cold log portion according to the log record sequence.

Growing community of inventors

Seattle, WA, United States of America

Raman Mittal