Amazon Technologies, Inc.

Murali Brahmadesam

Raman Mittal

Changan Han

Saleem Mohideen

Punit Rajgaria

Navaneetha Krishnan Thanka Nadar

Seungmin Wei

Anurag Windlass Gupta

Alexandre Olegovich Verbitski

Yuri L Volobuev

Xiaofeng Bao

Ramesh Shankar

Li-Che Hsiao

Saileshwar Krishnamurthy

Kamal Kant Gupta

Tengiz Kharatishvili

Nanda Kaushik

Gopi K Attaluri

Rui Chen

Yi Ding

Sangyong Hwang

Joseph Andrew Tucek

Sameer R Rajyaguru

Sumit Kumar

Niket Goel

Serguei B Stepaniants

Jeffrey Davis

Kartikey Bhatt

Michael Luis Collado

Grant Alexander McAlister

Bohan Liu

Debanjan Saha

Grant Alexander MacDonald McAlister

Shriram Sridharan

James Laurence Finnerty

Minsoo Oh

Aakash Shah

Sanjay Shanthakumar

Sean Oczkowski

Xiang Peng

Venkatesh Nandakumar

Jan Engelsberg

Konstantin Dubinets

Tiffany Jianto

Praveen Kannan

Kevin Donald Kelly

Robert Andrew Kreek

Gajanan Sharadchandra Chinchwadkar

Ahmad Mohammad Radi Ahmad Alsmair

Anupriya Mathur

Apoorv Birthare

Aravind Kumar Kumar

Haritabh Gupta

Benjamin S Ellis

Shen Li

Ajay Kant Singh

Davor Prugovecki

Raghavendran Gururajan

Amit Krishnan

Kiran Pillarisetty

Sambhavi Raghuraman

Awisha Makwana

Richard Bargreen

Ray Chen

Hancao Li

Vignesh Kannappan

Crosbie Matthew Smith

Kevin Liu

Aadithya Chandramalle Gowda

Donald Kane

Yi Wen Wong

Pramod Muralidharan

Alisdair William Owens

Volodymyr Verovkin

Dhruva Ranjan Chakrabarit

Aleksandr Valerevich Feinberg

Namrata Bapat

Andrew James Whitaker

Li Che David Hsiao

Meet Kiritkumar Bhagdev

Replication of a client-managed table may be configured across shards of system-managed tables in a database system for improved query performance. A client-managed table may be identified to replicate as a complete copy of the table respectively collocated with two or more shards of one or more other system-managed tables. The complete copy may be stored in respective storage volumes of the two or more shards of the one or more other system-managed tables. Metadata for performing access requests at a database system may be updated to identify the client-managed table as collocated with the two or more shards of the one or more other system-managed tables.

Configuring replication across shards of scalable database tables for improved query performance

Variable reclamation of data copies can be performed. Copies of data may be stored in a data store. The copies of the data may be available to restore or return the data to a state described in or based on one of the copies. One or more copies, or portions of a copy, may be selected for invalidation according to a variable rate. The variable rate may be determined based on accesses to the data. Selected copies, or portions of a copy, may then be invalidated.

Variable reclamation of data copies

Shard management may be implemented for scalable database tables. A table may be identified as aligned with another table of the database according to a common shard key across multiple shards that are mapped to respective storage volumes of a data store of the database according to a placement hierarchy. Different portions of the table may be stored across the shards as respective table slices with corresponding portions of the other table as respective other table slices with same values of the shard key. Metadata for performing access requests to the database may be updated to identify individual ones of the table slices stored with individual ones of the other table slices as respective table slice groups that are mapped to respective ones of the shards in the placement hierarchy.

Shard management for scaling database tables

Optimistic concurrency may be implemented for multiple writers to a database. Writes to a database from different writers may succeed unless a conflict is detected at the data store for the database. For detected conflicts, a request to resolve the conflict between writes may be submitted to a primary writer, which may determine and provide a resolution of the conflict to the conflicting writers. The data store may then be updated to commit the selected write identified by the resolution of the conflict.

Optimistic concurrency for a multi-writer database

Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.

Direct storage loading for adding data to a database

Storage level query processing may be implemented for processing database queries. Nodes that can access a database may perform parallel processing for at least a portion of a database query. An indication may be received that specifies parallel processing for the database query. The nodes can then be caused to perform the portion of the query as part of providing a result in response to the database query instead of a node, such as a query engine node, that received the database query.

Storage level parallel query processing

A database system may add a read-only query engine to perform read-only queries associated with points-in-time of a database. In various embodiments, the read-only query engine may be added in response to a manual request, an automatic refresh of a network endpoint, a query specifying a point-in-time, or a connection request. The read-only query engine may perform the point-in-time queries on a version the database at the point-in-time and return results for the queries. Upon completion of the queries or at a determined time, the database system may remove the read-only query engine. The specified point-in-time may refer to a current time, a prior time, or a future time with respect to the current time.

Adding a read-only query engine to perform queries to a point-in-time of a write-accessible database

A point-in-time clone may be created for a database. A request to create the point-in-time clone may be received. The clone may be provided with access to a storage for the database that stores a history of modifications to the database applicable to return data of the database according to a state of the data at the specified point in time. The clone may then be updated so that the updates made to the clone are stored for subsequent access by the clone.

Creating database clones at a specified point-in-time

Systems and methods are disclosed to implement an automatic recovery technique for a detected corruption of stored data in a database system. In embodiments, the database system implements a physical replication process that replicates the contents of a primary volume to a mirror volume, with a specified mirror lag. The database system also stores a log of updates that it has received. The database system is configured to detect an error caused by a recent write operation received during the mirror lag. Upon detection, physical replication to the mirror volume is stopped, and a logical replication is performed where write operations logged during the mirror lag are replayed on the mirror volume. The mirror volume is then promoted to replace the primary volume in the database system. Advantageously, the disclosed recovery technique can be performed very quickly, and with zero loss of client requests received before initiation of the recovery.

Automatic recovery from detected data errors in database systems

Techniques for distributed database storage utilizing split volume types for redo log records and change log records are described. A database engine sends redo log data to a first set of page storage nodes and change log data to a second set of tuple storage nodes, where the change log data is stored in an indexed manner Client access to change log data may be provided through a separate set of streaming storage servers instead of through the database engine.

Streaming database change data from distributed storage

Techniques are described for enabling a relational database service of a cloud provider network to provide online restore or 'backtrack' capabilities that are unlimited in terms of an amount of time in the past to which a database can be restored. Database systems typically provide restoration capabilities that allow users to undo or revert to a prior state of the database. Restoring a database to a prior state can be useful in many different scenarios. For example, a user might restore a database to remove or rollback errors introduced into a database as a result of inadvertent changes to data or to a schema of the data stored in the database. As another example, database administrators or other users can use restore operations as part of testing modifications to a database schema when developing applications or other tools that use the database.

Unlimited database change capture for online database restores

Data loss thresholds are enforced when performing updates to mirrored data sets. When an update is received for a data set that is mirrored to other copies of the data set, an evaluation of the difference between the data set and the other copies of the data set may be performed. If the evaluation determines that the difference fails to satisfy a threshold for acceptable data loss at the other copies upon a failure of the data set, a data loss mitigation action may be applied to performance of the update until the difference satisfies the threshold.

Enforcing data loss thresholds for performing updates to mirrored data sets

Online restore operations for a database engine can be performed. A request to restore a database to a previous state can be received. Previously stored content of the database, such as snapshot stored prior to the previous state, can be identified along with log records describing changes to be made to the content prior to the previous state. State information in a query engine can be updated based on the previously stored content and log records so that queries can be processed based on the state information at the database restore to the previous state.

Online restore to a selectable prior state for database engines

Systems and methods are disclosed to implement a synchronous database mirroring technique using delayed visibility write operations. In embodiments, a durable log of write operations in a primary database is replicated to a secondary database. The two databases do not make corresponding operations durable in a synchronous manner. However, the two databases coordinate to provide a visible view of the two durable logs that is guaranteed be durable in both databases. In embodiments, the primary database may monitor the state of the durable log of the secondary database, and periodically update the visible range of the logs to include operations that have been made durable in both databases. Advantageously, the visible range of the logs is permitted to grow asynchronously from the logs themselves, so that the two databases can provide a synchronized visible view of the log data without placing timing constrains on the data's durability.

Synchronous database geo-mirroring using delayed visibility write operations

Growing community of inventors

Woodinville, WA, United States of America

Murali Brahmadesam