Amazon Technologies, Inc.

Artem Danilov

Allan Henry Vermeulen

Timothy Daniel Cole

Christopher Richard Jacques De Kadt

John Michael Morkel

Andrew Wayne Ross

Tate Andrew Certain

Yevgeniy Mikhaylyuta

Aaron Gifford Freshwater

Michael Benjamin Deardeuff

Atreya Srivathsan

A transaction request compliant with a first version of a journal schema of a multi-data-store storage system is received at a journal manager. The journal schema indicates attributes of data objects which may be materialized at various data stores of the system. The journal manager stores an entry in the system's journal if the transaction meets acceptance criteria. Writes indicated in the entry are materialized at the data stores after verifying that the entry is compliant with the journal schema. After verifying that member data stores have approved a proposed change to the journal schema, another entry indicating a different version of the journal schema is added to the journal. Client-side components of the system obtain the current version of the journal schema to prepare the transaction requests.

Schema management for journal-based storage systems

A materialization configuration request is received via a programmatic interface from a client of a journal-based multi-data-store database. The request indicates a partitioning rule to be used to select, for respective writes indicated in committed transaction entries of a journal, the materialization node at which the writes are to be stored. A control plane component of the database verifies that a set of materialization nodes corresponding to the partitioning rule has been established, and initiates the propagation of writes from the journal to the materialization nodes by respective write appliers.

Customer-related partitioning of journal-based storage systems

A query restriction descriptor indicating one or more attributes of a data object of a multi-data-store storage system is determined at a client-side component of the system. The query restrictions descriptor is used to prepare at least a read set descriptor of a proposed transaction. The read set of the proposed transaction includes a query with a predicate directed to the one or more attributes. A transformation function is applied to the predicate, and the result of the transformation is included in the read set descriptor. The proposed transaction is sent to a journal manager of the system, which performs read-write conflict detection using the read set descriptor and write set descriptors of one or more previously-committed transactions.

Consistency management using query restrictions in journal-based storage systems

A snapshot manager examines a set of entries of a journal of a multi-data-store storage system. Each entry is formatted in a data-store-independent transaction language and various entries contain a respective indication of a committed change to a data object or to a journal schema. Corresponding to various data objects that are materialized at least at a data store of the storage system, respective compact state change representations are generated based on the set of journal entries affecting the data objects. A given compact state change representation, also formatted in the transaction language, may represent cumulative effects of several changes. A snapshot of the journal, comprising the compact state representations, is stored.

Compact snapshots of journal-based storage systems

A data store manager of a multi-data-store storage system detects an error condition associated with applying a state change indicated in a particular journal entry of the storage system to a first data store. A first redaction entry inserted in the journal at some point after the particular entry is identified. The first redaction entry indicates the particular entry as a redaction target. After verifying that the journal does not include another redaction entry targeting the first redaction entry, processing of the particular entry is terminated and processing of later-inserted entries can proceed.

Storage system journal entry redaction

A client-side component of a journal-based multi-data-store database identifies an application schema corresponding to a first application whose data is stored at the database. The application schema includes a writes section indicating a data object to which writes are to be issued from the first application. A journal schema defining one or more data objects for which a concurrency-control protocol is implemented by a journal manager of the database is obtained. Before a transaction request of the application is submitted to the journal manager, the client-side component verifies that the application schema is compatible with the journal schema.

Application schemas for journal-based databases

A partitioning policy for materializing writes indicated in journal entries of a multi-data-store database is determined. The policy indicates a partitioning attribute of a data object. A write applier associated with a materialization node of the database processes journal entries according to the partitioning policy. If the value of the partitioning attribute associated with a write of a particular journal entry indicates that the write belongs to a partition materialized at the materialization node, the write applier propagates the write to the materialization node for storage at the node's storage devices.

Partitioned data materialization in journal-based storage systems

An analytics tool of a journal-based multi-data-store database obtains respective rejection cause descriptors for a set of transactions rejected by the journal manager of the database due to read-write conflicts. A particular rejection cause descriptor indicates a query predicate used for a read set of a transaction request which was rejected by a journal manager of the database due to a read-write conflict between the read set and a previously-committed transaction. The analytics tool prepares a database contention analysis using the rejection cause descriptors, and transmits the analysis to a client.

Contention analysis for journal-based databases

A particular node of a journal-based multi-node distributed storage system requests the addition of an acceptability verification request entry to the journal, indicating a proposed state change of the system. The particular node examines contents of entries added to the journal after the acceptability verification request entry. After determining that a targeted set of acceptability verification response entries have been added to the journal, indicating that the potential state change meets acceptance criteria at respective other nodes of the system, the particular node requests an addition of a committed transaction entry to the journal, indicating an approval of the proposed state change.

Proactive state change acceptability verification in journal-based storage systems

A forking coordinator of a journal-based multi-data-store database determines that a forking criterion for a first journal has been met. The coordinator stores an indication that processing of transaction requests directed to a particular subset of the database is suspended. The coordinator instantiates a second journal for storing committed transaction entries for the particular subset. Metadata of the second journal indicates that transaction entries with sequence numbers in a particular range are stored in the first journal. The coordinator stores an indication that transaction processing associated with the particular subset of the system is to be resumed using the second journal.

Distributed storage system journal forking

A test coordinator for an application whose data is to be stored at a journal-based multi-data-store database using an optimistic concurrency control protocol identifies a test descriptor. The test descriptor indicates a particular order of a plurality of events associated with the application's data, such as reads from various data stores and propagations of committed writes from the database journal to the data stores. The test coordinator uses a set of control mechanisms to cause the events to occur in a selected execution environment in the particular order, and stores an indication of a state reached by the application in response to the events.

Test framework for applications using journal-based databases

A journal manager of a multi-data-store storage appends a committed transaction entry to a journal, indicating a state change which has been approved for commit using an optimistic concurrency control algorithm. A first representation of the state change is generated and stored at one materialization node. A different representation of the same state change, comprising at least one attribute for which a value is not included in the first representation, is generated and stored at a different materialization node.

Materialization strategies in journal-based databases

A data store manager of a multi-data-store journal-based database performs a sequential analysis of committed transaction entries of a journal. A particular entry includes a directive to determine a value of an attribute of a data object based on a result obtained from a value generator, and does not specify the value of the attribute. The data store manager determines the value using a local version of the value generator, and stores the value in a materialized version of the data object. In response to a programmatic read request, the data store manager provides the materialized version of the data object.

Growing community of inventors

Kirkland, WA, United States of America

Artem Danilov