Amazon Technologies, Inc.

John Michael Morkel

Timothy Daniel Cole

Allan Henry Vermeulen

Christopher Richard Jacques De Kadt

Artem Danilov

Andrew Wayne Ross

Tate Andrew Certain

Yevgeniy Mikhaylyuta

Derek Avery Lyon

Gregory Branchek Roth

Graeme David Baer

Khaled Salah Sedky

Daniel Stephen Popick

Ajith Harshana Ranabahu

Neil Andrew Eriksson

Bradley Jeffery Behm

Malcolm Featonby

David Everard Brown

Robin Alan Golden

Aaron Gifford Freshwater

Marc Andrew Bowes

Michael Benjamin Deardeuff

Sergejus Barinovas

Francois Norman Daniels

Manish Geverchand Jain

Gina Louise Morris

Atreya Srivathsan

Andrew Flower

Beth H Ginsberg

Keegan Carruthers Smith

Imogen A Wright

A transaction request compliant with a first version of a journal schema of a multi-data-store storage system is received at a journal manager. The journal schema indicates attributes of data objects which may be materialized at various data stores of the system. The journal manager stores an entry in the system's journal if the transaction meets acceptance criteria. Writes indicated in the entry are materialized at the data stores after verifying that the entry is compliant with the journal schema. After verifying that member data stores have approved a proposed change to the journal schema, another entry indicating a different version of the journal schema is added to the journal. Client-side components of the system obtain the current version of the journal schema to prepare the transaction requests.

Schema management for journal-based storage systems

A journaled database system may comprise data nodes that maintain a collection of data structured in accordance with a schema. A change to the schema may be applied by a journal module while a journal consumer continues to operate using a prior version of the schema. A buffer may be formed and have stored therein records describing state change instructions according to the prior view of the schema. An index of correspondence between the records in the buffer and in the source journal may be maintained.

Ejournal transcoding

A materialization configuration request is received via a programmatic interface from a client of a journal-based multi-data-store database. The request indicates a partitioning rule to be used to select, for respective writes indicated in committed transaction entries of a journal, the materialization node at which the writes are to be stored. A control plane component of the database verifies that a set of materialization nodes corresponding to the partitioning rule has been established, and initiates the propagation of writes from the journal to the materialization nodes by respective write appliers.

Customer-related partitioning of journal-based storage systems

A query restriction descriptor indicating one or more attributes of a data object of a multi-data-store storage system is determined at a client-side component of the system. The query restrictions descriptor is used to prepare at least a read set descriptor of a proposed transaction. The read set of the proposed transaction includes a query with a predicate directed to the one or more attributes. A transformation function is applied to the predicate, and the result of the transformation is included in the read set descriptor. The proposed transaction is sent to a journal manager of the system, which performs read-write conflict detection using the read set descriptor and write set descriptors of one or more previously-committed transactions.

Consistency management using query restrictions in journal-based storage systems

A method and apparatus for testing and simulating an access control policy are disclosed. Evaluating an access control policy may be performed by utilizing a deny statement that causes the access request to be rejected despite actions indicated in the access request being authorized. Further, an independent simulation environment may be utilized for testing access control policy evaluation.

Access control policy simulation and testing

A journaled database system may comprise data nodes that maintain a collection of data and provide read access to the data to a client in accordance with a read schema and write access to the client in accordance with a write schema. A change to the schemas may be proposed. A materialization schema may be identified based on correlated determinations that both of the read schema and the write schema are satisfiable based on the materialization schema. The proposed changes may be accepted when the read schema and write schema are simultaneously satisfiable.

Transaction pipeline materialization schemas

A journaled database system may comprise data nodes that collectively maintain a collection of data and provide clients with read and write access to the collection. Correlated classification functions may be associated with read and write operations, such that items affected by a write operation are classified similarly to items accessed during a read operation. Read and write signatures may be formed based on the classification. Conflicts may be detected by comparing the read and write signatures for equivalent or overlapping classifications.

Extended query families

A journaled database may comprise a computing node that forms a local version of a global state, synchronized with respect to the high-water mark of a data node that maintains the global state. Instructions to modify the state that have been committed, subsequent to forming the local version of the state, may be represented in a merge signature. An uncommitted transaction may be checked for conflicts with respect to a last-ordered committed instruction to modify the data state. Validation with respect to earlier instructions represented by the merge signature may be skipped.

Transaction pipelining in a journaled database

A snapshot manager examines a set of entries of a journal of a multi-data-store storage system. Each entry is formatted in a data-store-independent transaction language and various entries contain a respective indication of a committed change to a data object or to a journal schema. Corresponding to various data objects that are materialized at least at a data store of the storage system, respective compact state change representations are generated based on the set of journal entries affecting the data objects. A given compact state change representation, also formatted in the transaction language, may represent cumulative effects of several changes. A snapshot of the journal, comprising the compact state representations, is stored.

Compact snapshots of journal-based storage systems

A data store manager of a multi-data-store storage system detects an error condition associated with applying a state change indicated in a particular journal entry of the storage system to a first data store. A first redaction entry inserted in the journal at some point after the particular entry is identified. The first redaction entry indicates the particular entry as a redaction target. After verifying that the journal does not include another redaction entry targeting the first redaction entry, processing of the particular entry is terminated and processing of later-inserted entries can proceed.

Storage system journal entry redaction

Technology is described for management of shadowing for devices. A computing hub may store device data from a device in a buffer associated with a local device shadow of the device. The computing hub may determine a write status of the device data using a last write marker representing a last data entry written to the buffer. The computing hub may also determine a shadowing upload status of the device data using a last sent shadow marker representing a last data entry of the buffer sent to a device shadowing service in a service provider environment. The computing hub may send computing hub information that includes the last write marker and the last sent shadow marker to the device shadowing service in the service provider environment.

Management of shadowing for devices

A client-side component of a journal-based multi-data-store database identifies an application schema corresponding to a first application whose data is stored at the database. The application schema includes a writes section indicating a data object to which writes are to be issued from the first application. A journal schema defining one or more data objects for which a concurrency-control protocol is implemented by a journal manager of the database is obtained. Before a transaction request of the application is submitted to the journal manager, the client-side component verifies that the application schema is compatible with the journal schema.

Application schemas for journal-based databases

A system or device may implement configuration and testing of network-based service platform resources using a service platform specific language. A configuration file may describe one or more resources of a network-based service platform to be configured, as well as one or more tests to be performed. The configuration file may be described in a service platform specific language for the network-based service platform. When received, the configuration file may be interpreted according to the service platform specific language to identify configuration information for the resources and execution instructions for performing the tests. Requests may be generated and sent to the network-based service platform to configure and provision the resources according to the identified information. Results of the tests may be obtained.

Configuration and testing of network-based service platform resources using a service platform specific language

Systems with multiple components managing storage may implement efficient storage reclamation. A storage reclamation event may be detected for a system component. An indication to a request dispatching mechanism may be made which indicates that the system component is unable to perform further tasks in order to perform storage reclamation. A storage reclamation operation, such as a stop-the-world garbage collection technique, or a service process restart may be performed to reclaim at least some storage. Upon completion of the storage reclamation operation, an indication may be made to the request distribution mechanism that the service process has resumed performing tasks for the system. In some embodiments, a gossip protocol communicating availability information to different nodes of a distributed system may provide information to schedule storage reclamation operations at different compute nodes of the distributed system.

Growing community of inventors

Seattle, WA, United States of America

John Michael Morkel