Emc IP Holding Company LLC

Amazon Technologies, Inc.

Andrei Paduroiu

Mikhail Danilov

Srikanth Satyanarayana

Maksim Vazhenin

Thomas Kaitchuck

Yaron Burd

Yan Yan

Flavio Junqueira

Igor Medvedev

Shivesh Ranjan

Tom Kaitchuck

A distributed database identifies classifications of risk associated with stages of a query plan. The distributed database generates an execution plan in which incompatible risk classifications are assigned to separate stages of an execution plan that is derived from the query plan. The stages are assigned to computing nodes for execution based, at least in part, on the risk classifications. A result for the query is generated based on execution of the stages on the assigned computing nodes.

Secure query processing

A distributed database keeps an abstract syntax tree (AST) and an included user code from interacting with a query engine coordinator during the performance of query execution processes. The query engine coordinator is separated into a frontend and backend, where a compiler on the frontend generates an AST and serializes it to be sent to the backend. The backend is implemented in a security environment separate from the front end. The backend deserializes the AST, sanitizes it, and generates a query plan based on the sanitized AST.

A distributed database keeps user-defined functions separate from a query engine by using a frontend. The frontend allows a user-defined function to interact with a proxy application processing interface (API) that is based on an API of the query engine. The frontend sends serialized data to the query engine in order to interact with the API of the query engine. The user-defined function is executed in security environments separate from the frontend and the query engine.

The described technology is generally directed towards fine-grained data expiration to reduce storage capacity used in a streaming data storage system. Segments of events are scanned, with each event's payload evaluated to determine whether that event is expired or unexpired. For example, for a log event, an expiration period can be determined based on the event creation time and the event logging level, both of which are typically within the log event's payload. Unexpired events are copied to a new destination segment that replaces the original, scanned source segment; the expired events are deleted when the source segment is deleted, reclaiming storage capacity used by the expired events. Also described is maintaining and using segment and chunk metadata to bypass individual event processing when the metadata indicates that the chunk contains only expired events and can be safely deleted.

Log data management in a streaming data storage system

The described technology is generally directed towards fine-grained data retention management in stream-based data storage systems, which facilitates specifying retention policy at the event level. A fixed retention period corresponding to a retention end time is associated with each event, and the system keeps the longest of the retention end times for each segment of a data stream. Stream cuts are generated which divide the data stream into fragments, and are the basis for truncating the data stream. Each stream cut contains a stream cut retention end time obtained from the longest retention end time of the retention end times of the active segments. When an attempt to truncate a data stream from a stream cut is made, the attempt is blocked if the stream cut's retention end time is in the future, otherwise truncation from the stream cut point is allowed to proceed.

Event level retention management for data streams

Processing of continuously generated data using a rolling transaction procedure is described. For instance, a system can process a data stream comprising a first segment and a second segment. A transaction associated with the data stream can be initiated and in response to the transaction being initiated, a first transaction segment for the first segment and a second transaction segment for the second segment are generated. Further, a scaling event that modifies the second segment into a third segment and a fourth segment can be detected, and a data stream transaction procedure is executed to end the transaction.

Processing continuously generated data using a rolling transaction procedure

The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an ordered associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index. The table keys are additionally organized in a sorted data structure which is stored within the table segment along with other keys.

Storing an ordered associative array of pairs using an append-only storage medium

Data verification technology for ordered event stream (OES) events written into an ordered event stream storage system is disclosed. The verification technology provides perfect reliability. The verification technology further requires low storage overhead in comparison to typical checksums, storing replicated data, etc. Test event data can be generated in a reproducible manner based upon determined OES metadata. OES metadata can be determined from input received via a user interface, via characteristics of an OES storage system, etc., and can be stored for later use in data verification. The test event data can be stored to a portion of an OES storage system under test. The stored test event data can subsequently be verified by using the stored OES metadata to regenerate test event data for comparison to the stored test event data. The test event ordering can be verified via sequence information included in the stored test event data.

Test data verification for an ordered event stream storage system

The described technology is generally directed towards a streaming data storage system that can switch between a tiered mode of operation in which events are written to Tier-1 storage and later migrated to Tier-2 storage, and a direct mode of operation in which events are written to Tier-2 storage, bypassing the tiered mode. The switching from tiered mode to direct mode, and from direct mode to tiered mode, can be automatic and based on user configuration information. For example, an event size metric (e.g., average event size) can be evaluated against user defined thresholds to determine which mode to use. If the average event size goes below a low threshold value, the tiered mode is switched to and used for appending events to a segment of a data stream. If the average event size goes above a high threshold value, the direct mode is switched to and used.

Bypassing of tier-1 storage in a streaming data storage system

The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index.

Storing a non-ordered associative array of pairs using an append-only storage medium

The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Consistent data stream replication and reconstruction in a streaming data storage platform

Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.

Executing streaming data writes without duplication or loss

Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.

Migration of legacy data into an ordered event stream

The technology describes forking (dividing) a data stream into multiple data streams comprising appending a first new data stream and a second new data stream to the original data stream. After forking, the two new data streams have different sets of stream fragments, with one or more of the older stream fragments common to both sets. Data events before the forking operation are read from the original data stream, while data events appended after the forking operation are appended to and read from only one of the data streams, based on which stream each writer/reader is registered to. Also described is truncating the original data stream based on one stream identifier, while leaving the original data intact for another stream identifier. Only when no stream needs a portion of the original data (resulting from separate truncation operations) is the data portion deleted.

Forking data streams in a streaming data storage platform

The described technology is generally directed towards a streaming data storage system that maintains streamed events, including in a manner that facilitates the ordering of those events based on external ordering (position) data. An event is appended to an event segment, and an index segment maintains offset information to the event relative to the offsets of other previously appended events. The index segment is ordered by position data provided with the event, such that the events (e.g., for a given event routing key) can be read back in position order instead of append order. An ongoing read that returns an event (that matches ongoing read criteria) as the event arrives can also be handled. Also described is the use of a tail index and an indexing queue for more efficient handling of events prior to writing the offset information into the index segment.

Growing community of inventors

Bellevue, WA, United States of America

Andrei Paduroiu