Emc IP Holding Company LLC

Igor Medvedev

Mikhail Danilov

Qi Zhang

Shashwat Srivastav

Andrei Paduroiu

Lu Lei

Sriram Sankaran

Jun Luo

Gregory Skripko

Huapeng Yuan

Andrew Robertson

Peter Musial

Julius Jian Zhu

Jie Song

Liang Mei

Brett A Menzies

The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Retention management for data streams

Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Employing triggered retention in an ordered event stream storage system

Described herein, system that facilitates replication of data in a geographically distributed storage environment. According to an embodiment, a system can comprise determining whether a data storage system is in an active data storage mode or a passive data storage mode. In active data storage mode, the data storage system can accept connections from an application, permit the application to store new data objects, replicate the new data objects to other data storage systems, and accept replicated data. In passive data storage mode, the data storage system can block storage of new data objects by the application while accepting replicated data.

Data replication using active and passive data storage modes

Commutative coding in a geographically diverse data storage system is disclosed. Commutative coding can achieve a same result as more conventional hierarchical erasure coding of data, but can be more efficient. Commutative coding can employ Galois Field (GF) based bit-matrix operations. The bit-matrix operations can employ a reduced GF order in associated with expanding elements of input matrixes. A reduced GF order can perform matrix operations at a lower complexity, e.g., employing AND operations for a GF(2) in contrast to XOR operations for a GF(2), where w=4, 8, 16, etc. In an aspect, commutative coding can comprise generating a second-tier coding fragment based on applying a second erasure coding scheme, via bit-matrix operations, to a first-tier encoded fragment, wherein the first-tier encoded fragment is based on an input data fragment and a first erasure coding scheme.

Data protection via commutative erasure coding in a geographically diverse data storage system

The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Consistent data stream replication and reconstruction in a streaming data storage platform

Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exactly-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.

Ordered event stream event retention

The described technology is generally directed towards a stream browser that provides a human-readable (e.g., hierarchical) view of event data for a data stream of a data streaming storage service. A stream browser associated with a data stream maintains routing key and event data for the data stream, e.g., in a primary index. The routing keys can correspond to container structures, e.g., folders and subfolders, with the events associated with each routing key corresponding to hierarchical children of the routing key parent container structures (e.g., like files). A secondary index can be used to evict the oldest events from the primary index to limit the number of data stream events returnable by the stream browser to a practical number.

Stream browser for data streams

Stream retention in a data storage system is described herein. A method as described herein can include receiving, by a device operatively coupled to a processor, a hold for a portion of a data stream, wherein the data stream comprises respective events indexed by time, and wherein the portion of the data stream is bounded by a first event of the data stream that is associated with a first time and a second event of the data stream that is associated with a second time; generating, by the device, a synthetic stream comprising the portion of the data stream; and preventing, by the device, events of the data stream that are included in the synthetic stream from alteration.

Stream retention in a data storage system

The described technology is generally directed towards reference tracking garbage collection, including for operating in a geographically distributed storage system. Content Addressed Storage (CAS) data objects comprising BLOBs (Binary Large Objects) are referenced by data objects comprising C-Clip Descriptor Files (CDFs). BLOBs having no remaining reference, referred to as orphan BLOBs, are deleted in a two-stage deletion operation that ensures that no new reference has been created. An alive BLOB that became an orphan is demoted to a garbage collection candidate with limited access to the BLOB. The BLOB is only deleted if the garbage collection candidate status of the BLOB is distributed across geographically distributed zones, and there is no new reference to the BLOB.

Reference tracking garbage collection for geographically distributed storage system

A computer-executable method, computer program product and system for managing metadata in a distributed data storage system, wherein the distributed data storage system includes a first node and one or more data storage arrays, the computer-executable method, computer program product and system comprising partitioning management of metadata created in the distributed data storage system into one or more portions of metadata, wherein the first node manages a first portion of the one or more portions of metadata, and storing the metadata using the first node.

Growing community of inventors

Seattle, WA, United States of America

Igor Medvedev