Emc IP Holding Company LLC

Yohannes Altaye

Mikhail Danilov

Data retention for a tiered ordered event stream (OES) data storage system is disclosed. Data retention can be applied to demoted data, e.g., data demoted from a first tier to a second tier of data storage devices. The data retention disclosed can avoid demoted events, generally aggregated via a chunk object, file object, etc., from being subject to both a stream-cut operation and a retention period. In embodiments, where a stream-cut instance corresponds to events occurring later in time than a retention end time (RET), all demoted events, e.g., as aggregated by a chunk, file, etc., can be affiliated with the RET. However, where the stream-cut instance corresponds to events occurring earlier in time than the RET, then a portion of the events, e.g., those that will be truncated by performing the stream-cut instance, can be demoted and affiliated with the RET.

Demoted data retention via a tiered ordered event stream data storage system

Scaling an ordered event stream (OES) based on an information-unit (IU) metric is disclosed. The IU metric can correspond to an amount of computing resources that can be consumed to access information embodied in event data of an event of the OES. In this regard, the amount of computing resources to access the data of the stream event itself can be distinct from an amount of computing resources employed to access information embodied in the data. As such, where an external application, e.g., a reader, a writer, etc., can connect to an OES data storage system, enabling the OES to be scaled in response to burdening of computing resources accessing event information, rather than merely event data, can aid in preservation of an ordering of events accessed from the OES.

Information-unit based scaling of an ordered event stream

Storage of data according to mapped reliability groups (MRGs) of mapped redundant arrays of independent nodes (mapped clusters) is disclosed. Real reliability groups (RRGs) of a real cluster can provide compartmentalization that can reduce a probability of a data loss event occurring in a cluster storage system. MRGs for a mapped cluster employing the RRGs of the real cluster can provide additional compartmentalization that can further reduce a probability of a data loss event occurring. A data protection set can be stored according to a first group of MRGs and a redundant representation of the data protection set can be stored according to a second group of MRGs, wherein a network distance between the first and second group of MRGs is greater than a network distance between constituent MRGs of either the first or second group of MRGs.

Mapped redundant array of independent nodes employing mapped reliability groups for data storage

The described technology is generally directed towards fine-grained data event expiration in a streaming data storage system. An event to append is given an expiration period, and the expiration time for the events in a data stream or segment of a data stream is the largest expiration time among events in the data stream or segment. Different segments can have different expiration times for their events. In a segment comprising a group of events, a subgroup of expired events prior to a stream cut are deleted by an expiration task. For a subgroup of unexpired events prior to a stream cut, the expiration task retains (does not delete) the subgroup of events. If a scaling operation is performed on a segment, the new successor segment or segments inherit the largest expiration time of the predecessor segment or segments.

Data expiration for stream storages

A geographically diverse data storage system that can protect data via replication of data among relevant zones according to a determined replication topology is disclosed. The replication topology can be determined based on replication times between the relevant zones. In an aspect, a tree topology can provide advantages over a star topography. In an embodiment, a tree topology can be generated, or an existing topology can be modified, via selection of a next replication task(s) based on the replication times. In an aspect, the replication times can be determined from measurable characteristics of the geographically diverse data storage system. In some embodiments, the replications times can be based on historical measurements, time limited historical measurements, inferences from machine learning, etc. A determined topology can be ranked relative to other viable topologies based on criteria such as speed, monetary cost, computing resource usage, etc. Accordingly, a selected topology, or selected modification to a topology, can provide for improved replication that can provide protection for data stored in the geographically diverse data storage system.

Geographically diverse data storage system employing a replication tree

The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams. Alternatively, an application writer can request that an event be committed to multiple writer-specified data streams via a cross-stream micro-transaction, in which the streaming data storage system obtains the appropriate segment store instances and segments for each data stream, and coordinates the direct appending of the event to the appropriate segments.

Cross-stream transactions in a streaming data storage system

Framed event access in an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES and can have an inherent write sequence. Segments can be parallel segments. Reading events from parallel segments can result in a read sequence that does not match the write sequence. This mismatch can be more severe as segment length increases, as event density disparities increase, as access times diverge for different segments, or for numerous other reasons. Event framing can compartmentalize divergence between the write and read sequence. In an aspect, readers in the several segments of the OES can be constrained to read within a frame defined by frame boundaries until all readers have reached the frame boundary, then can advance to a next frame. The restriction can act as a pseudo-synchronization of readers that can mitigate difference between write and read sequences.

Framed event access in an ordered event stream storage system

The described technology is generally directed towards replicating metadata representing a virtual data structure corresponding to replicated legacy data instead of the actual data for the data structure. Once virtual chunks are replicated to a remote, newer storage system, the corresponding legacy data is locally read into the virtual chunks to transform the virtual chunks into real data chunks of the remote newer storage system. A checksum can be replicated for the remote newer storage system to evaluate the consistency of the data. Efficient data storage migration is thus accomplished in a replicated environment based on relatively negligible replication traffic between two remote locations, while still assuring the consistency of migrated data.

Data storage migration in replicated environment

Disk access event control for mapped nodes of a cluster storage system supporting a redundant array of independent nodes (mapped RAIN) system is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters. In an embodiment, disk access events can be routed via a storage service instance supporting a mapped node. In another embodiment, disk access events can be routed via another storage service instance that does not support the mapped node. Routing the disk access event via another storage service instance that does not support the mapped node can reduce the use of computing resources. Further, the routing of the disk access event can be according to a proportional disk operation value determined based on historical disk access event characteristics.

Disk access event control for mapped nodes supported by a real cluster storage system

Mitigating the effects of a real node failure in a doubly mapped redundant array of independent nodes, e.g., doubly mapped cluster is disclosed. In response to a change in an accessibility to data stored on an extent of a real storage device of a real node of a real cluster, wherein the extent of the real storage device corresponds to a portion of a mapped storage device of a mapped node of a doubly mapped cluster, substituting a reserved extent of a real storage device for the extent of the real storage device. The substituting the reserved extent of the real storage device can correspond to a change in a topology of the doubly mapped cluster, wherein the change in the topology comprises replacing the portion of the mapped storage device with a substitute portion of a mapped storage device that corresponds to the replacement extent of the real storage device. The changed topology can enable writing of data to the substituted portion of a mapped storage device that can cause writing of corresponding data to the reserved extent of the real storage device.

Mitigating real node failure of a doubly mapped redundant array of independent nodes

A mapped redundant array of independent nodes (mapped RAIN) for data storage is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters, wherein the real clusters can comprise storage devices of different storage capacities. Mapping of data storage locations for a mapped RAIN cluster to real storage devices can be based on an affinity value determined for pairs of real nodes of the real data clusters. A normalized affinity can be employed to enable allocation of real storage to mapped nodes of mapped clusters that can be based on the heterogeneous capacities of the storage devices. This can provide improved data availability and data recovery over other techniques where heterogeneity of hardware can make efficient resource allocation a non-trivial task. The disclosed subject matter can facilitate more efficient allocation of Mapped RAINs in a heterogeneous cluster storage construct.

Heterogeneous mapped redundant array of independent nodes for data storage

Selective instantiation of a mapped cluster storage service (MCSS) is disclosed for storage system supporting a doubly mapped redundant array of independent nodes environment. An instance of a MCSS can be run on computing resources of the storage system supporting a doubly mapped redundant array of independent nodes environment, wherein a real node of a real cluster comprises a real storage device that comprises real extents, and wherein extents of the storage system map to a mapped node of a mapped cluster. The instance of the MCSS can be run near to the highest number of real extents participating in a mapped node. This can reduce computing resource consumption associated with running the MCSS farther from the more concentrated real extents.

Selective instantiation of a storage service for a doubly mapped redundant array of independent nodes

Watermarking of an ordered event stream (OES) is disclosed. Writing events to an OES can advance populating of the OES, which can be correspond to progress values. Progress values can enable a reader to know where it is in an OES. A reported progress value can be determined from the progress values based on an event being committed to the OES. Moreover, an effective progress value can be determined from the progress values and monitoring the OES for events with that defer being committed to the OES. A watermark for the OES can be determined from the effective progress value such that reading beyond the effective progress value is restricted. This can provide confidence that the reader has read out properly committed events and has not omitted any events having deferred commitment.

Watermarking of events of an ordered event stream

Access control for an ordered event stream (OES) storage system is disclosed. Access to a portion of an OES can be controlled at a key-level in relation to a key space of the OES. An application instance can be identified to enable determining a correspondence to one or more keys. The correspondence can be embodied in stored data, for example, via an advanced access control list (AACL) that can be in the form of a list, a table, etc. Application instance access to the portion of the OES can be controlled by determining if an access rule is satisfied, e.g., determining if the key space the application instance wants to access comports with the one or more keys corresponding to the application instance identity. In an aspect, screening data corresponding to the AACL can enable preliminary access screening external to the OES storage system.

Access control for an ordered event stream storage system

The described technology is generally directed towards a streaming data storage system that compacts older stream data by aggregating that data based on an aggregation function, and replacing the older data with the aggregated data. The technology works with a data stream in different storage tiers (e.g., Tier-1 and Tier-2), provides consistent data stream compaction of data streams having multiple segments, works with different reader applications at different event positons in the data stream, and allows the use of multiple, different aggregation functions. The technology facilitates more efficient compaction for a data stream with a lesser number of routing keys by maintaining per-chunk aggregated data for such data streams.

Growing community of inventors

Dumfries, VA, United States of America

Yohannes Altaye