Emc IP Holding Company LLC

Dell Products Inc.

Thomas Kaitchuck

Andrei Paduroiu

Flavio Junqueira

Raúl Gracia-Tinedo

Yi Liu

Srikanth Satyanarayana

A method for managing a data stream (DS) includes: receiving a first request from an administrator; obtaining the DS from a streaming storage system (SSS); performing a first analysis of the DS to extract relevant data; making a first determination that the relevant data is worth indexing; generating a set of indexes corresponding at least a portion of the DS; receiving a query about a data item in the DS from a user; sending a second request to a database engine comprising the query; performing a second analysis of the query; making a second determination that a re-indexing of the portion of the DS is not required; identifying an index from the set of indexes that is associated with the query; sending position information of the data item to an API service; retrieving the data item from the SSS; and initiating displaying of the data item to the user.

Method and system for content-based indexing of data streams in streaming storage systems

A first user-initiated data write request including stream data is received from an edge device executing in a first zone. The stream data is partitioned into at least one segment and an event included in the stream data is written into the at least one segment based on an event's routing key. A segment container of a segment store hosts the at least one segment. The stream data is transmitted to a tier-1 storage of the first zone in order to provide a low-latency access to the stream data for the first user. Thereafter, the stream data is moved to a tier-2 storage of a second zone for permanent storage of the stream data. When requested by a second user operating in the second zone, a read-only version of the segment container retrieves the stream data from the tier-2 storage.

Method and system for addressing zone-awareness in a heterogeneous environment

Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.

Executing streaming data writes without duplication or loss

Implementations are provided herein for encoding and transmitting streaming data from a client application to a server for storage without prior knowledge of the size of the streaming data. A header can be sent that includes a batch size chunk size. Raw streaming data can be packaged into the chunk. Chunks can be packaged and sent at any time prior to filling up with streaming data, by padding the chunk and including a footer that delineates the amount of raw stream data in the chunk. Chunks that are full can have a footer that delineates the entire chunk is raw stream data. It can be appreciated that you do not need to buffer data on the client side as chunks do not need to be full to send. Latency on processing streaming data can also be reduced by limited or eliminated buffering.

Encoding and transmitting stream data without prior knowledge of data size

Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.

Atomically committing related streaming data across multiple distributed resources

Embodiments enable clients to have read-write access to a stream where all other clients are aware of all changes being made to the stream. A pravega node can track the data that has been written to the stream. The clients can dynamically read the stream. A client among the clients can update the stream by sending a request to the pravega node that includes the update and a total length of the stream that was written to the stream at the time of the last read update by the client. If the total length of the stream received from the client matches the actual length of the stream maintained by the pravega node, the pravega node will update the stream. If not, a failure message can be sent to the client and the client can process more reads to the stream before making another attempt to update the stream.

Growing community of inventors

Portland, OR, United States of America

Thomas Kaitchuck