Emc Corporation

Qi Bao

Feng Zhang

Qin Xin

Bo Huang

Qiang Zou

Martin Wang

Peter Chen

A data structure comprising two or more sub data structures representing a given data set is maintained. Each of the two or more sub data structures comprises an array of bit positions and has a set of hash functions associated therewith. Each of the hash functions is operable to map an element of the given data set to at least one of the bit positions of the array. One of the two or more sub data structures is recognized as a master sub data structure and the others of the two or more sub data structures as slave sub data structures. Insertion and deletion of elements in the data structure is based on the recognition of each of the two or more sub data structures as the master sub data structure or one of the slave sub data structures.

Rolling bloom filter for data with retention policy

An improved data storage technique achieves a desired level of reliability by providing sufficient redundancy in erasure coded data to maintain the data, without repair, for a prescribed period of time. The improved technique employs a newly devised, continuous-time Markov chain model. The model can be applied in computerized systems to establish erasure coding parameters for storing and reliably maintaining data for a designated period of time, without any need to repair the data to reestablish an original or previous level of erasure coding redundancy.

Selection of erasure code parameters for no data repair

An improved technique for storing and processing video data includes separating video data derived from a video camera into multiple video fragments, distributing the video fragments to different computing nodes of a network, and performing parallel image processing on the video fragments in a distributed manner, by the respective computing nodes that receive the video data. Examples of image processing include pattern matching, such as facial recognition, and license plate matching.

Technique for parallel, distributed video processing

A technique performs policy-based intelligent data placement in an electronic environment. The technique involves selecting, from a pool of candidate local area networks (LANs) of the electronic environment, a plurality of LANs within which to store the data based on a set of policy priority levels assigned to the data. The technique further involves generating a set of information elements (e.g., data fragments, erasure codes, etc.) from the data, and placing the set of information elements on storage nodes of the plurality of LANs. Such a method enables the data to be stored in a distributed manner and alleviates the need for a central storage facility. Since the data is distributed among the storage nodes of the plurality of LANs, system capacity and infrastructure is able to grow (i.e., scale) in a manner which does not create a problematic bottleneck.

Policy based intelligent data placement

An improved technique for distributed data storage employs multiple DHT (distributed hash table) rings provided at different levels of a network hierarchy. A computing node is identified for data storage by performing multiple hashing operations, one for each DHT ring. The hashing operations for the different rings are distinct in that they are performed using different hashing functions and/or are performed on different data sources. In one example, a 2-level DHT is formed from a group of LANs forming a first DHT ring and a group of computing nodes, within each LAN, forming a second DHT ring.

Multi-level distributed hash table for data storage in a hierarchically arranged network

An overlay network has a hierarchical cluster tree (HCT) structure on an underlying physical network, the HCT structure having clusters of local area networks (LANs) associated with respective sub-spaces of a hash key space. A cluster proxy node (CP) maintains a routing table and channels to both a parent cluster and member LANs, with routing information associating each LAN with a physical-network address (PA) of a corresponding LAN representative node (LR). Nodes sending routing request messages each containing a key value to the CP node, which returns routing response messages each including PA information from the routing table and/or the parent cluster. The PA information in the response for a given request includes a PA of an LR of a request-specific LAN whose sub-space contains the key value in the request message. A node uses the PA information to send a message to the LR node using the physical network.

Network-assisted routing for topology-aware overlay networks

A Hierarchical Cluster Tree (HCT) overlay network reflects underlying physical network topology including inter-node distances (e.g., hop count), and an HCT structure groups nodes based on distance measurements. Local area networks (LANs) are basic building blocks. The network includes first and second level-1 clusters of LANs, each LAN including computerized data processing and/or storage nodes. A distance between LANs in the level-1 clusters is less than or equal to a predefined level-1 maximum distance. The LANs of the first and second level-1 clusters form a level-2 cluster of LANs in which a distance between cluster medoid nodes of the level-1 clusters is less than or equal to a predefined level-2 maximum distance greater than the level-1 maximum distance. Clustering can be extended to higher levels having successively greater maximum distances.

Growing community of inventors

Boxborough, MA, United States of America

Qi Bao