Dell Products Inc.

Quest Software Inc.

Cisco Technology, Inc.

Druva Inc.

Murali Bashyam

Goutham Rao

Abhijit Dinkar

Vinod Jayaraman

Eric Brueggemann

Carter George

Mark Taylor

Abhijit Patra

Michael E Root

Tarun Tripathy

Naresh Saladi

Sreekanth Garigala

Norman William Finn

Nagaraj Bagepalli

David Wei-Shen Chang

Parthiban Munusamy

Mahesh Jethanandani

Hui Huang

Micah Wilson

Techniques and mechanisms described herein provide for global deduplication in a cloud-based storage system. According to various embodiments, a global segment reference map can be created for data segments when a data segment has not been previously added to the global segment reference map. For each data segment not added to the global segment reference map, those data segments can be deleted from a cloud storage location.

Global deduplication in a cloud-based storage system

Techniques and mechanisms described herein provide for verification of data across cloud-based and on-premises data storage systems. According to various embodiments, a backup client implemented on a first compute node can store a data file in a backup data repository. A data guardianship can store first data file state information describing the data file in a key-value store accessible via the internet. A data verification instance can analyze the backup data repository to verify that the data file is stored intact in the backup data repository.

Data guardianship in a cloud-based data storage system

Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Data block migration

Techniques and mechanisms described herein provide for facilitating communications between one or more client machines and one or more cloud storage providers. According to various embodiments, a virtual machine may communicate with one or more client machines via a standard data storage protocol. The virtual machine may also communicate with one or more cloud storage providers via the internet. The virtual machine may then perform operations such as receiving data from a client machine and storing it to the cloud, retrieving data from the cloud and providing it to the client machine, querying data stored in the cloud, reporting on and verifying data stored in the cloud, and transferring and/or transforming data stored in the cloud.

Data management across cloud storage providers

Techniques and mechanisms described herein provide for verifying data across cloud-based and on-premises data storage systems. According to various embodiments, data can be received by a gateway from a client machine and stored in a file data repository accessible via the internet. The stored data can have a common master namespace. Indications of updates to the master namespace can be received. Updated namespace data and metadata can be transmitted to the gateway.

Global namespace in a cloud-based data storage system

In general, certain embodiments of the present disclosure provide techniques or mechanisms for creating efficient and consistent replication of file overwrites. According to various embodiments, a file map with a sequence file number is created for a data file on a source. The file map includes one or more nodes that each include a sequence node number. When an entry in the file map is changed, the sequence file number is incremented, and the corresponding node is updated by, among other things, setting the sequence node number to equal the current sequence file number. All other nodes in the file map that point to the updated node are similarly updated. When replication occurs, only the updated nodes in the file map are replicated and any unreplicated non-updated nodes are maintained.

Optimized and consistent replication of file overwrites

Techniques and mechanisms described herein facilitate the strengthening of a data fingerprint. According to various embodiments, a data stream may be parsed into a plurality of data chunks. A hash function may be applied to each of the data chunks to determine a respective data chunk hash value. A respective data chunk fingerprint for each of the plurality of data chunks may be transmitted to a remote server. Each data chunk fingerprint may include the respective data chunk hash value and one or more respective intermediate hash values.

Data fingerprint strengthening

Techniques and mechanisms described herein facilitate the encryption of content using content-based encryption keys. According to various embodiments, data stream may include one or more data chunks. A client machine may apply a hash function to a data chunk to determine a fingerprint value. A cryptographic protocol shared with a remote server may be applied to the fingerprint value to determine a data chunk encryption key. The data chunk encryption key may be used to encrypt the data chunk, and the encrypted data chunk may be sent to the remote server for storage.

Content-based encryption keys

Systems, methods, and devices are disclosed herein for data deduplication and storage. Methods include identifying first criteria associated with a data block included in a plurality of data blocks, the data block being identified based on a blockmap and an index, the blockmap characterizing a file stored in a deduplication repository, the index identifying data store identifiers, reference count data, and second criteria associated with the plurality of data blocks. The second criteria include the first criteria, and the first criteria characterize input/output (I/O) activity associated with the data block. The methods may also include identifying data stores based on the first criteria, the data stores being included in data storage tiers. The data stores are identified based on the I/O activity associated with the data block. The methods further include updating the first criteria stored in the index and associated with the data block.

Multi-tiered data storage in a deduplication system

Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Transferring differences between chunks during replication

A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

File aware block level deduplication

Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.

Heat indices for file systems and block storage

Techniques and mechanisms described herein facilitate the performance of duplicate data block instruction identification. According to various embodiments, a data block update operation message may be received at a communications interface in a secondary storage node. The secondary storage node may be configured to store secondary data mirroring primary data stored on a primary storage node. The primary data and the secondary data may each include a respective plurality of data blocks. The data block update operation message may include a data block update instruction for updating a designated one of the plurality of secondary storage node data blocks. The data block update operation message may include a primary storage node data block sequence number designating an update operation status. When it is determined that the data block update instruction is not a duplicate, the data block update instruction may be performed.

Growing community of inventors

Fremont, CA, United States of America

Murali Bashyam