Emc IP Holding Company LLC

Ramprasad N Chinthekindi

Abhinav Duggal

Srikanth Srinivasan

Kalidas Balakrishnan

Nitin Madan

Lan Bai

Shyam Burkule

Philip N Shilane

Kalyan Chakravarthy Gunda

Subrahmanyam Josyula

Ashwani Mujoo

Rahul Goyal

Neeraj Bhutani

George Mathew

Mahesh Kamat

Tony T Wong

Sandeep Nirmale

Smriti Thakkar

Fani Jenkins

Gobikrishnan Sundharraj

Shah Veeral

Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

Processing out of order writes in a log structured file system for improved garbage collection

Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments, the dead segments are deleted by copying live segments into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices that can be run as needed in the cloud.

Garbage collection for a deduplicated cloud tier with encrypted segments

Data segments and metadata segments to be stored in a storage system, where the data segments are deduplicated segments and each of the metadata segments includes a fingerprint for the corresponding data segment, for each of the metadata segments. It is determined that the metadata segment contains one or markers inserted by a client device of the storage system. The metadata segment is examined to determine whether the metadata segment satisfies a predetermined condition. In response to determining that the metadata satisfies the predetermined condition, the metadata segment is compressed using a predetermined compression algorithm. The compressed metadata segment is stored in the storage system, otherwise the metadata segment is stored in the storage system without compression. Thereafter, the data segments are stored in the storage system.

Intelligent metadata compression

A microservice or serverless process consistency check process comprising locating all the necessary metadata and data objects in the cloud by storing the data objects in the cloud and synchronously mirroring the metadata, which is separately stored in local storage, to the cloud. The process generates a list of data objects in the cloud as 'Set A' and the list of metadata objects in the same prefix range as the data objects as 'Set B.' The consistency check then verifies whether all objects in Set A are referred to by objects in set B. In the case where there are gaps between the sets, non-existent objects are marked as missing, and unreferenced objects are marked as orphan objects. The list of missing and orphan objects is then sent back to the backup server for analysis and further processing.

System and method for consistency checks in cloud object stores using microservices

Embodiments are directed to a cost-aware object selection for cloud garbage collection that deletes completely dead objects and also selects low-live objects up to a carefully selected liveness threshold value. This threshold is dynamically chosen per cloud garbage collection cycle by balancing costs including egress, input/output operations (IOPs), storage cost of cleaning partial live objects, and the storage cost incurred by leaving behind dead data by not cleaning the object. The threshold value is dynamically calculated to accommodate different cost models for different cloud providers and also caters to different costs for different storage tiers.

Cost-aware garbage collection for cloud storage

System generates data structure based on unique identifiers of objects in storages and sets indicators in positions corresponding to hashes of unique identifiers of objects. The system copies active objects from one storage to another, if number of active objects in storage does not satisfy threshold, and resets indicators in positions in data structure corresponding to hashes of unique identifiers of active objects copied to the other storage. The system generates another data structure based on unique identifiers created while generating data structure, positions in other data structure corresponding to hashes of the unique identifiers. System sets indicators in positions in the other data structure corresponding to hashes of unique identifiers of data objects in active storages while generating data structure. System resets indicators in positions in data structure corresponding to hashes of the unique identifiers corresponding to indicators set in positions of the other data structure.

Garbage collection integrated with physical file verification

Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments or dead and live regions such as compression regions, the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a serverless approach.

Garbage collection for a deduplicated cloud tier using functions

Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live regions such as compression regions, the dead segments in the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices based approach.

Garbage collection for a deduplicated cloud tier using microservices

An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.

Non-disruptive and efficient migration of data across cloud providers

Embodiments of method and system are directed to a best effort method to reconstruct the corrupt files from available nodes or tiers and heal or repair the corrupt files. The method looks for the existence of fingerprints of the lost segments in available tiers on the nodes or across nodes within a cluster. The method discovers the copies of these missing segments, creates an Lp tree referring these discovered segments and creates a virtual file in the namespace. The file is then moved to the destination tier where the corruption exists, thus healing the corruption.

Reconstructing lost data objects by generating virtual user files from available tiers within a node

A garbage collection assisted deduplication process determines whether or not data segments should be deduplicated or not based on the liveness of segment data in a region, and the number of segments subject to deduplication in the region. Ingested data is divided into a plurality of segments, and a fingerprint is calculated for each segment. An index table entry maps a fingerprint to a region and container ID, and a perfect hash vector is setup for this mapping. A percentage of live segments in the region relative to a liveness threshold is determined, as is a number of segments in the region subject to deduplication relative to a deduplication threshold. If a region is sufficiently live, deduplication is performed, but if the region is dead, deduplication is not performed. For a live region, if the number of deduplicated segments is too low, deduplication is not performed.

Garbage collection assisted deduplication in a log structured file system

A method of migrating or consolidating cloud data includes generating a container on a cloud platform and receiving, at the container, source metadata identifying a set of data to be migrated from a source cloud bucket associated with a source data domain to a destination cloud bucket associated with a destination data domain. The method further includes copying, by the container, set of data from the source cloud bucket to the destination cloud bucket based on the source metadata and generating, by the container, destination metadata for the set of data as stored at the destination cloud bucket.

Consolidation and migration of cloud data

Embodiments of method and system are directed to a best effort method to reconstruct corrupt files from available nodes or tiers and heal or repair the corrupt files within a cluster of nodes. The method looks for the existence of fingerprints of the lost segments in available tiers on the nodes or across nodes within a cluster. The method discovers the copies of these missing segments, creates an Lp tree referring these discovered segments and creates a virtual file in the namespace. The file is then moved to the destination tier where the corruption exists, thus healing the corruption.

Growing community of inventors

Cupertino, CA, United States of America

Ramprasad N Chinthekindi