Emc IP Holding Company LLC

Emc Corporation

Dell Products Inc.

Philip N Shilane

Grant R Wallace

Frederick Douglis

Abhinav Duggal

Joseph S Brandt

Kimberly R Lu

George Mathew

Tipper Truong

Atul Avinash Karmarkar

Nicholas A Noto

Mark L Huang

Hyong Shim

Cheng Li

Sorin Faibish

Kevin Xu

Ming Benjamin Zhu

Stephen Smaldone

Nitin Garg

Mariah Arevalo

Philippe Armangau

Vamsi K Vankamamidi

Kai Yi Li

R Hugo Patterson, Iii

Edward Kihyen Lee

Istvan Gonczi

Ivan Basov

Chegu Vinod

Venkata Ravi Chandra Bandlamudi

Windsor Hsu

Shuang Liang

Assaf Natanzon

Leehod Baruch

Fabiano C Botelho

Ramprasad N Chinthekindi

Jianqiang Luo

Yamini Allu

Guanlin Lu

Vishal Tiwary

Radia Joy Perlman

Mahesh Kamat

Kadir Ozdemir

Dhanabal Ekambaram

Xing Lin

Yijian Wang

Hao Tong

Metadata of a file system is maintained in a key value store. The metadata includes a namespace represented as a B+ tree having pages written to the key value store. Identifiers to the pages include a snapshot identifier and a page number. The identifiers are stored as keys in the key value store. The snapshot identifiers are used to track pages that are shared between first and second snapshots. Upon a request to write to an existing page, a determination is made from the snapshot identifiers as to whether the existing page belongs to the first or second snapshot. If the existing page belongs to the second snapshot, the write to the existing page is allowed. If the existing page belongs to the first snapshot, a copy on write (COW) is performed to generate a new page for the write.

Storing namespace metadata in a key value store to facilitate space efficient point in time snapshots

A distributed and deduplicating file system that stores backup data to object storage is described. A node, in a cluster of nodes, receives a write operation to back up a data file, and identifies backup data associated with the write operation. The node identifies deduplicated data by deduplicating the backup data. The cluster stores the deduplicated data to each of the buffers which corresponds to a node in the cluster. If the node determines that the size of information, which includes the deduplicated data, stored to each of the buffers has accumulated to become greater than or equal to an object threshold, then the node writes the information, which includes the deduplicated data, stored in each of the buffers to an object storage which stores data for the nodes in the cluster.

Distributed and deduplicating file system for storing backup data to object storage

A process of recovering from a failure of a solid-state device (SSD) storing a persistent volume (PV) for logging in a cluster network. The PV is recreated on a different SSD that has spare capacity and IOPS resources, and log information is redirected from an application to the relocated PV on the different SSD. Upon replacement of the original failed SSD, the PV can be copied from the different SSD to the new replacement SSD, or the replacement SSD can be left empty, and the relocated PV on the different SSD can continue to be used. This process provides a high degree of redundancy with regard to logging so that a failure of single SSD does not bring down the node or the entire cluster network.

Failure recovery of solid state disk (SSD) storage for cluster file system serviceability

A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs and to debug and perform other serviceability tasks provided by a vendor of the cluster system. Logs are collected from all components of the system and aggregated using a central persistent volume for non-critical applications and additional dedicated persistent volumes for critical tasks. The logs are implemented to minimize the risk of losing log lines and reduce resource overheads. Logs are formatted to provide a consistent logging format utilizing industry standard data formats, such as timestamps. Log files are stored persistently outside of pods to ensure high availability for any audit log management needs.

Cluster network logging for debugging and audits

One example method includes identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, determining a total amount of data to be transferred, using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier, where a number of worker nodes created is based on the amount of data, transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier, and the transferring of the segments is performed by the worker nodes, and for each of the objects, placing metadata associated with that object in a bucket.

Cold tiering microservice for deduplicated data

One example method includes, in a data buffer that includes one or more words and whitespaces, calculating a hash value of data in a window that is movable within the data buffer, comparing the hash value to a mask, and when the hash value matches the mask, identifying a position of the window in the data buffer as a chunk anchor position, searching for a whitespace nearest the chunk anchor position, and designating an offset of the whitespace as a segment boundary.

Word aware content defined chunking

Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.

Mapping service level agreements for backup systems to multiple storage tiers in a clustered network

A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Supporting multiple fingerprint formats for data file segment

A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Distributed object replication architecture

A method includes receiving an object from a client as part of a data ingestion process, by a masker worker, generating a list that identifies one or more byte ranges, of the object, to be masked, providing the list to an object access microservice, by the object access microservice, creating an object recipe of the object, and the object recipe includes one or more slices, and each of the slices includes one or more segments of data, masking only those segments of a slice that are in a byte range identified in the list, creating a masked slice that comprises the masked segments, and creating a masked object recipe that contains a reference to the masked slice, and the masking is performed before the object is stored in storage, creating a masked object that includes the masked slice, and deduplicating the masked object.

Data masking in a microservice architecture

Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Data migration and namespace management across domains in multi-domain clustered file systems

A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Distributed and deduplicating file system for storing backup metadata to object storage

Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Scale out deduplicated file system as microservices

Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.

Scalable garbage collection for deduplicated storage

Growing community of inventors

Newtown, PA, United States of America

Philip N Shilane