Emc IP Holding Company LLC

Dell Products Inc.

Abhinav Duggal

Philip N Shilane

Ramprasad N Chinthekindi

George Mathew

Tony T Wong

Atul Avinash Karmarkar

Kevin Xu

Lan Bai

Chegu Vinod

Hemanth Satyanarayana

Srikanth Srinivasan

Smriti Thakkar

Jehuda Shemer

Nitin Madan

Dheer Moghe

Senthil Ponnuswamy

Mahadev Karadigudda

Anand Rudrabhatla

Ashwani Mujoo

Bhimsen Bhanjois

Mahesh Kamat

Abdullah Reza

Kimberly R Lu

Tipper Truong

Kalidas Balakrishnan

Fani Jenkins

Philip Shilane

Chinthekindi Ramprasad

Shah Veeral

Anshita Agrawal

Ranganathan Dhathri Purohith

Yu Qiu

Rahul Nihalani

Pei Jie Sim

Joseph Jobi

Metadata of a file system is maintained in a key value store. The metadata includes a namespace represented as a B+ tree having pages written to the key value store. Identifiers to the pages include a snapshot identifier and a page number. The identifiers are stored as keys in the key value store. The snapshot identifiers are used to track pages that are shared between first and second snapshots. Upon a request to write to an existing page, a determination is made from the snapshot identifiers as to whether the existing page belongs to the first or second snapshot. If the existing page belongs to the second snapshot, the write to the existing page is allowed. If the existing page belongs to the first snapshot, a copy on write (COW) is performed to generate a new page for the write.

Storing namespace metadata in a key value store to facilitate space efficient point in time snapshots

A cluster service receives a cluster message based on a removal of a first service from membership in a first node in a cluster, and requests a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests to access resources stored by the shared storage. The cluster service receives the new access key from the shared storage, and sends the new access key in at least some cluster messages to at least a second service in a second node in the cluster of nodes. The second service creates a key based on the new access key. The second service uses the key to create a request to access a resource stored by the shared storage, sends the request to the shared storage, and then accesses the resource, which was previously accessed by the first service.

Fencing off cluster services based on shared storage access keys

A distributed and deduplicating file system that stores backup data to object storage is described. A node, in a cluster of nodes, receives a write operation to back up a data file, and identifies backup data associated with the write operation. The node identifies deduplicated data by deduplicating the backup data. The cluster stores the deduplicated data to each of the buffers which corresponds to a node in the cluster. If the node determines that the size of information, which includes the deduplicated data, stored to each of the buffers has accumulated to become greater than or equal to an object threshold, then the node writes the information, which includes the deduplicated data, stored in each of the buffers to an object storage which stores data for the nodes in the cluster.

Distributed and deduplicating file system for storing backup data to object storage

Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.

Mapping service level agreements for backup systems to multiple storage tiers in a clustered network

One example method includes generating a fingerprint:tag dictionary that includes a group of fingerprints and a group of tags, and the fingerprint:tag dictionary identifies, for each fingerprint, the tag or tags which include that fingerprint, computing a similarity matrix based on the fingerprint:tag dictionary, and the similarity matrix identifies, for each pair of tags in the fingerprint:tag dictionary, a relative similarity of the tags in the pair to each other, running a clustering algorithm to identify groups of similar tags in the similarity matrix, and deduplicating, based on the groups of similar tags, respective data associated with the fingerprints.

Efficient dictionary data structure to find similar backup clients

A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Supporting multiple fingerprint formats for data file segment

A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Distributed object replication architecture

A fencing service receives a cluster message based on a first service's removal from membership in a first node in a cluster, and generates an access key based on a generation identifier in the cluster message. The fencing service sends the access key to a shared storage, thereby enabling the shared storage to create signing keys based on the access key, and which are used to validate signed requests to access resources stored by the shared storage. A second service in a second node in the cluster receives the cluster message, and generates the access key based on the generation identifier. The second service accesses a resource previously accessed by the first service, by generating a signing key based on the access key and a request to access the resource stored by the shared storage, and then sending the request signed by the signing key to the shared storage.

Fencing off cluster services based on access keys for shared storage

Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Data migration and namespace management across domains in multi-domain clustered file systems

A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Distributed and deduplicating file system for storing backup metadata to object storage

Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Scale out deduplicated file system as microservices

Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

Processing out of order writes in a log structured file system for improved garbage collection

If a client is configured to create fingerprints for data file segments, the configured client identifies any format specified for fingerprints created by the configured client, and identifies fingerprint formats, including a first fingerprint format and/or a second fingerprint format, supported by a backup server associated with the configured client. If the fingerprint formats supported by the backup server includes any format specified for fingerprints created by the configured client, the configured client partitions the backup server to store fingerprints created by the configured client. The configured client creates fingerprints for segments of a data file identified for backup, and sends the created fingerprints to the configured backup server. The configured client identifies a request from the configured backup server for a segment, of the data file, which corresponds to a non-duplicate fingerprint; and sends the requested segment to the configured backup server, which stores the requested segment.

Client support of multiple fingerprint formats for data file segments

One example method includes scanning a storage device to obtain data and metadata concerning operation of a computing system, analyzing the data and, based on the analyzing, deriving data groups that include some of the data, and deriving data relationships among some of the data, receiving, by an expert system, a query from a user, and the query specifies a sample object for the expert system to investigate, but the query does not indicate purpose of the user in submitting the query, analyzing the query, based on the data groups and data relationships, and based on the analyzing of the query, generating, by the expert system, query results that comprise a set of user-selectable investigation directions that relate to the sample object, and presenting, by the expert system, the set of user-selectable investigation directions to the user.

Interactive query by example expert system for autonomous data protection

Embodiments are described for a multi-domain and multi-tier architecture for clustered network file systems. This system allows a user to create sub-cluster of physical nodes, called domains, and file system resources for the data placed in a domain are allocated only from the nodes in the domain. It limits the impact of system failures to the files within a domain. A file system redirection service manages a global namespace spanning the domains and redirects file accesses to the appropriate domain where it is stored. In each domain, there are different classes of storage, tiers, with different cost and performance characteristics. Files can be placed on a set of tiers depending on a storage level agreement (SLA) specified for a file. Tier examples include a higher performance tier consisting of SSDs and a lower performance tier of HDDs.

Growing community of inventors

Santa Clara, CA, United States of America

Abhinav Duggal