Emc Corporation

Emc IP Holding Company LLC

International Business Machines Corporation

Datrium, Inc.

Vmware, Inc.

Alibaba Group Holding Limited

Windsor Hsu

Soumyadeb Mitra

Frederick Douglis

Jian Xing

R Hugo Patterson, Iii

Teng Xu

Junxu Li

Philip N Shilane

Hyong Shim

Hangwei Qian

R Hugo Patterson

Abhishek Rajimwale

Kevin M Greenan

Ao Ma

Andrew W Leung

Lan Chin

Grant R Wallace

Stephen Smaldone

Surendar Chandra

Shaunchi Ong

Jiesheng Wu

John Davis Palmer

Lesley Mbogo

Xing Lin

Methods and systems implement local file systems of computing nodes which translate random-access read and random-access write operation calls to append-based operation calls in accordance with a distributed file system ('DFS') implemented across storage nodes of a cloud network. Computer-executable applications running on the computing nodes may generate kernel-level read and write system calls by application programming interfaces (“APIs”) such as the Portable Operating System Interface (“POSIX”) standard, to a local file system. The local file system may translate these read and write system calls to file operations at a DFS implementing an append-only file system, as well as perform storage reclamation upon the DFS periodically and/or upon storage thresholds being exceeded. Storage for computing workloads on the computing nodes may be decoupled from individual computing nodes, in comparison to conventional architectures for computing clusters and storage clusters in cloud networks, enabling storage to be scaled without static allocations.

Distributed file system servicing random-access operations

Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Distributed data method for encrypting data

According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to a request for writing a file to the storage system, the file is stored in a first of the FS partitions that is selected based on a time attribute of the file, such that files having similar time attributes are stored in an identical FS partition.

Time-based data partitioning

In response to a request for accessing a file stored in a storage system, data objects associated with the file are retrieved from a storage device of the storage system. The data objects of the file are cached in a cache memory. An access sequence of the cached data objects within the file is determined based on metadata of the file, where the access sequence represents a sequential order in time of accessing the cached data objects within the file. In response to a request for cache space reclamation, one or more cached data objects are identified whose next access is a farthest in time from a data object currently being accessed amongst the cached data objects based on the access sequence of the data objects. The identified data objects are evicted from the cache memory whose next access is a farthest amongst the cached data objects.

System and method for cache replacement using access-ordering lookahead approach

According to one embodiment, in response to a request for verifying a first prime representing a consistent point of a file system of a storage system having a plurality of storage units, each of a plurality of prime segments collectively representing the first prime is examined to determine whether the corresponding prime segment has been previously verified. Each of the prime segments is stored in one of the storage units, respectively. At least a first of the prime segments that has not been previously verified is verified, without verifying a second of the prime segments that has been previously verified. The first prime, when at least the first prime segment has been successfully verified, can be used to construct the consistent point of the file system.

System and method for verifying consistent points in file systems

Techniques for reducing overheads of primary storage while transferring modified data are described. According to one embodiment, in response to a request at a primary storage for a backup of a plurality of data blocks, a snapshot of the plurality of data blocks is captured, wherein the snapshot represents a consistent state of the plurality of data blocks. Then a set of data block of the captured snapshot that are to be copied to a secondary storage is identified and the identified set of data blocks are transferred to a secondary storage over a network. In response to receiving data to be written to a first data block at the first primary storage during the transfer, current contents of the first data block is preserved if the first data block is in identified set of data blocks, and the received data is committed to the first data block.

Method and apparatus for reducing overheads of primary storage while transferring modified data

Techniques for copying forward live data within storage regions used by a file system to new storage regions are provided. Storage regions refer to ranges of persistent storage within a storage medium. One or more live-clump-identifying workers identify a set of live-clump-references. The live-clump references are references to live clumps that belong to live objects. Live clumps represent a unit of data. After identifying the set of live-clump-references, one or more container-identifying workers identify a set of storage regions that contain clumps that correspond to the set of live-clump references. After identifying the set of storage regions corresponding to the set of live-clump-references, a plurality of copy-forward task workers perform copy-forward operations on the set of storage regions to copy live data to new storage regions.

Distributed data parallel method for reclaiming space

A system for storing files comprises a processor and a memory. The processor is configured to break a file into one or more segments; store the one or more segments in a first storage unit; and add metadata to the first storage unit so that the file can be accessed independent of a second storage unit, wherein a single namespace enables access for files stored in the first storage unit and the second storage unit. The memory is coupled to the processor and configured to provide the processor with instructions.

System and method for providing long-term storage for data

A method, article of manufacture, and apparatus for transferring data from a source tier to a target tier. In some embodiments, this may include dividing an object stored in the source tier into a plurality of segments, determining if the target tier is missing at least one of the segments, writing the missing segments to the target tier, and deleting the object from the source tier. In some embodiments, the object's metadata is updated to indicate that the object is in the target tier.

Efficient storage tiering

According to one embodiment, a request is received for obtaining a consistent point of data stored in a file system of a storage system having a plurality of storage units. In response to the request, retrieving a prime dependency list from a first prime segment stored in a first of the storage units, where the prime dependency list includes information identifying at least a second prime segment stored in a second of the storage units. The first and second prime segments collectively form a prime segment representing a consistent view of the file system. Each of the prime segments listed in the prime dependency list is ascertained in an attempt to generate the consistent point of data.

System and method for maintaining consistent points in file systems using a prime dependency list

According to one embodiment, recovering file systems without a replica is described. In one example, a method includes discovering an error in a first storage partition of the file system containing a current version of a namespace metadata, finding a second storage partition containing a most recent copy of the namespace metadata, and restoring the found copy of the namespace metadata from the second storage partition to the first storage partition for use as a current version of the namespace metadata.

System and method for recovering file systems without a replica

According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to an input and output (IO) request for accessing a first of the FS partitions, a second of the FS partitions is selected that is currently in a ready state for access. The second FS partition is then removed from the ready state for access. The first FS partition is brought into a ready state for access.

Resource efficient scale-out file systems

A computer-implemented proxy server local to a backup target system performs a backup method for a system supporting a plurality of virtual machines. The backup method includes receiving an initialization of a connection over a network from a backup server identifying the system to backup to the backup target system, establishing a communication connection over the network with the system to backup including identifying an interface for accessing data of the system to backup, receiving an image of the system to backup from the backup server, retrieving a data set corresponding to the image from the system to backup over the network, and writing the data set to a local file in the backup target system.

Dynamic proxy server assignment for virtual machine backup

A request is received to determine a consistent point of data stored in a file system of a storage system having storage units. In response to the request, a prime dependency list is retrieved from a first prime segment stored in a first storage unit, the prime dependency list including information identifying at least a second prime segment stored in a second storage unit. The first and second prime segments are identified by a first prime segment identifier (ID) and a second prime segment ID, respectively, which collectively identify a prime representing a first consistent view of the file system. The consistent point of data is determined based the prime segments listed in the prime dependency list, where the consistent point of data represents a file system state at a point in time for restoration of the file system back to a prior known state.

Growing community of inventors

San Jose, CA, United States of America

Windsor Hsu