Emc IP Holding Company LLC

Dell Products Inc.

Emc Corporation

Marc A De Souter

Jean-Pierre Bono

Adrian Michaud

Rohit K Chawla

Alexander S Mathews

Morgan A Clark

Ahsan Rashid

Pranit Sethi

JoonJack Yap

Philippe Armangau

Sudhir Srinivasan

Soumyadeep Sen

William C Davenport

Walter Forrester

Sitaram D Pawar

Yining Si

Qin Xin

A method facilitating an update to a file system comprises storing a set of transaction log entries at log space of a storage log associated with the file system, the transaction log entries comprising instructions for fulfilling a set of transactions that define change to the file system, assigning, in an ordered sequence corresponding to a sequence order of journalling of the set of transaction log entries to the log space, transaction markers to a set of transaction marker lists, wherein the transaction markers reference the set of transaction log entries, and, for individual transaction markers of the transaction markers, preventing reuse of the log space corresponding thereto until an evaluation of the set of transaction marker lists results in a determination that writing of all modified system metadata corresponding thereto has been executed and that the transaction marker is set equal to a global head transitioning the transaction markers.

Multiple bitmap single head/tail retirement transaction order

Described is technology that facilitates journalling a transaction log entry to chunks of storage while allowing for obtaining of modified system metadata associated therewith in the case of a computing device issue. An associated method comprises at least storing, by a system operatively coupled to a processor, a set of transaction log entries at log space of a storage log associated with a file system, wherein the transaction log entries comprise instructions for fulfilling a set of transactions, and wherein the transactions define changes to the file system, generating a checkpoint marker corresponding to a set of log storage objects of the storage log at which the transaction log entries are stored, and based on the checkpoint marker, preventing, by the system, reuse of the log space until writing of all modified system metadata corresponding to the transaction log entries has been executed.

Multi-log checkpoint-based retirement transaction order in multi-chunk log

A method for securing data, the method including obtaining, from a metadata node and by file system (FS) client executing on a client application node, a data layout and an encryption key, encrypting, by the client application node, the data stored on the client application node using the encryption key to obtain encrypted data, generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request, wherein the at least one I/O request specifies a location in a storage pool, wherein the location is determined using the data layout, and issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the encrypted data being stored at the location.

Methods and systems for securing data in a distributed storage system

Described is a technology that facilitates directory level snapshots at any level of a filesystem's namespace. A directory inode mapping file associates a first directory's virtual inode number(s) with real inode number(s). A snapshot of the first directory creates a second directory with a second directory inode mapping file copied from the first directory inode mapping file, thereby sharing the real inode data structure(s) of the first directory. In the event that one of the directories is to be modified, the directory real inode is split to provide a new directory real inode file for the to-be-modified directory, with the modification after the split.

Directory snapshots based on directory level inode virtualization

The technology described herein efficiently determines whether a real inode is shared among views, or owned. In-memory data structures include a view snapshot generation counter that is increased as a snapshot that generates a view is created, and an inode total weight. An in-memory virtual inode cache dataset for a filesystem object associated with the view is instantiated with the value of snapshot generation counter, sharing-related data based on the inode mapping file entry for the object, and an inode access weight. To determine whether the inode is shared (and needs to be split), such as on a write to the object, the in-memory data is evaluated. The real inode is shared if the generation counters are unequal, if the sharing-related data indicates sharing at an intermediate indirect block level, or indicates sharing at the inode level and the inode access weight is less than the inode total weight.

Determining a sharing relationship of a file in namespace snapshots

A method for storing data, the method comprising receiving, by an offload component in a client application node, an augmented write request originating from an application executing in an application container on the client application node, wherein the augmented write request is associated with data and wherein the offload component is located in a hardware layer of the client application node, and processing, by the offload component, the augmented write request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data being written to a location in a storage pool.

Methods and systems for storing data in a distributed system using offload components and a GPU module

A method for processing requests, the method comprising: receiving, from a client application node and by a metadata node, a request for a data layout, determining a QoS tag associated with the request, identifying at least one file system block (FSB) in a sparse virtual space, wherein the FSB is mapped to a location in a storage pool that is associated with the QoS tag, generating the data layout based on the at least one identified FSB; and providing the data layout to the client application, wherein the client application node uses the data layout to directly write data to the location in the storage pool, wherein the metadata node does not participate in the writing of the data to the location in the storage pool.

Methods and systems for seamless tiering in a distributed storage system

Described is hard link handling when a directory snapshot exists that includes the hard link's connected file object. A hard link is created by allocating a virtual inode number for the hard link, with the virtual inode number mapped to a real inode number that identifies a real inode of the file object; the hard link is assigned weight. A total weight associated with the real inode is increased by the hard link weight, and a hard link data store is updated with an entry for the hard link. Upon receiving data write request to the hard link, weight data determines that the file object is shared as a result of the snapshot; the hard link is disassociated from the real inode file, and associated with a new real inode number and new real inode of a new file object. The data is written based on the new real inode.

Hard link handling with directory snapshots

A method for storing data, the method comprising receiving, by an offload component in a client application node, a request originating from an application executing in an application container on the client application node, wherein the request is associated with data and wherein the offload component is located in a hardware layer of the client application node, and processing, by the offload component using an advanced data services pipeline, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data in a location in a storage pool.

Methods and systems for storing data in a distributed system using offload components with advanced data services

A method for processing requests includes receiving a request to write data, in response to the request, identifying a sparse virtual space segment using an available space tracking metadata hierarchy, and initiating writing of the data to a physical segment, wherein the physical segment is associated with the sparse virtual space segment.

Methods and systems for allocating free space in a sparse file system

A method for storing data, comprising: obtaining, from a metadata node and by file system client executing on a client application node, a data layout, generating, by the client application node, an error detection value (EDV) for the data stored on the client application node; generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request specifying a location in a storage pool, wherein the location is determined using the data layout; issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and after issuing the at least one I/O request to the storage pool, transmitting the EDV to the metadata node, wherein the metadata node stores the EDV.

Methods and systems for error detection and correction in a distributed storage system

A method for deduplicating data comprising: obtaining, from a metadata node and by file system client executing on a client application node, a data layout; generating, by the client application node, a fingerprint for the data stored on the client application node; generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request specifying a location in a storage pool, wherein the location is determined using the data layout; issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and after issuing the at least one I/O request to the storage pool, transmitting the fingerprint to the metadata node, wherein the metadata node attempts to deduplicate the data using the fingerprint.

Methods and systems for methods and systems for in-line deduplication in a distributed storage system

A method for storing data, the method comprising receiving, by an offload component in a client application node, a request originating from an application executing in an application container on the client application node, wherein the request is associated with data and wherein the offload component is located in a hardware layer of the client application node, and processing, by the offload component, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data in a location in a storage pool.

Methods and systems for storing data in a distributed system using offload components

A method for rebuilding data, comprising: obtaining, from a metadata node, a source file data layout for a source file and a target file data layout for a target file, wherein the source file is associated with a degraded mapped RAID group and the target file is associated with a new mapped RAID group; generating, by the client application node, a plurality of input/output (I/O) requests to read a portion of the data associated with the source file using the source file data layout; obtaining, in response to the plurality of I/O requests, the portion of the data associated with the source file; rebuilding a second portion of the data associated with source file using the portion of the data; and initiating, storage of at least the second portion of the data associated with the source file in the storage pool using the target file data layout.

Methods and systems parallel raid rebuild in a distributed storage system

A method for storing data, comprising receiving, by a file system (FS) client executing on an offload component, a first request from a translation module, wherein the translation module translated a second request that was to be performed on an emulated block device into the first request, wherein the first request is specified using file semantics, wherein the first request is associated with data, wherein the offload component is located in a hardware layer of a client application node, and wherein the translation module is located on the offload component, and processing the first request by the FS client and a memory hypervisor module, wherein the FS client and the memory hypervisor module are executing in a modified client FS container on the offload component, wherein processing the first request results in at least a portion of the data being stored in a location in a storage pool.

Growing community of inventors

Wayne, NJ, United States of America

Marc A De Souter