Emc Corporation

Isilon Systems, Inc.

Igneous Systems, Inc.

Emc IP Holding Company LLC

Igeneous Systems, Inc.

Eric Michael Lemar

Neal Thomas Fachan

Darren P Schack

Jeffrey Douglas Hughes

Triantaphyllos Byron Rakitzis

Justin Husted

Kiran V Bhageshpur

Robert J Anderson

Aaron Passey

Nathan N White

David Richards

Asif Arif Daud

Matthew D Fleming

Anu Engineer

Jonathan Mark Walton

Alain Hamel

Mayank Rajawat

Nathan Edward Dire

Harsha Mahuli

Mohd Fahadullah

Andrew Martin Pilloud

Ronald Stuart Steinke

Kalen Albert Petersen

Sourav Ghosh

Sean MacQueen Smith

Jeffrey Tremaine

Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Distributed storage data repair air via partial data rebuild within an execution path

Implementations are provided herein for data deduplication, and more particularly, to post-process data deduplication on a large scale out storage system. Multiple techniques and implementations are disclosed that offer greater efficiency, higher performance, and more stability when performing post-process data deduplication at large scale. Disclosed implementations are based on a process for data deduplication involving four main phases: enumeration, commonality, sharing, and update. Multi-level hashing can be used to identify candidates for deduplication during the enumeration phase, providing a more efficient use of compute resources. In addition, datasets can be phase rotated through the post-process deduplication steps providing a more controllable deduplication environment as well as a more efficient use of resources.

Scalable post-process deduplication

Embodiments are directed to storing data in a storage system. Data portions may be provided from blocks of write data provided by client computers. A storage location identifier may be associated with each data portion based on the content of the each data portion. Data portions may be stored in a memory buffer that may correspond to the storage location identifiers. If an amount of data stored in the memory buffer exceeds a threshold value, or if a maximum latency time expires, the data portions are stored on a storage location that corresponds to the memory buffer, such that, the maximum latency time is selected to prevent a collision caused by simultaneously storing of data portions in the same storage location. A write acknowledgement message may be provided to the client computers indicating that the blocks of write data are stored to stable storage.

Latency optimizing for scalable journaling

Embodiments are directed towards managing designations of a plurality of files across multiple clusters. Files may be designated as primary files or secondary files. Primary files at one cluster may be asynchronously replicated at another cluster as secondary files. Secondary files may be synchronously validated with remote primary files based on local file access requests. If a write request for a local secondary file is provided, then the local file may be designated as the primary file and the remote file may be designated as the secondary file. Local access may then be provided to the local primary file. If a read request for a local secondary file is provide, then local access may be provided to the local secondary file without changing file designations. In some embodiments, file designations may be preemptively changed based on a file change designation schedule for clusters at different geographic locations.

Geographic distribution of files

Embodiments are directed towards storing data in a storage system. A data controller may obtain a write request and write data from a client computer. A write message may be generated and provided to a data coordinator computer. The data coordinator may communicate the write message to a plurality of L-node computers. The data coordinator may obtain write confirmation messages from the L-node computers that indicate that the write data is stored. If enough write confirmation messages are obtained to indicate that a quorum is reached, the data coordinator may communicate a save confirmation message to the data controller. The data controller may generate a write acknowledgement message based on the save confirmation message provided by the data coordinator. The data controller may provide the write acknowledgement message to the client computer that made the original write request.

Stateless processing of replicated state for distributed storage systems

Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Prioritized repair of data storage failures

Embodiments are directed towards cloning files in a distributed file system. In response to a request to create file clones, a file cloning application can be employed to generate file clones based on a source file. The file cloning application can generate a new inode for each new file clone and at least one new shadow store. Data blocks from the source file can be transferred to the shadow store. The transferred data blocks can be replaced with reference blocks that point to the data blocks transferred to the shadow store. Likewise, additional reference blocks pointing to data blocks in the shadow store can be generated for the new file clone inodes. Further, the shadow store can maintain reference counters that track the number of reference blocks pointing to each shadow store block. Also, file clones can be generated based on a point-in-time snapshot of the source file.

File clones in a distributed file system

A treewalk for splitting a file directory is disclosed for parallel execution of work items over a filesystem. The given work item is assigned to a worker. Thereafter, a request is sent to split the file directory to share a portion of the file directory with another worker. The worker examines the file directory for a next splittable directory entry key after a current in-progress location within a key space for the file directory. If no key is detected, the file directory is identified as unsplittable. If an entry is detected, the key space is progressively searched for a next entry between a next available directory entry key and an upper boundary. The worker identifies the file directory as splittable if the next available entry is found, providing a range of entries within the file directory for which parallel execution of the work item.

Parallel file system processing

Embodiments are directed towards storing data in a storage system using an extensible data path. Data files may be provided to a caching tier in a storage system. If data files remain in the caching tier longer than a time limit, those data files may be removed from the caching tier and provided to a processing pipeline. The processing pipeline may be coupled to a capacity tier of the storage system. Filters to include in the processing pipeline may be determined based on the type of the data files. The data files may be updated based on applying each filter, such that each update corresponding to each filter may be cumulatively applied to each data file. Each updated data file may be stored in the capacity tier of the storage system after each filter in the processing pipeline has been applied.

Extensible data path

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. In an embodiment, a snapshot tracking data structure is maintained for efficient creation and deletion of the snapshot.

Systems and methods for a snapshot of data

In one embodiment, a user or client device communicates with a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. In some embodiments, systems and methods intelligently choose whether to use copy-on-write or point-in-time copy when saving data in a snapshot version of a file whose current version is being overwritten. In some embodiments, systems and methods allow snapshot users to return from a snapshot directory to the immediate parent directory from which the user entered into the snapshot.

Systems and methods for adaptive copy on write

In general, the invention relates to supporting multiple different character encodings in the same file system. In one embodiment, a method is provided for filename lookup that supports multiple character encodings. The method comprises storing filename data in a first character encoding into an indexed data structure. The method further comprises receiving filename data in a second encoding. The method also comprises looking up filename data in the indexed data structure using the second encoding.

Systems and methods of directory entry encodings

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories. According to one embodiment, the system stores representations of the snapshots that govern a file or directory and its descendants within metadata associated with that file or directory. In one embodiment, before modifying a file or directory, the system updates the representations of the governing snapshots by traversing the metadata of ancestors of the file or directory.

Growing community of inventors

Seattle, WA, United States of America

Eric Michael Lemar