Emc IP Holding Company LLC

Emc Corporation

Los Alamos National Security, LLC

Jingwang Zhang

Sorin Faibish

John Michael Bent

Zhenhua Zhang

Xuezhao Liu

Haiyun Bao

Percy Tzelnic

Gary Alan Grider

A computer-executable method, system, and computer program product for managing metadata in a distributed data storage system, wherein the distributed data storage system includes a first burst buffer having a key-value store enabled to store metadata, the computer-executable method, system, and computer program product comprising receiving, from a compute node, metadata related to data stored within the distributed data storage system, indexing the metadata at the first burst buffer, and processing the metadata in the first burst buffer.

Advanced metadata management

Methods and apparatus are provided for simulating asynchronous modifications of shared data objects by a number of distributed processes within a distributed application. An exemplary method comprises obtaining a plurality of simulation parameters that define parameters of a simulated distributed application, parameters of a plurality of distributed compute nodes on which the simulated distributed application executes and a storage system employed by the simulated distributed application; and applying the plurality of simulation parameters to a simulation engine, wherein the simulation engine simulates both synchronous and asynchronous modifications of at least one shared data object by a plurality of distributed processes within the simulated distributed application. One or more exemplary techniques evaluate a degree to which an asynchronous shared data modification model can improve overall performance of a distributed application or reduce the required capabilities of the storage system needed to support the distributed application, relative to the same distributed application using a synchronous shared data modification model.

Simulation of asynchronous modifications of shared data objects by a distributed application

A cluster file system comprises a parallel file system and a plurality of burst buffers, with the burst buffers collectively configured to provide an interface between the parallel file system and compute nodes of a computer system. Each of the burst buffers comprises a virtual file system component and a metadata server component. The virtual file system components of respective ones of the burst buffers collectively implement a virtual file system. Metadata relating portions of the virtual file system to portions of the parallel file system is stored in a distributed manner over the metadata server components of the respective burst buffers. Each burst buffer independently manages its own flash memory, but the burst buffers are configured such that data consistency is ensured across their flash memories. The virtual file system illustratively comprises a parallel log-structured file system (PLFS) and the metadata server components illustratively comprise respective TO forwarding servers.

Cluster file system comprising multiple burst buffers each including virtual file system and metadata server components

Techniques are provided for storing data and metadata on sharded storage arrays. In one embodiment, data is processed in a sharded distributed data storage system that stores data in a plurality of shards on one or more storage nodes by providing a plurality of addressable virtual shards within each of the shards, wherein at least a first one of the addressable virtual shards stores the data, and wherein at least a second one of the addressable virtual shards stores the metadata related to the data; obtaining the data from a compute node; and providing the data and the metadata related to the data stored to the sharded distributed data storage system for storage in the respective first and second addressable virtual shards. The metadata related to the data is stored together at a portion of a corresponding stripe for the data in the second one of the addressable virtual shards. A third one of the addressable virtual shards optionally stores a checksum value related to the data.

Storing data and metadata in respective virtual shards on sharded storage systems

End-to-end data integrity is provided in parallel computing systems, such as High Performance Computing (HPC) environments. An exemplary method is provided for processing data in a distributed data storage system by obtaining the data and one or more corresponding checksum values from a compute node; and providing the data and the one or more corresponding checksum values to the distributed data storage system for storage. One or more checksum values corresponding to the data can be generated if the one or more checksum values are not received from a compute node. Exemplary processes are provided for copy; slice; merge: and slice and merge functions. The distributed data storage system comprises, for example, one or more Parallel Log-Structured File System (PLFS) storage elements and/or key-value storage elements storing one or more key-value pairs.

End-to-end data integrity in parallel storage systems

Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.

Growing community of inventors

Beijing, China

Jingwang Zhang