Emc Corporation

Emc IP Holding Company LLC

Los Alamos National Security, LLC

Triad National Security, LLC

Emc Cororation

Uday K Gupta

Sorin Faibish

John Michael Bent

Percy Tzelnic

Uresh K Vahalia

Stephen A Fridella

Xiaoye Jiang

Stephen J Todd

Dennis P J Ting

Mark A Parenti

James M Pedone, Jr

Andreas L Bauer

Douglas A Wood

Michael B Evans

James Theodore Compton

Gary Alan Grider

Peter C Bixby

Tao Peng

Betti Porat

Roy E Clark

Wayne William Duso

John Forecast

Dennis P Ting

Charles H Hopkins

Stephen A Rago

Surrendranath R Doohkan

Nikolay Popov

Philippe Armangau

Walter A O'Brien, Iii

David Lionel Black

Kiran Prakash Madnani

John T Cardente

Joseph T Frank

Dinesh Venkatesh

Humberto Rodriguez

Brian James Martin

David C Butchart

James Pedone

David Bonnie

Aju John

Hsing-bung Chen

ChiiShing Lin

Thomas G Magorka

Jean-Pierre Bono

Richard P Ruef

Milena Bergant

Ivan Basov

Jiannan Zheng

Christopher H Stacey

Roger Frederick Osmond

William C Whitney

Karthikeyan Srinivasan

Scott R Ostapovicz

Eyal Zimran

Paul M Bober

Gerald Cotter

Jeffrey Louis Alexander

Giridhar Basava

Chris Youngworth

Raymond Angelone

Suchitra Raman

Sumeet K Malhotra

Dennis Pei Jean Ting

John A Hilliar

Christian J Chuba

Richard G Chandler

Sudhir Srinivasan

Yikang Xu

Dennis Ting, Jr

Suresh Krishnappa

Michael Liberty

Sairam Iyer

Todd R Brune

Matthew T Brooks

Leslie E Kuczynski

Example embodiments of the present invention relate to a method, a system, and a computer program product for maintaining dynamically loadable namespaces. The method includes presenting, to an application, a file system having a global single namespace (SN) comprising a plurality of dynamically-loadable namespaces (DLNs) maintained in a capacity tier (CT) of object storage of a storage system. In response to a job submitted by the application, dynamically loading a first dynamically-loadable namespace, including the metadata only for the first dynamically-loadable namespace, associated with the job from the capacity tier of object storage into a fast tier (FT) of storage of the storage system.

Method and system for reducing metadata overhead in a two-tier storage architecture

A cluster file system comprises a burst buffer appliance coupled to a plurality of object storage servers via a network. The burst buffer appliance comprises a controller configured to implement storage tiering control functionality for at least first and second storage tiers comprising respective disjoint subsets of the plurality of object storage servers. The burst buffer appliance is thereby configured to control movement of data between the first and second storage tiers. The object storage servers in the first storage tier may be configured to interface with object storage targets of a first type and the object storage servers in the second storage tier may be configured to interface with object storage targets of a second type different than the first type. For example, the object storage targets of the first type may comprise non-volatile electronic storage devices such as flash storage devices, and the object storage targets of the second type may comprise disk storage devices.

Cluster file system with a burst buffer appliance for controlling movement of data among storage tiers

A set of data chunks stored in a first data storage system is accessed. The set of data chunks includes original data chunks and replicated data chunks respectively corresponding to the original data chunks. A given original data chunk and the corresponding replicated data chunk are stored in separate storage nodes of the first data storage system. For each of at least a subset of storage nodes of the first data storage system, unique ones of the original data chunks and the replicated data chunks stored on the storage node are aggregated to form a data object. The data objects thereby formed collectively represent a given data volume. Each of the data objects is stored in separate storage nodes of a second data storage system.

Data archiving in data storage system environments

Metadata associated with a plurality of sub-files associated with a single shared file is stored in a parallel file system. A plurality of processes generate a shared file. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file and metadata for the at least one portion of the shared file on one or more of the plurality of object storage servers. The compute node is further configured to store the metadata by striping the metadata across a plurality of subdirectories of the shared file. The metadata is optionally striped across the plurality of subdirectories in a round-robin manner. The plurality of subdirectories are stored on one or more of the object storage servers. Write and read processes optionally communicate using a message passing interface. A given write process optionally writes metadata for a given portion of the shared file to an index file in a particular one of the subdirectories corresponding to the given portion.

Parallel file system with striped metadata

An apparatus comprises a multi-tier storage system having a front-end storage tier, a back-end storage tier and a data mover module coupled to the front-end and back-end storage tiers and configured to control movement of data between the storage tiers. A translator at least partially incorporated in or otherwise associated with the multi-tier storage system implements a plurality of translation services and one or more index tables. A given one of the translation services is configured to access at least one of the index tables in order to determine a dynamically loadable namespace identifier associated with an application running on an analysis node that communicates with the multi-tier storage system over at least one network, and to instantiate a name node corresponding to the dynamically loadable namespace identifier. Particular objects of an object store of the back-end storage tier are made accessible to the application via the name node.

Translation-based name node configuration for object access in a multi-tier storage system

An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The storage system comprises a software-defined storage controller configured to implement the front-end storage tier as a unified memory cluster accessible to respective processors of a plurality of compute nodes. The software-defined storage controller is implemented using at least one processing device comprising a processor coupled to a memory. The unified memory cluster in some embodiments comprises portions of respective memories of at least a subset of the compute nodes. The portions of respective memories of at least a subset of the compute nodes may comprise, for example, respective flash memories of those compute nodes, or respective dynamic random access memories of those compute nodes. Other illustrative embodiments include systems, methods and processor-readable storage media.

Multi-tier storage system having front-end storage tier implemented utilizing software-defined storage functionality

A multi-tier storage system is provided that employs a burst buffer middleware appliance for batch messaging. An exemplary multi-tier storage system comprises at least first and second storage tiers comprising storage for storing data; and at least one burst buffer appliance, wherein the at least one burst buffer appliance comprises an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device in at least one of the first and second storage tiers, wherein the at least one persistent storage device provides a list application programming interface, wherein the software interface module provides a batch operation comprised of a plurality of individual operations to the at least one persistent storage device using an aggregated message. The batch operation optionally comprises a batch operation for processing metadata. A burst buffer appliance is also provided for implementing the batch messaging techniques described herein.

Multi-tier storage environment with burst buffer middleware appliance for batch messaging

Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.

Content-aware storage tiering techniques within a job scheduling system

Improved techniques are provided for storing bursty data, such as checkpoints, in parallel computing environments. A parallel file system is provided comprising at least first and second storage tiers comprising respective disjoint subsets of storage; and at least one processing device configured to store burst data from a plurality of distributed processes for a given burst operation during the given burst operation on both of the at least first and second storage tiers. The given burst operation comprises a multi-phase input/output (IO) task, such as a checkpoint, having alternating periods of idle time and bursts of write and/or storage activity. A method is also provided for provisioning a parallel file system by obtaining a specification of burst data requirements for the parallel file system and determining an amount of storage required for at least first and second storage tiers comprising respective disjoint subsets of storage to satisfy the burst data requirements.

Storage of bursty data using multiple storage tiers with heterogeneous device storage

A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

Key-value store with internal key-value storage interface

Cluster file systems are provided having a FUSE interface. A cluster file system comprises a FUSE object storage device that communicates with at least one object storage server of the cluster file system; and provides an interface between the cluster file system and at least one additional file system having a FUSE implementation. The additional file system optionally includes a burst buffer appliance. The burst buffer appliance can be configured, for example, to control the movement of data between first and second storage tiers. The burst buffer appliance can optionally communicate with a plurality of clients over the network and process requests from the clients. The FUSE object storage device provides an interface to, for example, a parallel log-structured file system and/or a Lustre clustered file system.

Cluster file system with a fuse file system interface

Data from a group of distributed processes to a shared file is written using a parallel log-structured file system. A metadata server of a cluster file system is configured to communicate with a plurality of object storage servers of the cluster file system over a network. The metadata server further configured to implement a Parallel Log Structured File System (PLFS) library to coordinate storage on one or more of the plurality of object storage servers of a plurality of portions of a shared file generated by a plurality of applications executing on compute nodes of the cluster file system and to store metadata for the plurality of portions of the shared file. Concurrent writes to the shared file are decoupled by writing the plurality of portions of the shared file generated by each of the plurality of applications to independent write streams for each application. The metadata server communicates with a plurality of applications executing on the compute nodes over the network to process metadata requests from the applications.

Cluster file system with metadata server for storage of parallel log structured file system metadata for a shared file

A computer-executable method, system, and computer program product of managing I/O received within a Lustre file system, the computer-executable method, system, and computer program product comprising: receiving a data I/O request, wherein the data I/O request relates to data stored within the Lustre file system; processing the data I/O request in a journal stored on a fast data storage device within the Lustre file system; analyzing the journal to make a determination related to the data I/O request; and responding to the data I/O request.

Lustre file system

A method is used in applying XAM processes. A set of content is received via any of a file system interface, a block based interface, an object based interface to an object addressable data storage system. An object derived from the set of content and having an object identifier is stored in the object addressable data storage system. The object is made available for retrieval via the object based interface using the object identifier.

Applying XAM processes

A cluster file system comprises a metadata server and a plurality of object storage servers each having one or more object storage targets. The object storage servers and their corresponding object storage targets are arranged in a first storage tier of the cluster file system. The cluster file system further comprises a second storage tier that includes a plurality of nodes of a scale-out network attached storage cluster. The cluster file system is configured to control movement of data between the first and second storage tiers. For example, in some embodiments, the object storage servers comprise one or more components of parallel log-structured file system (PLFS) middleware configured to control migration of data between the first and second storage tiers.

Cluster file system comprising object storage server tier and scale-out network attached storage tier

Growing community of inventors

Westford, MA, United States of America

Uday K Gupta