Emc Corporation

Emc IP Holding Company LLC

Los Alamos National Security, LLC

Dennis P J Ting

Sorin Faibish

Uday K Gupta

John Michael Bent

Percy Tzelnic

James M Pedone, Jr

Uresh K Vahalia

Gary Alan Grider

Wayne William Duso

John Forecast

James J Moore

Robert Wilson

Aju John

David Bonnie

John A Hilliar

Sasan Teymouri

William C Conway

Leslie E Kuczynski

Latchesar A Ionkov

Mehamood Hosein

Techniques are provided for encryption and decryption of time series data using a digital filter array. A plurality of digital time series samples generated by a digital time series generation device are applied to a digital filter array (e.g., a finite impulse response filter or an infinite impulse response filter) that combines delayed versions of the digital time series samples using a plurality of coefficients to generate a plurality of encrypted digital time series samples. The plurality of coefficients comprise a portion of an encryption key associated with the digital time series generation device. The digital time series generation device has an associated device identifier, and the device identifier and the corresponding encryption key are registered with a server. The plurality of encrypted digital time series samples and the device identifier are provided to the server for decryption using an inverse digital filter array based on the encryption key.

Encryption and decryption of time series data using a digital filter array

Improved multi-tier storage techniques are provided for storing data, such as checkpoints or other bursty data, in parallel computing environments. A burst buffer appliance is provided for use in a first storage tier of a multi-tier storage system comprising at least the first storage tier and a second storage tier. The exemplary burst buffer appliance comprises a memory for storing data; and at least one processing device to transform at least a portion of the data for storage on the second storage tier based on one or more performance characteristics of the second storage tier. In at least one embodiment, the at least one processing device is further configured to perform at least one function on the at least the portion of the data on behalf of the second storage tier. The performance characteristics of the second storage tier comprise, for example, a stripe size and/or network topology information.

Multi-tier data storage using burst buffer appliance for workload transformation based on performance characteristics of at least one tier

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

Computing erasure metadata and data layout prior to storage using a processing platform

A multi-tier storage system is provided that employs a burst buffer middleware appliance for batch messaging. An exemplary multi-tier storage system comprises at least first and second storage tiers comprising storage for storing data; and at least one burst buffer appliance, wherein the at least one burst buffer appliance comprises an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device in at least one of the first and second storage tiers, wherein the at least one persistent storage device provides a list application programming interface, wherein the software interface module provides a batch operation comprised of a plurality of individual operations to the at least one persistent storage device using an aggregated message. The batch operation optionally comprises a batch operation for processing metadata. A burst buffer appliance is also provided for implementing the batch messaging techniques described herein.

Multi-tier storage environment with burst buffer middleware appliance for batch messaging

Techniques are provided for content-aware storage tiering techniques within a job scheduling system. A data node in a job scheduling environment receives at least one job to be executed for a portion of a larger logical file, and notifies one or more sibling data nodes to prefetch one or more related portions of the larger logical file. A write operation performed by an application client to store data on the data node is optionally intercepted at the data node and the associated data is stored in a performance storage tier of the data node. The data is optionally moved from the performance storage tier to a capacity storage tier if one or more predefined eviction criteria are satisfied. Data objects to be evicted are selected based on an anticipated future access of the data objects. The eviction selection is optionally based on grey box knowledge of the application client and/or information obtained from the application client.

Content-aware storage tiering techniques within a job scheduling system

A key-value store is provided having one or more key-value storage interfaces. A key-value store on at least one compute node comprises a memory for storing a plurality of key-value pairs; and an abstract storage interface comprising a software interface module that communicates with at least one persistent storage device providing a key-value interface for persistent storage of one or more of the plurality of key-value pairs, wherein the software interface module provides the one or more key-value pairs to the at least one persistent storage device in a key-value format. The abstract storage interface optionally processes one or more batch operations on the plurality of key-value pairs. A distributed embodiment for a partitioned key-value store is also provided.

Key-value store with internal key-value storage interface

A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata request can be processed by a given metadata server independently of other metadata servers in the cluster file system. A distributed metadata storage environment is also disclosed that comprises a plurality of metadata servers having an abstract storage interface to at least one shared persistent key-value metadata store.

Distributed metadata servers for cluster file systems using shared low latency persistent key-value metadata store

A computer-executable method, system, and computer program product for managing I/O requests from a compute node in communication with a data storage system, including a first burst buffer node and a second burst buffer node, the computer-executable method, system, and computer program product comprising striping data on the first burst buffer node and the second burst buffer node, wherein a first portion of the data is communicated to the first burst buffer node and a second portion of the data is communicated to the second burst buffer node, processing the first portion of the data at the first burst buffer node, and processing the second portion of the data at the second burst buffer node.

Hybrid data storage system in an HPC exascale environment

A file system that utilizes Virtual Local Area Network (VLAN) addressing and routing schemes to permit implementing multiple networked file system applications in a single data processing system. In particular, a networked file server or data mover has associated with it a mass storage device such as one or more disk drives, as well as one or more Network Interface Cards (NICs). A network interface to the data mover is associated with VLAN identifiers (IDs) such that a VLAN ID can be assigned to each file system. At the physical layer of the data mover, VLAN information is encapsulated and passed up to various higher protocol layers, such as a network layer and/or a transport layer, eventually to a selected file system as dictated by the VLAN ID. The file system application in effect becomes an addressable element of the VLAN. A single data mover can thus be deployed to service multiple groups of end users, as long as the members of each group have different VLAN identifiers for the multiple file systems. The result provides security, access restrictions, and all the other benefits of VLAN protocols that would be obtained by deploying separate data movers for each VLAN, without incurring the acquisition expense and administrative overhead associated with having to purchase and maintain multiple data mover systems.

File system for virtual local area network

A method and apparatus for implementing a remote mirroring data facility for a computer system comprising a central processing unit (CPU); a first storage system that is coupled to the CPU so that the CPU can store information in the first storage system; a second storage system coupled to the CPU via a communication link; and a mirroring controller to mirror at least some of the information stored in the first storage system in the second storage system by transferring the at least some of the information over the communication link. The communication link can be implemented via a network cloud, which may be the Internet or an intranet. Multiple pipes can be used to pass data through the network cloud in parallel. Alternatively, the communication link can be implemented using wireless technology.

Method and apparatus for implementing a remote mirroring data facility without employing a dedicated leased line to form the link between two remotely disposed storage devices

A trunk in a network file server includes at least two communications ports. The trunk is created by combining the communications ports into a single virtual device and assigning ownership of the communications ports to the virtual device. A failsafe network device including at least two trunks, with each trunk coupled to different switches can be created by creating a third virtual device. Ownership of a first virtual device including communication ports coupled to a first switch and a second virtual device including communication ports coupled to a second switch is assigned to the third virtual device.

Virtual network devices

A file server includes a plurality of stream server computers linking data storage to a data network, and at least two controller servers for controlling the stream server computers. The controller servers are programmed so that at a given time one of the controller servers is active in controlling the stream server computers, and another of the controller servers is inactive. The inactive controller server is programmed to respond automatically to a failure of the active controller server by becoming active. For example, each of the controller servers has a respective flag for indicating whether or not the controller server is active. Each controller server is programmed so that, upon booting, it will read the flag of the other stream server, and if the flag of the other controller server indicates that the other controller server is active, then the controller server becomes inactive. Otherwise, the stream server assumes an active or inactive state based on a predetermined arbitration method. The active controller server also reports failure conditions to a customer service center, monitors the inactive controller server to ensure that the inactive controller server is in a state of readiness, and monitors itself to determine whether it should become inactive.

Hardware and software failover services for a file server

There is a performance loss associated with servicing a pipe or stream for a connection oriented process by maintaining a connection between a server thread and a client for a series of messages. As a result of maintaining this connection, there is less balance; some threads work harder than others, causing a loss of performance. To solve this problem, a collector queue combines messages from the connection oriented process with messages from the other concurrent processes. The threads receive messages from the collector queue rather than individual pipes. Any idle thread can pick up a message from the collector queue. The collector queue keeps track of which pipe each message came from so that the reply of the server to each message is directed to the same pipe from which the message came from. Therefore the collector queue ensures thread balance and efficiency in servicing the messages. In the preferred implementation, each entry in the collector queue includes a message pointer and a pipe pointer. The message pointer points to allocated memory storing the message in a message buffer. The pipe pointer points to the pipe from which the message originated. The collector queue is a singly linked list. A free thread takes an entry off the collector queue, interprets the message of the entry, sends a reply, and deallocates the memory of the entry and the allocated memory storing the message in the message buffer.

Network file server having a message collector queue for connection and connectionless oriented protocols

A network file server includes a first set of data processors for receiving requests from clients, and a second set of data processors for accessing read-write file systems. A respective data processor in the second set is assigned to each file system for exclusive management of locks on the file system. The file server can detect failure of a failed data processor and automatically recover from the failure. When a failure of a data processor in the first set is detected, a spare data processor is programmed with the logical and physical network addresses of the failed data processor so that the spare data processor assumes the network identity of the failed data processor. When a failure of a data processor in the second set is detected, responsibility for management of the locks on each file system managed by the failed data processor is transferred to an operational data processor. Preferably the responsibility is transferred to the operational data processors in such a way as to balance loading on the operational data processors. The data processors can be commodity digital computers for low cost, and a cached disk storage subsystem or file system caches and remote dual copy techniques can be used to ensure high performance and high data availability.

Growing community of inventors

Groton, MA, United States of America

Dennis P J Ting