Emc Corporation

Huawei Technologies Co., Limited

Emc IP Holding Company LLC

Hongliang Tang

Sairam Veeraswamy

Lili Chen

Li Wan

Steven T McClure

Peter C Bixby

Ning Wu

Zhihao Tang

Arieh Don

Jean-Pierre Bono

Xiangping Chen

Qi Zhang

Hui Lei

David W Harvey

Uresh K Vahalia

Felix Shvaiger

Patrick Brian Riordan

Stephen Richard Ives

Alexander S Mathews

John Michael Hayden

Morgan A Clark

Andrew Chanler

Mark K Ku

Vadim Longinov

Xiali He

Kevin Rodgers

Pranit Sethi

Matthew H Fredette

Yong Wang

Michael Scheer

Serge Joseph Pirotte

T David Evans

Qing Liu

Animesh Singh

H Austin Spang, Iv

Peter E Tolvanen

Steve Chalmer

Siyu Zhou

Sairam Iyer

Richard G Chandler

Velmurugan Rathnam

Sethu N Madhavan

Lixin Pang

Theodoros Gkountouvas

Gabriel BenHanokh

In some implementations, a storage node receives data processing instructions from a compute node of a network architecture. The storage node detects a Directed Acyclic Graph (DAG) within the data processing instructions. The DAG includes a plurality of data processing operations arranged within the DAG according to an execution. The data processing operations further include user-defined function (UDF) information associated with a UDF. The storage node may retrieve a first data object from an object store of the distributed storage node based on object keys information within the plurality of data processing operations. The storage node may execute the plurality of data processing operations using the retrieved first data object to generate a DAG execution result, the executing of the plurality of data processing operations based on the execution order. The storage node may communicate the DAG execution result to the compute node in response to the data processing instructions.

Near data processing (NDP) in network nodes

A computer implemented method comprises accessing data stored in memory of the computer system, wherein the memory includes a faster access storage medium and a slower access storage medium; determining respective exponential moving averages of a number of access operations performed on respective memory locations of the memory; and transferring a portion of the data into the faster access storage medium and transferring a portion of the data into the slower access storage medium according to the determined exponential moving average.

I/O hotspot detection using exponential moving average

An interconnected computer system includes a Peripheral Component Interconnect Express (PCIe) fabric, a first computer system communicatively coupled to the PCIe fabric, a second computer system communicatively coupled to the PCIe fabric, and a shared single-access hardware resource coupled to the PCIe fabric. The first computer system includes a first processor and first memory coupled to the first processor configured to store a first flag indicating a desire of the first computer system to access the shared single-access hardware resource and a turn variable indicating which of the first computer system and the second computer system has access to the shared single-access hardware resource. The second computer system includes a second processor and second memory coupled to the second processor configured to store a second flag indicating a desire of the second computer system to access the shared single-access hardware resource.

Apparatus and method for locking PCIE network having non-transparent bridging

A clustered storage system includes a plurality of storage devices, each of which contributes a portion of its memory to form a global cache of the clustered storage system that is accessible by the plurality of storage devices. Cache metadata for accessing the global cache may be organized in a multi-layered structure. In one embodiment, multi-layered structure has a first layer first including a first address array, and the first address array include addresses pointing to a plurality of second address arrays in a second layer. Each second address array in the second layer includes addresses, each of which points to data that has been cached in the global cache.

Method for accessing caches in clustered storage system, involves determining and reading second address from second address array in accordance with request

A computer-implemented method for on-demand data replication in a network architecture includes activating an on-demand program to pair a network-based replication agent with a data source. Compute resources for the replication agent are allocated based on one or more hints received from the data source. The one or more hints are indicative of at least a size of an upcoming data traffic transmission from the data source. Upon receiving data from the data traffic transmission, the replication agent is activated via the on-demand program to replicate the received data using the allocated replication resources. The allocated compute resources are released upon completion of data replication of the received data. The on-demand program is a serverless function configured to activate the replication agent on-demand, upon receiving the data from the data traffic transmission.

On demand network-based replication agent

Described are techniques for processing requests at a data storage system. A request is received from a client to perform an operation with respect to a first data portion stored on physical storage devices of the data storage system. The first data portion is exposed through a set of at least two data nodes each accessing a same copy of the first data portion stored on the physical storage devices. The request is received at a first of the data nodes of the set. The request is processed with respect to the first data portion using the same copy accessible to each of data nodes of the set. The physical storage devices may be configured in a RAID group and the data nodes, optionally along with a name node providing metadata, may be embedded in the data storage system and execute in a virtualized environment.

Distributed file system deployment on a data storage system

In a system in which a plurality of computing elements share a cache, each computing element owns a stripe of the cache. Each stripe contains cache objects that are accessible to all computing elements but managed only by the owning computing element. Each computing element maintains an LRU FIFO queue in local memory for the cache objects owned by that computing element. Each computing element also maintains a separate hash table in local memory for each other computing element. The hash tables indicate access to cache objects that are owned by those other computing elements. Each computing element updates its LRU FIFO queue when it accesses cache objects that it owns. The hash tables are periodically distributed by all computing elements via RDMA so that the LRU FIFO queues of all computing elements can be updated based on accesses to owned cache objects by other non-owner computing elements.

Phased based distributed LRU for shared cache systems

Described are techniques for performing caching. Write operations of a single transactions are received. Write data of the write operations is cached in one or more cache pages of a first cache of a first storage processor. State information is set for the one or more cache pages of the first cache from an invalid state to a prepared state. The write data of the write operations is also stored in a second cache of a second storage processor. A transaction commit state for the single transaction is set indicating that transaction commitment processing for the first cache has commenced and is pending. State information of the one or more cache pages of the first cache is set from the prepared state to a dirty state. The transaction commit state for the single transaction is updated to indicate that the transaction commitment processing for the first cache is completed.

Techniques for maintaining and restoring dirty caches across CPU resets

Global memory of a storage system may be used to provide NVRAM capabilities to guest operating systems accessing the storage system. The non-volatility of NVRAM (i.e. that retains its information when power is turned off) provides that an NVRAM device provided by global memory may be used as a journaling device to track storage operations and facilitate recovery and/or failover processing in a storage system without needing to add additional hardware and/or other installed devices. Use of the global memory according to the system described herein to provide an NVRAM device, that may function as a journaling device, provides for the speeding up of transactions, thereby improving metadata intensive operations performance and reducing recovery time and/or failover time of a storage system without adding additional hardware support.

Global memory as non-volatile random access memory for guest operating systems

When a guest OS loads within the context of a container provided by the host OS, the guest OS uses PCI or other protocol to specify a virtual hardware device. The guest OS enumerates the virtual hardware device to establish the size for the BARs and establish its view of physical addresses for the memory locations. A server running in the context of the container receives read/write requests from the guest OS, maps the read/write requests to host OS physical address space, and posts responses to the virtual hardware device. Since the guest OS executes memory related operations using its own memory space, exits to the container code are not required to implement storage related actions by the Guest OS. This allows performance of an application executing in the context of the guest OS to approximate performance of an application executing in the context of the host OS.

Method and apparatus for enabling access to storage

Methods and apparatus for organizing a plurality of versions of a file in a file system. The plurality of versions of the file are stored in a data structure having a hierarchical organization that reflects relationships among the plurality of versions. Each version of the plurality of versions is assigned a name that identifies its position in the hierarchical organization. The hierarchical organization of the data structure and the descriptive naming convention used for the plurality of versions in the data structure facilitates deleting a version while retaining other versions derived from the version following the deletion of the version, and enables a staged initialization of the data structure upon system start-up.

Methods and apparatus for managing multiple point in time copies in a file system

In a file server, a file system is built upon a volume of data storage. The file system includes multiple cylinder groups. Each cylinder group includes file system blocks. The file system blocks include allocated blocks and free blocks. The volume includes slices of storage, and each slice stores at least one of the cylinder groups. A hierarchical slice map has a top level that includes a count of free blocks in the file system, an upper level that includes a count of free blocks in groups of the slices, an intermediate level that includes a count of the free blocks in sub-groups of the slices, and a bottom level that includes a count of the free blocks in each slice of storage. To find a free block for allocation to a file, the slice map hierarchy is searched in a top-down fashion.

Hierarchical mapping of free blocks of cylinder groups of file systems built on slices of storage and linking of the free blocks

A method is used in determining alternate paths in faulted systems. It is detected that a first processor has a problem accessing a target. The first processor has a first path to the target. An alternate path to the target is determined by identifying that a second processor has better access to the target than the first processor. The second processor has a second path to the target. Responsibility for I/O communication with the target is transferred from the first processor to the second processor. Responsibility for I/O communication with the target is transferred back to the first processor upon resolution of the problem.

Determining alternate paths in faulted systems

Embodiments of the invention relate to maintaining reference counts in a file system to track when a block is available to be freed. When a snapshot copy of a file is created, rather than copying all the blocks of the file for the snapshot copy, the snapshot inode is given pointers to the blocks of the file. Rather than updating the reference counts for all these blocks to indicate that they are now in use by more than one file, the update of reference counts for at least some of these blocks may be deferred until a later time.

Methods and apparatus for creating a branch file in a file system

An apparatus and method are described for defining, maintaining and displaying a checkpoint result set to facilitate identification of desirable checkpoints during system recovery. The checkpoint result set may be mapped to a user-customizable namespace. The user customizes the namespace by selecting one or more characteristics of desirable file system checkpoints. Being able to customize the namespace in this manner enables the user to be provided a more streamlined checkpoint result set in the event of data corruption. As a result an appropriate checkpoint can be more quickly selected for use in recovery.

User customizable CVFS namespace

Growing community of inventors

Shrewsbury, MA, United States of America

Hongliang Tang