Netapp, Inc.

Network Appliance, Inc.

James F Lentini

Stephen Michael Byan

Michael Neil Condict

Arkady Kanevsky

Anshul Madan

Thomas M Talpey

Alexandros Batsakis

Keith Arnold Smith

Christopher Small

Peter Macko

William Karl Jannen

One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.

Host side deduplication

A write request is received to write a data block having a logical block address to a nonvolatile storage device. The method includes writing a value of the data block to the nonvolatile storage device. The writing includes locating a position in a tree-based data structure that includes first and second nodes. The first node is configured to store a first set of data blocks having logical block addresses in a first numerical range, and the second node is configured to store a second set of data blocks having logical block addresses in a second numerical range. The position is located in the first node or the second node depending on the value of the logical block address. The writing includes storing the value of the data block in the position in the tree-based data structure.

Using a tree-based data structure to map logical addresses to physical addresses on a storage device

Systems and methods for efficiently using solid-state devices are provided. Some embodiments provide for a data processing system that uses a non-volatile solid-state device as a circular log, with the goal of aligning data access patterns to the underlying, hidden device implementation, in order to maximize performance. In addition, metadata can be interspersed with data in order to align data access patterns to the underlying device implementation. Multiple input/output (I/O) buffers can also be used to pipeline insertions of metadata and data into a linear log. The observed queuing behavior of the multiple I/O buffers can be used to determine when the utilization of the storage device is approaching saturation (e.g., in order to predict excessively-long response times). Then, the I/O load on the storage device may be shed when utilization approaches saturation. As a result, the overall response time of the system is improved.

Specializing I/O access patterns for flash storage

The present invention provides a system and method for a shared write address protocol (SWAP) that is implemented over a remote direct memory address (RDMA) connection. Each party to a connection establishes a flow control block that is accessible to its partner via a RDMA READ operation. The novel protocol operates so that each module needs to have one outstanding RDMA READ operation at a time, i.e., to obtain the current flow control information from its partner. In operation, if data to be transmitted is less than or equal to a buffer size, an INLINE message data structure of the SWAP protocol is utilized to send the data to be target. However, if the data is greater than the buffer size, a second determination is made as to whether sufficient space exists in the message pool for the data. If insufficient space exists, the sender will wait until sufficient space exists before utilizing a novel WRITE operation of the SWAP protocol to transmit the data.

System and method for a shared write address protocol over a remote direct memory access connection

A system and method for creating and managing a space-efficient, durable key-value map is disclosed. A key management engine initializes a key-value map by associating a plurality of keys with a first slot of the slots in the key-value map. A first key-value pair is then assigned to the first slot of the key-value map. The key management engine subsequently receives an indication that the first key is to be invalidated; and responsively reuses the first slot of the key-value map by assigning a second key-value pair to the first slot. The first key is then recycled in response to the first key becoming valid. The first and second key-value pairs include respective first and second values and respective first and second keys of the plurality of keys associated with the first slot. The first key becomes valid when it is no longer referenced by a data buffer.

Space-efficient, durable key-value map

Scheduling operations such as asynchronous file system operations in a network storage system is accomplished by applying a bid-price online auction methodology, in which bid (willingness-to-pay) values and price (cost) values are dynamically set by storage clients and a storage server, respectively, based on utilization of computing resources. The system provides a framework for adaptively scheduling asynchronous file system operations, managing multiple key resources of the distributed file system, including network bandwidth, server I/O, server CPU, and client and server memory utilization. The system can accelerate, defer, or cancel asynchronous requests to improve application-perceived performance. Congestion pricing via online auctions can be employed to coordinate the use of system resources by clients, so clients can detect shortages and adapt their resource usage.

Adaptive scheduling of storage operations based on utilization of a multiple client and server resources in a distributed network storage system

Automatic network configuration of cluster paths to access data within a cluster is described. Each node of the cluster presents to clients an image of the cluster as a single system. Each node also stores information regarding what node has particular data, as well as information about network addresses and corresponding network capabilities for the various network addresses for each node. When a node receives a request for data, the node determines a node where the request data can be accessed, determines network addresses and network capabilities for each network address of the node, and selects a path to access the node based on the determined information. The receiving node may select the path based on network type and capability, as well as load and performance of the network.

Automatic network cluster path management

A computer-implemented method and apparatus for enumerating program code dependencies is provided. According to an embodiment of the invention, a user selects or otherwise indicates a code base to be processed and analyzed for the purpose of enumerating those program code modules required by the code base in order to properly compile and execute the code base. Next, the code base is processed in accordance with instructions designed to mirror those of a compiler for compiling the code base. Accordingly, external program code dependencies are identified using compile time compilation parameters, as well as extensive programming language analysis.

Method and apparatus for enumerating external program code dependencies

Method and system are provided where a memory key structure is used for authenticating access to a memory location that is registered for a remote direct memory access (RDMA) operation. The memory key structure not only includes a standard memory key that is expected by an RDMA enabled network interface card (RNIC), but also includes an endpoint network address identifier and a transport identifier. The endpoint network address identifier and the transport identifier are verified before an entity is granted access to the registered memory location.

Method and system for secure remote direct memory access

An embodiment of the invention provides an apparatus and method for performing RDMA (Remote Direct Memory Access) network configuration. The apparatus and method measure a performance of each RDMA operation for different data message sizes and determine an RDMA operation to be applied for a particular packet size sent by an application, based on the measured performance. As an example, the RDMA operations are, e.g., RDMA send/receive, RDMA write, RDMA read, memory registration and memory un-registration, or memory bind and memory unbind. The measured performance can be, for example, the total time to perform an RDMA operation for different packet sizes.

Growing community of inventors

Woburn, MA, United States of America

James F Lentini