Netapp, Inc.

Samsung Electronics Co., Ltd.

Timothy Clayton Bisson

Shankar Pasupathy

Mark Walter Storer

Changho Choi

Kiran Nenmeli Srinivasan

Nam Le

Ling Zheng

Steven Robert Kleiman

Stephen Lee Manley

Garth Richard Goodson

Anahita Shayesteh

Gaurav Makkar

Minglong Shao

Andrew W Leung

Ke Chen

Yuvraj Patel

A solid-state drive (SSD) includes: a plurality of data blocks; a plurality of flash channels and a plurality of ways to access the plurality of data blocks; and an SSD controller that configures a block size of the plurality of data blocks. A data file is stored in the SSD with one or more key-values pairs, and each key-value pair has a block identifier as a key and a block data as a value. A size of the data file is equal to the block size or a multiple of the block size.

System and method for leveraging key-value storage to efficiently store data and metadata in a distributed file system

Inventive aspects include a method, apparatus, and system for supporting a native key-value distributed storage system. The system includes a namenode having a KV-SSD and one or more datanodes each including one or more KV-SSDs. The system includes a client device that is communicatively coupled to the namenode and the one or more datanodes. The client device includes a native key-value storage and networking stack. Some embodiments include a hybrid block-based native key-value distributed storage system that supports both block-based files and native key-value tuples.

Native key-value storage enabled distributed storage system

Techniques for a data storage cluster and a method for maintaining and updating reliability data and reducing data communication between nodes, are disclosed herein. Each data object is written to a single data zone on a data node within the data storage cluster. Each data object includes one or more data chunks, and the data chunks of a data object are written to a data node in an append-only log format. When parity is determined for a reliability group including the data zone, there is no need to transmit data from other data nodes where the rest of data zones of the reliability group reside. Thus, inter-node data communication for determining reliability data is reduced.

Methods for managing storage in a data storage cluster with distributed zones based on parity values and devices thereof

The disclosed technique uses virtual machines in solving a problem of persistent state for storage protocols. The technique provides for seamless, persistent, storage protocol session state management on a server, for higher availability. A first virtual server is operated in an active role in a host system to serve a client, by using a stateful protocol between the first virtual server and the client. A second, substantially identical virtual server is maintained in a passive role. In response to a predetermined event, the second virtual server takes over for the first virtual server, while preserving state for a pending client request sent to the first virtual server in the stateful protocol. The method can further include causing the second virtual server to respond to the request before a timeout which is specific to the stateful protocol can occur.

Seamless takeover of a stateful protocol session in a virtual machine environment

A high-performance device interface (HPDI) provides flexible and high-performance access by applications residing in a Virtual Machine (VM) to high-performance devices. The technique enables VM applications to use a single interface even when multiple device drivers exist, and can pass data efficiently between a VM application and a front-end device driver (a device driver implemented in a VM, such as may be used in a paravirtualization environment). It improves overall performance of a VM by reducing the copying of data during communications between a VM application and a front-end device driver, which reduces processor and memory usage.

Interface for enabling an application in a virtual machine to access high performance devices

Using append only log format in data storage cluster with distributed zones for determining parity of reliability groups

Techniques for a data storage cluster and a method for deduplicating data in the data storage cluster in a scalable manner, by (among other things) using an epoch-based global chunk data structure, are disclosed herein. A global chunk data structure for an epoch is distributed and maintained at a plurality of metadata nodes within the data storage cluster. Fingerprints and identifiers of data chunks are written to the cluster after a particular epoch are written to delta chunk data structures stored in different metadata nodes of the cluster. When the data storage cluster advances to the next epoch, the global chunk data structure is updated using the delta chunk data structures. At any given time, data deduplication in the data storage cluster can be conducted based on the global chunk data structure for the current epoch.

Distributed deduplication using global chunk data structure and epochs

A method and apparatus for proxying search requests for a storage system and maintaining a central index for performing the search requests is described herein. An index manager on the storage system may initially produce the central index by examining each file in a file system and update the central index thereafter by examining only those files that have changed since the central index was initially produced or last updated. The index manager may receive a changed file list from a differencing layer configured for comparing snapshots of the file system at different time points to produce changed file lists. A search proxy module may receive search requests in a search protocol and proxy the search requests to a search engine by converting the search requests to another search protocol compatible with the search engine. The search engine may then use the central index for performing the search request.

Proxying search requests for files to a search engine

Embodiments of the present invention provide mechanisms that overcome limitations of existing indexes by creating two-dimensional (2D) spatial indexes to quickly locate data containers that match two or more predicates. This is accomplished by representing metadata attributes describing a data container as dimensions in a 2D space so that a data container can be expressed as a point or a cell in a 2D space with coordinates being a pair of values of the selected attributes. A space filling curve is used to traverse the 2D space and convert each pair of the 2D coordinates to a single space filling curve value. A 2D spatial index is then created based on the computed space filling curve values so that one value can be associated with one or more points (data containers) in the index. Advantageously, the created spatial index provides for searching and processing fewer metadata entries, thereby decreasing the time typically used to search for data.

Two-dimensional indexes for quick multiple attribute search in a catalog system

A facility for comparing two datasets and identifying metadata differences between the two datasets irrespective of the manner in which the data is stored. In some embodiments, the facility includes a comparison unit and a catalog unit. The comparison unit compares a hierarchical hash of a first dataset with a hierarchical hash of a second dataset, the hierarchical hashes each including a plurality of hierarchical hash values, to identify differences in metadata of the first and second datasets by progressively comparing the hierarchical hash values of the first and second hierarchical hashes without comparing the metadata of the first and second datasets. The catalog unit generates a catalog of differences between the first and second datasets, the catalog indicating differences in metadata of the first and second datasets.

Incremental file system differencing

Methods and systems for concurrently reading direct and indirect data blocks of a data object stored in a network storage server system. In one embodiment, upon receiving a request to read a data object, the storage server identifies a location of an indirect data-block of the data object and a total number of direct data-blocks associated with the data object. Using this information, the storage server concurrently reads, using a single read operation, both the indirect data-block and a specific number of data blocks that are contiguous with the location of the indirect data-block. The specific number is commensurate with the total number of direct-data blocks associated with the data object. In one embodiment, the storage server verifies whether the data object is represented using a contiguous-mode layout scheme before performing the concurrent single-read operation to read the data object.

Methods and systems for concurrently reading direct and indirect data blocks

A system and method reclaims unused storage space from a data container, such as a logical unit number (LUN) of a storage system. In particular, a novel technique is provided that allows a storage system to reclaim storage space not used by a client file system for which the storage system maintains storage, without requiring assistance from the client file system to determine storage usage. In other words, storage system may independently reclaim storage space not used by the client file system, without that file system's intervention.

Storage system-based hole punching for reclaiming unused space from a data container

A network storage server implements a method to perform fast crawling of a hierarchical storage structure. The hierarchical storage structure contains data entities stored by a network storage server. The hierarchical storage structure can be recursively divided into a plurality of sections. A plurality of parallel-processing threads can be used to process the plurality of sections. Each thread selects and processes one of the plurality of sections at a time to generate a sorted list of metadata corresponding to the section of the hierarchical storage structure. The sorted lists generated by the plurality of threads are merged to a baseline list. The baseline list contains sorted metadata for entities managed by the hierarchical storage structure. The baseline list can then be outputted as a representation of the state of data stored by the network storage server.

Process for fast file system crawling to support incremental file system differencing

An apparatus to compare two datasets, each of which includes multiple data blocks, includes a comparison unit and a report generator. The comparison unit identifies block-level differences therebetween, by comparing block-level metadata between the first and second datasets, without comparing the contents of the data blocks. The report generator generates a human-readable report of the differences between the first and second version of the dataset, including the differences in individual data blocks between the first and second version of the dataset.

Growing community of inventors

San Jose, CA, United States of America

Timothy Clayton Bisson