Vmware, Inc.

Springpath, Inc.

Other

Cisco Technology, Inc.

Faraz Ahmed Shaikh

Krishna Yadappanavar

Mallikarjunan Mahalingam

Sandip Agarwala

Murali Vilayannur

Praveen Vegulla

Smit Shah

Shravan Gaonkar

Abhishek Chaturvedi

Satyam B Vaghani

Jyothir Ramanan

Mayuresh Vartak

Kiran R Joshi

Sazzala Reddy

Jeffrey Zabarsky

Ashwin Pednekar

Swapnil Pimpale

Bhavesh Mehta

Manjunath Rajashekhar

Presented herein are methodologies for implementing erasure coding in a distributed log structured storage system. A method includes receiving a write request for first data from a file system, selecting a physical sector on a selected storage device in an array of storage devices on which to store the first data, assigning a key to the physical sector, storing the key and an indication of the physical sector in a key-to-physical medium map, erasure coding the data, including generating parity data associated with the first data, writing the first data and the parity data as a data stripe to each storage device in the array of storage devices and, in response to receiving the write request, sending the key to the file system. Read, update, and delete procedures in the context of a log structured framework are also described.

System and method for garbage collecting inline erasure coded data for a distributed log structured storage system

System and method for inline erasure coding for a distributed log structured storage system

Creating and using highly available transaction logs in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a transaction entry at a first storage pool of the distributed storage system. The storage controller server further looks up a transaction log to store the transaction entry, where the transaction log is associated with a second storage pool. The storage controller server routes the transaction entry to the second storage pool, wherein the second storage pool stores the transaction entry.

Highly available transaction logs for storing multi-tenant data sets on shared hybrid storage pools

A distributed storage system called StorFS that performs distributed data deduplication is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server determines if an entity is duplicated in the distributed storage system in line with an incoming input/output operation. The storage controller server determines if the entity is duplicated in the distributed storage system by receiving the entity and looking up the entity in a data deduplication table. If the entity exists in the data deduplication table, the storage controller server updates the metadata for the entity to point to the duplicate entity.

Distributed deduplication in a distributed system of hybrid storage and compute nodes

A distributed storage system that dispatches an input/output request is described. In an exemplary embodiment, a storage controller client receives the input/output request, wherein the distributed storage system includes the storage controller client, a plurality of storage controller servers, a plurality of virtual nodes distributed among a plurality of physical nodes, and each of the plurality of physical nodes is hosted on one of the plurality of storage controller servers. The storage controller client further computes a target virtual node for the input/output request, where the target virtual node is one of the plurality of virtual nodes. Using the computed target virtual node, the storage controller client determines a target physical node that corresponds to the target virtual node, where the target physical node is one of the plurality of physical nodes. In addition, the storage controller client dispatches the input/output request to a target storage controller that corresponds to the target physical node, wherein the target storage controller server is one of the plurality of storage controller servers. In addition, the virtual node includes a set of one or more mirrored copies across different fault domains for ensuring resiliency and high availability.

Systems and methods for implementing an enterprise-class converged compute-network-storage appliance

A distributed snapshot in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server receives a request to create the snapshot in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes and a plurality of physical nodes, and the source object includes a plurality of stripes. The storage controller server further determines a set of virtual nodes from the plurality of virtual nodes, where each of the set of virtual nodes owns one of the plurality of stripes of the source object. For each of the set of virtual nodes, the storage controller server sends a clone request to that virtual node, where the request is to create a snapshot for the stripe hosted by that virtual node.

System and methods for efficient snapshots in a distributed system of hybrid storage and compute nodes

A distributed multi-layer cache in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server determines if an object is in a cache that stores a plurality of objects. In addition, the distributed storage system includes the cache and a distributed object layer for persistently storing the plurality of objects. The cache further includes a de-duplicated cache layer. The storage controller server accesses the object from the cache if the object is in the cache and accesses the object from the distributed object layer if the object is not in the cache.

Distributed multi-level caching for storage appliances

Free storage blocks previously allocated to a logical block device are released back to an underlying storage system supporting the logical block device in a manner that does not conflict with write operations that may be issued to the free storage blocks at about the same time. According to a first technique, write operations on the same storage blocks to be released are paused until the underlying storage system has completed the releasing operation or, if the write operations are issued earlier than when the underlying storage system actually performs the releasing operation, such storage blocks are not released. According to a second technique, a special file is allocated the free storage blocks, which are then made available for safe releasing.

Consistent unmapping of application data in presence of concurrent, unquiesced writers and readers

A distributed garbage collection in a distributed storage system is described, where the storage controller functions of the distributed storage system are separated from that of distributed storage system storage media. In an exemplary embodiment, a storage controller server generates a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers. The storage controller server further scans the plurality of block segments to generate segment summary statistics, where the segment summary statistics indicates the number of live objects stored in the plurality of block segments. In addition, the storage controller server compacts each of the plurality of block segments that have a low utilization based on the segment summary statistics. Furthermore, the live object map is a probabilistic data structure storing a list of valid objects.

System and methods for removing obsolete data in a distributed system of hybrid storage and compute nodes

A distributed storage system that performs automated load balancing is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server detects a load balancing event in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes distributed across a plurality of physical nodes. In response to detecting the load balancing event, the storage controller server determines that a current virtual node is to move from a source physical node to a destination physical node. In addition, the current virtual node is one of the plurality of virtual nodes and the source and destination physical nodes are in the plurality of physical nodes. The storage controller server further moves the current virtual node from the source physical node to the destination physical node.

Automated load balancing across the distributed system of hybrid storage and compute nodes

A virtualized computer system employs a virtual disk. Multiple snapshots of the virtual disk can be created. After a snapshot is created, writes to the virtual disk are captured in delta disks. Two snapshots are consolidated by updating block references in snapshot meta data. Block reference update takes advantage of the fact that blocks for the two snapshot are managed within the same storage container and, therefore, can be moved in the snapshot logical space without incurring data copy operations. Consolidation of delta disks also gracefully handles failures during the consolidation operation and can be restarted anew after the system has recovered from failure.

Virtual disk snapshot consolidation using block merge

Input/output operations (IOs) are issued to a storage system using request queues that are each maintained for a resource targeted by the IOs. When an IO is requested, the target resource for the IO is first identified. If a request queue is maintained for the target resource, the IO is added to the request queue and the IO is issued to the storage system as the target resource becomes available. The availability of the target resource may be determined through periodic checks or by monitoring completions of IOs issued out of the request queue.

Managing storage commands according to input-output priorities and dependencies

A virtualized computer system employs a virtual disk with a space efficient (SE) format to store data for virtual machines running therein. Data within a virtual disk with a SE format is stored in a grain, where multiple grains are included in a storage block. Writes to a grain within shared storage block in a virtual disk with an SE format are serviced by allocating a new grain and storing the write data to the new grain. Metadata associated with the client that transmitted the write request to the virtual disk is then updated to point to the new grain instead of the grain within the shared storage block.

Growing community of inventors

Sunnyvale, CA, United States of America

Faraz Ahmed Shaikh