Vmware, Inc.

Vmwre, Inc.

Prasanna Aithal

Mahesh Shivanagouda Hiregoudar

Prasad Rao Jangam

Rohan Pasalkar

Srinivasa Shantharam

Pradeep H Krishnamurthy

Asit Desai

Bryan Branstetter

Srikanth Mahabalarao

Raghavan Pichai

Sunil Satnur

Krishna Yadappanavar

Nicholas Michael Ryan

Kashish Bhatia

Vivek Patidar

Randhir Singh

Rahul Dev

Gautham Swamy

Kamal Jeet Charan

Dhanashankar Venkatesan

Akshay Jindal

A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

Avoiding data inconsistency in a file system using 2-level synchronization

Example methods and systems to process input/output (I/O) requests in a distributed storage system in a virtualized computing environment are disclosed. One example method includes executing a first thread to destage one or more data writes, wherein the one or more data writes correspond to a first bucket; executing a second thread to destage the one or more data deletes, wherein the one or more data deletes correspond to a second bucket; in response to executing the first thread, buffering write I/Os associated with the one or more data writes in a logical queue; in response to executing the second thread, buffering delete I/Os associated with the one or more data deletes in the logical queue; and adjusting a number of slots in the logical queue dedicated to buffer the delete I/Os based on a relationship between the first bucket and the second bucket.

Method and system to process data delete in virtualized computing

An example method of handling, at a hypervisor on a host in a virtualized computing system, a write input/output (IO) operation to a file on a storage device having a virtual machine file system (VMFS) is described. The method includes: generating logical transactions for the write IO operation having updates to metadata of the VMFS for the file; estimating, for the logical transactions, common space reservations for those of the updates to common fields in the metadata for the file; estimating, for the logical transactions, exclusive space reservations for those of the updates to exclusive fields in the metadata for the file; batching the logical transactions into a physical transaction, which includes a single reservation of space in a journal of the VMFS based on the common space reservations and a reservations of space in the journal for each of the exclusive space reservations, respectively.

Journal space reservations for virtual disks in a virtualized computing system

A method of deleting a first pointer block of a plurality of pointer blocks of a file system from a storage device used by a plurality of applications, wherein the plurality of pointer blocks are each subdivided into sub-blocks, includes the steps of: determining that a first sub-block of the first pointer block is marked as being empty of any addresses of the file system at which storage space is allocated to files of the applications; determining that a second sub-block of the first pointer block has not been marked as being empty; in response to the determining that the second sub-block has not been marked as being empty, determining that the second sub-block does not contain any addresses of the file system at which storage space is allocated to the files of the applications; and deleting the first pointer block from the storage device.

Optimal method for deleting sub-blocks of a pointer block that do not have on-disk metadata headers for addresses

An example method of handling, at a hypervisor on a host in a virtualized computing system, a write input/output (IO) operation to a file on a storage device having a virtual machine file system (VMFS) is described. The method includes: sorting, at the hypervisor, a scatter-gather array for the write IO operation into sets of scatter-gather elements, each of the sets including at least one scatter-gather element targeting a common file block address; resolving offsets of the sets of scatter-gather elements to identify a first scatter-gather array of transaction-dependent scatter-gather elements; generating logical transactions for the first scatter-gather array having updates to metadata of the VMFS for the file; batching the logical transactions into a physical transaction; and executing the physical transaction to commit the updates to the metadata of the VMFS on the storage device for the file.

Write input/output optimization for virtual disks in a virtualized computing system

A method of deleting a first pointer block of a plurality of pointer blocks of a file system from a storage device used by a plurality of applications, wherein the plurality of pointer blocks are each subdivided into sub-blocks, includes the steps of: determining that a first sub-block of the first pointer block is marked as being empty of any addresses of the file system at which storage space is allocated to files of the applications, determining that a second sub-block of the first pointer block has not been marked as being empty; in response to the determining that the second sub-block has not been marked as being empty, determining that the second sub-block does not contain any addresses of the file system at which storage space is allocated to the files of the applications; and deleting the first pointer block from the storage device.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reclaiming one or more portions of storage resources in a computer system serving one or more virtual computing instances, where the storage resources in the computer system are organized in clusters of storage blocks. In one aspect, a method includes maintaining a respective block tracking value for each storage block that indicates whether a call to reclaim the storage block is outstanding; determining, from the block tracking values, a respective cluster priority value for each of the clusters based on a count of storage blocks in the respective cluster for which a call to reclaim is outstanding; and reclaiming a first portion of storage resources in the computer system in accordance with the cluster priority values.

Reclaiming storage resources

The disclosure herein describes managing a rate of processing unmap requests for a data storage volume. Unmap requests are received from a cluster of active hosts that are associated with the data storage volume. Latency data values of each active host are then accessed. A long-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a long-term time period and a short-term cluster latency average value is calculated based on the accessed latency data values of all active hosts over a short-term time period. An unmap rate adjustment value is calculated based on a difference between the long-term cluster latency average value and the short-term cluster latency average value. The rate of processing unmap requests for the data storage volume is adjusted based on the unmap rate adjustment value and the unmap requests are performed based on the adjusted rate.

Management of unmap processing rates in distributed and shared data storage volumes

The systems described herein are configured to enhance the efficiency of memory usage and access in a VM file system data store with respect to allocating memory in large and small file block clusters using affinity metadata and propagating and maintaining the affinity metadata in support of the described allocation. In order to maintain affinity metadata of the large file block cluster, affinity generation values stored on the large file block cluster are read and cached affinity generation values for each small file block cluster are read from an in-memory cache associated with the large file block cluster. When the stored affinity generation values and the cached affinity generation values do not match, affinity metadata from all the small file block clusters associated with the large file block cluster is used to update the affinity metadata of the large file block cluster and the associated cache.

Propagating affinity data to large file block clusters in a file system

A method is provided for a computer to allocate a resource from a clustered file system (CFS) volume stored on one or more physical storage devices to a file. The CFS volume includes resources organized into resource clusters and the resource clusters make up regions. The method includes, for each region of resource clusters, determining a first count of resources allocated to the host computer and a second count of resources allocated to all other host computers, and calculating a region weight based on the first count and the second count. The method further includes sorting a list of the regions based on their region weights, selecting a region at or near the start of the list, and allocating the resource from a resource cluster in the selected region to the file.

Region to host affinity for block allocation in clustered file system volume

Techniques for decoupling the commit and replay of file system metadata updates in a clustered file system (CFS) are provided. In one embodiment, a CFS layer of a computer system can receive a file I/O operation from a client application, where the file I/O operation involves an update to a file system metadata resource maintained on persistent storage. In response, a journaling component of the CFS layer can execute a commit phase for committing the update to a journal on the persistent storage. The CFS layer can then return an acknowledgment to the client application indicating that the file I/O operation is complete, where the acknowledgement is returned prior to completion of a replay phase configured to propagate the update from the journal to one or more locations on the persistent storage where the file system metadata resource is actually stored.

Decoupling the commit and replay of metadata updates in a clustered file system

The systems described herein are configured to enhance the efficiency of memory usage and access in a VM file system data store with respect to allocating memory in large and small file block clusters using affinity metadata and propagating and maintaining the affinity metadata in support of the described allocation. During storage of file data, an affinity identifier of the file data is determined. The affinity identifier is used to identify a large file block cluster and a small file block cluster within the identified large file block cluster. The file data is stored in the selected small file block cluster and affinity metadata of the selected small file block cluster is updated to reflect the storage of the file data.

Growing community of inventors

Bengaluru, India

Prasanna Aithal