Vmware, Inc.

Yunshan Luke Lu

Wenguang Wang

Richard Paul Spillane

Christos Karamanolis

Maxime Austruy

Jayanth Gummaraju

Rawlinson Rivera

Tariq Magdon-Ismail

Ying Jian He

Razvan Cheveresan

Junping Du

Examples disclosed herein relate to propagating changes made on a file system volume of a primary cluster of nodes to the same file system volume also being managed by a secondary cluster of nodes. An application is executed on both clusters, and data changes on the primary cluster are mirrored to the secondary cluster using an exo-clone file. The exo-clone file includes the differences between two or more snapshots of the volume on the primary cluster, along with identifiers of the change blocks and (optionally) state information thereof. Just these changes, identifiers, and state information are packaged in the exo-clone file and then exported to the secondary cluster, which in turn makes the changes to its version of the volume. Exporting just the changes to the data blocks and the corresponding block identifiers drastically reduces the information needed to be exchanged and processed to keep the two volumes consistent.

Container runtime image management across the cloud

Management of applications across nodes using exo-clones

System and method for accessing a distributed storage system uses a storage-level access control process at a distributed file system that interfaces with the distributed storage system to determine whether a particular client has access to a particular first file system object using an identifier of the particular client and storage-level access control rules in response to a file system request from the particular client to access a second file system object in the particular first file system. The storage-level access control rules are defined for a plurality of clients and a plurality of first file system objects of the distributed storage system to allow the particular client access to the second file system object in the particular first file system object only if the particular client has been determined to have access to the particular first file system object according to the storage-level access control rules.

Multi-level access control for distributed storage systems

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.

Multi-tenant distributed computing and database

Systems and methods for allocating space in persistent storage are provided. A modified bitmap and a tree of bitmap summary pages are used to manage the free space of a large scale storage system. The bitmap is separated into fixed size pages and has bitmap summary entries to summarize the information in the bitmap. Bitmap summary pages can be further summarized into secondary summary pages. The tree data structure can continue to N levels until a topmost level has one bitmap summary page.

Scalable space management using bitmap summary tree

System and method for supporting a user-level file system transaction use cloned files of original existing files specified in the user-level file system transaction to perform file system operations on at least one of the original existing files using the cloned files, which include metadata of the original existing files. After at least one of the cloned files have been modified, a batch rename operation is executed to atomically commit every modified cloned file of the user-level file system transaction.

System and method of supporting user level file system transactions using batch rename and file clones

Exemplary methods, apparatuses, and systems include a controller node receiving a request to perform a consistency check of a distributed file system. The controller node transmits, to each of a plurality of nodes, a request for the node to use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata, determine which of the plurality of nodes own actual portions of the physical metadata, transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node. The controller node receives a result of the comparison from each of the nodes, aggregates the received results, and generates an error report.

Distributed file system consistency check

System and method for managing storage metadata utilize a metadata data structure containing allocation information of storage blocks of a storage system in which a portion of the metadata data structure that corresponds to a group of the storage blocks can be reserved to a requesting client, which then manages the portion of the metadata data structure using a copy of the portion of the metadata data structure.

Scalable storage space allocation in distributed storage systems

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Production, test, and development deployments of a Hadoop application may be executed using multiple compute clusters and a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Data nodes executing as virtual machines (VMs) for test and development deployments can be linked clones of data nodes executing as VMs for a production deployment to reduce duplicated data and provide a shared storage space.

Multi-tenant production and test deployments of Hadoop

A transaction manager for handling operations on data in a storage system provides a system for executing transactions that uses a versioned tuple cache to achieve fast, abortable transactions using a redo-only log. The transaction manager updates an in-memory key-value store and also attaches a transaction identifier to the tuple as a minor key. Opportunistic locking can be accomplished due to the low cost of aborting transactions.

Abortable transactions using versioned tuple cache

A virtual file system and method for performing virtual file system operations uses a vnode descriptor to access a vnode for a vnode operation. If the vnode is not found in a vnode cache using the vnode descriptor, the vnode is reconstructed using information regarding the vnode found outside of the vnode cache using the vnode descriptor.

Virtual file system with vnode reconstruction capability

A file system uses a B-tree data structure to organize file data. The file system may maintain an index node (mode) representing a file and having entries that map to extents of the file. When the file system detects an index node, through updates, has exceeded a threshold number of extents, the file system converts the file to a copy-on-write (COW) B-tree data structure containing the entries representing the extents of the file. To clone the file, the file system uses copies of the index node and the root node of the COW B-tree data structure.

Fast file clone using copy-on-write B-tree

A sorted key-value store is implemented using a write-back cache maintained in memory, a B-tree data structured maintained in disk, and a logical and physical log for providing transactions. The logical log and write-back cache are used to answer client requests, while dirty blocks in the write-back cache are periodically flushed to disk using the physical log.

Log-structured B-tree for handling random writes

A virtualized computing system for executing a distributed computing application, such as Hadoop, is discussed. The virtualized computing system stores data in a distributed filesystem, such as Hadoop Distributed File System, and processes data using a topology awareness that takes into account the virtualization layer of the virtualized computing system. The virtualized computing system employs locality-related policies, including replica placement policies, replica choosing policies, balancer policies, and task scheduling policies that take advantage of the awareness of the virtualization topology.

Growing community of inventors

San Jose, CA, United States of America

Yunshan Luke Lu