Vmware, Inc.

Apple Inc.

Other

Wenguang Wang

Richard Paul Spillane

Vamsi Gunturu

Junlong Gao

Eric Shannon Knauft

Maxime Austruy

Enning Xiang

Christos Karamanolis

Yunshan Luke Lu

Abhay Kumar Jain

Abhishek Gupta

Pascal Renauld

Kapil Chowksey

Jorge Guerra Delgado

Srinath Premachandran

David Alexander Majnemer

Satish Pudi

Pranay Singh

Petr Vandrovec

Robert T Johnson

Tao Xie

Rob T Johnson

Sandeep Rangaswamy

Deric S Horn

Ilia Sokolinski

Radu Berinde

Mounesh Badiger

Vamsidhar Gunturu

Sriram Patil

Hardik Singh Negi

Hong Li

Marcos Kawazoe Aguilera

Adrian Drzewiecki

Asit Desai

Ilya Languev

Ye Zhang

Ruiling Dou

Haoran Zheng

Zongliang Li

Nitin Rastogi

Owen Strain

Jin Zhang

Jin Feng

Zhaohui Guo

Luke Lu

Long Yang

Subhradyuti Sarkar

Zhao Xia Jin

Kiran R Joshi

Ilia Langouev

Matthew Benjamin Amdur

Christoph Klee

Xiang Yu

Mansi Shah

Xin Liang Li

Yu Wu

Christian Dickmann

John Garvey

Rawlinson Rivera

Bryan Fink

Kevin Rayfeng Li

Jiaqi Zuo

Chandrakanth Gadhiraju

Matt Amdur

Xinyu Zhang

Blake H Lewis

He Wang

Li Ding

Joseph Sokol, Jr

Varun A Shah

Mark Steven Day

Ashish Kaila

Zhen Yu Liu

Abhishek Srivastava

Edward Joseph Goggin

Zhou Huang

Julien Freche

Jyothir Ramanan

Gopala Suryanarayana

Sifan Liu

Radhika Vullikanti

Puja Dilip Gupta

Yangyang Zhang

Banghui Luo

Yifan Wang

Tian Luo

Ruocheng Li

Yizheng Chen

Kamal Jeet Charan

HaiFeng Li

Xin Li

Wangping He

Xiaoyun Gong

Rick Spillane

Pushpesh Sharma

Peter Macko

Xiaoxuan Meng

Jeffrey D Chung

Christopher Yoshito Emura

Aspects of the disclosure generate independent encryption keys for objects (e.g., virtual machine disks (VMDKs)) without requiring the management of multiple keys. An encryption manager obtains a primary encryption key, an object identifier (ID) comprising a globally unique ID (GUID) for an object, a data salt comprising the object ID and a data salt string, and a metadata salt comprising the object ID and a metadata salt string. A data encryption key is generated using the primary encryption key, the data salt, and a one-way function. A metadata encryption key is generated using the primary encryption key, the metadata salt, and the one-way function. Because the data salt string and metadata salt string differ, the data encryption and metadata encryption keys differ. Object IDs for different objects differ, so each object and its metadata have globally unique keys. Key generation (other than the primary key) is deterministic, simplifying key management.

Generating independent encryption keys for storage objects

A method for encryption in a distributed datastore is provided. The method generally includes receiving random data from a virtualization management platform as a wrapped data encryption key (DEK), retrieving a key encryption key (KEK) from a key management server (KMS), decrypting the wrapped DEK using the KEK to determine a blank DEK, where the blank DEK is common to a plurality of hosts of a host cluster accessing a distributed datastore; encrypting first data using the blank DEK, and storing the encrypted first data in one or more disks of the distributed datastore, the one or more disks belonging to the plurality of hosts.

Encryption in a distributed storage system utilizing cluster-wide encryption keys

The disclosure herein describes deduplicating data chunks using chunk objects. A batch of data chunks is obtained from an original data object and a hash value is calculated for each data chunk. A first duplicate data chunk is identified using the hash value and a hash map. A chunk logical block address (LBA) of a chunk object is assigned to the duplicate data chunk. Payload data of the duplicate data chunk is migrated from the original data object to the chunk object, and a chunk map is updated to map the chunk LBA to a physical sector address (PSA) of the migrated payload data on the chunk object. A hash entry is updated to map to the chunk object and the chunk LBA. An address map of the original data object is updated to map an LBA of the duplicate data chunk to the chunk object and the chunk LBA.

Deduplicating data chunks using chunk objects

The disclosure herein describes converting a disk cluster to a different format. A format conversion instruction associated with a disk cluster is received. A first subgroup of disks of the disk cluster that are the emptiest disks of the disk cluster are identified and all data is evacuated from the first subgroup of disks to other disks of the disk cluster. The first subgroup of disks is reformatted based on the received format conversion instruction. A group of data objects stored in the disk cluster is converted based on the format conversion instruction and the converted group of data objects are written to the reformatted first subgroup of disks. The process iterates through the disks of the disk cluster to reformat all disks and convert all data objects based on the received format conversion instruction. The process reduces the write operations required to convert the format of the disk cluster.

Converting the format of a distributed object storage with reduced write operations

Linked clone read performance when retrieving data from a clone is improved at least by aggregating block mapping metadata efficiently. Primary metadata for a child clone maps a logical block address (LBA) for data in a data region of the child clone to a physical sector address (PSA) for data in the data region of the child clone. At least a portion of primary metadata for a parent clone of the child clone is copied into archival metadata for the child clone. In response to a read request, data is returned from the child clone, parent clone, or another ancestor of the child clone based on whether or not a read request LBA is within the primary metadata for the child clone, or within the archival metadata.

Aggregating block mapping metadata to improve linked clone read performance

The present disclosure relates to bootstrapping an encrypted single node VSAN cluster. One method includes receiving a request to create an encrypted VSAN cluster from a single host in a software-defined datacenter, deploying a virtual server on a VSAN datastore of the software-defined datacenter, registering a native key provider (NKP) in the virtual server, creating an empty VSAN cluster encrypted by the NKP, adding the single host to the encrypted empty cluster to create a one-host encrypted cluster, registering a KMIP KMS in the virtual server, switching encryption of the one-host encrypted cluster from the NKP to the KMIP KMS, and adding another host to the one-host encrypted cluster to create the encrypted cluster.

Bootstrapping an encrypted single node VSAN cluster

Example methods and systems for accessing data in a log-structured file system having a plurality of snapshots of storage objects backed by a first-level copy-on-write (COW) B+ tree data structure and a plurality of second-level B+ tree data structures have been disclosed. One example method includes obtaining a first first-level mapping associated with a first snapshot from the plurality of snapshots based on a first logical block address, wherein each of the plurality of snapshots corresponds to each of the plurality of second-level B+ tree data structures, identifying a first second-level B+ tree data structure corresponding to one of the plurality of snapshots based on the first first-level mapping, obtaining a first second-level mapping based on the first logical block address in the first second-level B+ tree data structure, obtaining a first physical block address based on the first second-level mapping, and accessing data at the first physical block address.

Two-level logical to physical mapping mechanism in a log-structured file system

A method for efficient write-back for journal truncation is provided. A method includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction associated with one or more pages in an ordered data structure and maintaining a dirty list including an entry for each page indicated by a record in the journal. Each entry in the dirty list includes a respective first log sequence number (LSN) associated with a least recent record of the plurality of records that indicates the page and a respective second LSN associated with a most recent record of the plurality of records that indicates the page. The method includes determining to truncate the journal. The method includes identifying one or more records, of the plurality of records, from the journal to write back to a disk, where the identifying is based on the dirty list.

Efficient write-back for journal truncation

A method for efficient journal truncation is provided. A method for journal truncation includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction in an ordered data structure. The method includes maintaining a truncation queue in the memory including one or more entries. Each entry in the truncation queue includes a physical on-disk offset associated with a different record of the plurality of records. The method includes determining to truncate the journal and truncating records, of the plurality of records, from the journal starting from a beginning record in the journal up to the record with the physical on-disk offset associated a least recent entry of the one or more entries in the truncation queue, where the truncating includes removing the records from the memory.

Efficient incremental journal truncation policy

A method for modifying key-value pairs of a B+ tree is provided. The method receives a request to modify a particular key-value pair. Each node of the tree has a modification number. The method traverses a path on the tree from the root node toward the particular node. The traversing includes upon reaching a parent node of the path, acquiring a shared lock on both the parent node and a child node one level below the parent node. Upon determining that the child node is the particular node, the method stores the modification number of the particular node, releases the shared lock on the particular node, compares a current modification number of the node with its stored number, and acquires an exclusive lock on the node if the numbers are the same. The method increments the current modification number of the node and modifies it while in the exclusive lock.

Enhanced locking mechanism for B+ tree data structures

Components of a distributed data object are synchronized using streamlined tracking metadata. A target component of the distributed data object is detected as it becomes available and stale. A source component that is up-to-date and that mirrors the address space of the detected target component is identified. A set of mapped address ranges and a set of unmapped address ranges of the identified source component are obtained. A mapped address range of the target component that corresponds with an unmapped address range of the source component is identified. The identified mapped address range of the target component is then synchronized with the corresponding unmapped address range of the source component. Thus, unmapped address ranges are synchronized without using tracking metadata of the source component.

Efficient unmap handling using synchronized redundant components of data objects

Linked clone read performance (e.g., retrieving data) is improved at least by minimizing the number of input/output (I/O) operations. For a child clone, a local logical extent and an inherited logical extent are generated. The local logical extent comprises a logical block address (LBA) for data in a data region of the child clone and a physical sector address (PSA) corresponding to the LBA for the data in the data region of the child clone. The inherited logical extent spans logical extents that are accessible to the child clone. The inherited logical extent comprises an LBA for data in a data region of an ancestor of the child clone and a corresponding identifier (ID) of the ancestor. Data for an LBA in a read request may be rapidly found in the child clone (local logical extent) or an ancestor (inherited logical extent).

A method for managing replication of cloned files is provided. Embodiments include determining, at a source system, that a first file has been cloned to create a second file. Embodiments include sending, from the source system to a replica system, an address of the first extent and an indication that a status of the first extent has changed from non-cloned to cloned. Embodiments include changing, at the replica system, a status of a second extent associated with a replica of the first file on the replica system from non-cloned to cloned and creating a mapping of the address of the first extent to an address of the second extent on the replica system. Embodiments include creating, at the replica system, a replica of the second file comprising a reference to the address of the second extent on the replica system.

Efficient replication of file clones

The disclosure describes growing a data cache using a background hash bucket growth process. A first memory portion is allocated to the data buffer of the data cache and a second memory portion is allocated to the metadata buffer of the data cache based on the cache growth instruction. The quantity of hash buckets in the hash bucket buffer is increased and the background hash bucket growth process is initiated, wherein the process is configured to rehash hash bucket entries of the hash bucket buffer in the increased quantity of hash buckets. A data entry is stored in the data buffer using the allocated first memory portion of the data cache and metadata associated with the data entry is stored using the allocated second memory portion of the metadata buffer, wherein a hash bucket entry associated with the data entry is stored in the increased quantity of hash buckets.

Dynamic growth of data caches using background processes for hash bucket growth

The present disclosure provides techniques for deduplicating files. The techniques include creating a cache or subset of a large data structure. The large data structure organizes information by random hash values. The random hash values result in a random organization of information within the data structure, with the information spanning a large number of storage blocks within a storage system. The cache, however, is within memory and is small relative to the data structure. The cache is created so as to contain information that is likely to be needed during deduplication of a file. Having needed information within memory rather than in storage results in faster read and write operations to that information, improving the performance of a computing system.

Small in-memory cache to speed up chunk store operation for deduplication

Growing community of inventors

Santa Clara, CA, United States of America

Wenguang Wang