Cohesity, Inc.

Google Inc.

Zhihuan Qiu

Anubhav Gupta

Ganesha Shanmuganathan

Chung Tin Kwok

Mohit Aron

Anirvan Duttagupta

Venkata Ranga Radhanikanth Guturi

Praveen Kumar Yarlagadda

Lei Zhong

Sarthak Agarwal

Malini Mahalakshmi Venkatachari

Chinmaya Manjunath

Sachin Jain

Sidharth Mishra

Apurv Gupta

Shreyas Talele

Yu Liu

Siddesh Shinde

Hyung J Kim

Rajan Patel

Sharath Kumar Naeni

Yinzen Hwang

Deepak Ojha

Prashant Saraswat

Ryan H Moulton

Nandan Paramashiva

Siming Liu

Techniques are disclosed for efficient file system directory reverse name lookup. An example method includes storing, at one or more of a plurality of storage locations of a key map, a file system directory including one or more key-value pairs, wherein each key-value pair of the file system directory maps a respective object identifier to corresponding object metadata; and in response to detecting an indication of a file system event for a file: storing, to a storage location of the key map, a key-value pair that maps an object identifier of the file to object metadata of the file; generating, based on the object identifier, an indication of the storage location; including, by the data platform, the indication of the storage location in the object metadata; and storing, by the data platform, the object metadata to a data store.

Efficient file system directory reverse name lookup

Techniques are described for fast large file truncation. An example method includes receiving, by a data platform, a request to truncate a file, wherein data of the file is stored in a tree data structure including a plurality of nodes with leaf nodes corresponding to data of the file; determining a first node of the plurality of nodes including a plurality of child nodes; determining, based on a maximum key, a subset of the child nodes corresponding to a portion of the data that is to be retained; based on determining not to traverse the first node, updating a second node that is a parent of the first node to include a pointer to each child node of the subset of child nodes; and moving, from a subset of leaf nodes referenced by the subset of child nodes, a leaf node to rebalance the tree data structure.

Fast truncation of large files

Techniques are described for performing direct archive of data chunkfiles. A computing system comprising a storage device and processing circuitry having access to the storage device may be configured to perform various aspects of the techniques. The processing circuitry may be configured to predict an incoming data rate of native format to be archived to obtain a predicted incoming data rate for the native format data to be archived, and compare the predicted incoming data rate to a first threshold data rate. The processing circuitry may also be configured to, responsive to determining that the predicted incoming data rate exceeds a first threshold data rate, segment the native format data into chunks, and directly write the chunks to an archive storage system as a chunkfile.

Adaptive archive of data chunkfiles

A specification of content to be stored in a cloud storage is received at a client-side component. A first portion of the content is divided into a plurality of data chunks. One or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage are identified. It is determined whether a batch size of the one or more identified data chunks does not meets a threshold size. One or more data chunks of a second portion of the content that are to be stored in the cloud storage are identified. It is determined that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size. It is determined that a batch period is greater than or equal to a batch threshold period. The second batch of data chunks is written to a storage of a cloud server included in a data plane.

Efficiently storing data in a cloud storage

Maintenance is performed to determine one or more content files associated with a stored backup snapshot having a size that is greater than a threshold size. It is determined that the size of a content file of the one or more content files is greater than the threshold size. In response to determining that the size of the content file is greater than the threshold size, a new tree data structure is generated and a component file metadata structure corresponding to the content file is split into a plurality of component file metadata structures for the content file.

Large content file optimization

A plurality of portions of a content file are stored. It is determined that the content file has a size that is greater than a threshold size. In response to determining that the content file has the size that is greater than the threshold size, a plurality of component file metadata structures are generated for each of the plurality of portions of the content file. A component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file. Each of the plurality of component file metadata structures includes corresponding metadata that enables data chunks associated with a corresponding portion of the content file to be located.

Data associated with a source system is ingested. After the data is ingested, a post-processing metadata conversion process is performed including by selecting an entry of a chunk metadata data structure and determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects. In response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects, metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk is updated to store a reference to a chunk file storing the data chunk and the selected entry is removed from the chunk metadata data structure.

Reducing memory usage in storing metadata

A cloud server component determines that a size of a first cloud storage element object is at least below a first threshold. In response to the first determination, a client-side component is requested to store additional data in the cloud storage element object including by having the client-side component update the first cloud storage element with an updated version that includes previously existing data of the first cloud storage element and the additional data. The first cloud storage element object is added to a set of one or more cloud storage element objects available for update. The client-side component is configured to generate an updated version of the first cloud storage element object that has a size that is greater than or equal to the first threshold.

A first group of chunk identifiers associated with a first content identifier structure of a first metadata element and a second group of chunk identifiers associated with a second content identifier structure of a second metadata element are determined. A common sequence of chunk identifiers across at least a portion of the first group of chunk identifiers associated with the first content identifier structure and the second group of chunk identifiers associated with the second content identifier structure is determined. A portion of the first group of chunk identifiers associated with the first content identifier structure and a portion of the second group of chunk identifiers associated with the second content identifier structure is updated to reference a common sequence identifier in place of the determined common sequence of chunk identifiers.

Deduplicating metadata based on a common sequence of chunk identifiers

A selected data chunk associated with an object is determined to be sent to a destination. A chunk compression grouping storing the selected data chunk associated with the object is identified. The identified chunk compression grouping includes a plurality of data chunks compressed together. A data content version that includes the selected data chunk associated with the object to be provided to the destination is determined from a plurality of data content versions based at least in part on a metric associated with the identified chunk compression grouping.

Adaptively providing uncompressed and compressed data chunks

Data is ingested from a source system. Ingesting the data includes determining corresponding chunk identifiers for a plurality of data chunks corresponding to the ingested data and for each of the plurality of data chunks, verifying whether the corresponding chunk identifier is included in a data structure tracking identifiers of data chunks that were already stored in a storage of a storage system before the data ingestion started and storing the data chunk in a storage based on the verification. After the ingesting is completed, deduplication of the ingested data chunks stored in the storage having a same chunk identifier is performed and the data structure is updated based on the deduplication.

Partial in-line deduplication and partial post-processing deduplication of data chunks

An indication to perform a backup of file system data is received. The file system data includes a content file having a size greater than a threshold size. A plurality of component file metadata structures for the content file are generated. File metadata of the content file split across the plurality of component file metadata structures is stored. The plurality of component file metadata structures are associated with different portions of the content file. A component file metadata structure of the plurality of component file metadata structures stores file metadata corresponding to a portion of the content file. The file metadata corresponding to the portion of the content file includes one or more references to locations of data chunks associated with the portion of the content file.

A specification of content to be stored in a cloud storage is received at a client-side component. A portion of the content is divided into a plurality of data chunks. One or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage are identified. It is determined whether a batch size of the one or more identified data chunks meets a threshold size. Based on the determination of whether the batch size meets the threshold size, a cloud storage destination among a plurality of different cloud storage destinations associated with different performance tiers is selected.

Data is ingested from a source system including by storing a plurality of data chunks in one or more chunk files and storing corresponding chunk identifiers associated with the plurality of data chunks in a first data structure. After data ingestion is complete, one or more duplicate data chunks that were stored during the data ingestion are determined and a second data structure is updated to include one or more entries corresponding to one or more determined duplicate data chunks.

A virtual machine container file is analyzed to determine which portion of the virtual machine container file corresponds to a virtual machine file system metadata of the virtual machine container file. One or more differences between a first version of a virtual machine container file and a second version of the virtual machine container file are determined at least in part by traversing a snapshot structure associated with the virtual machine container file. The determined one or more differences that corresponds to the virtual machine file system metadata portion of the virtual machine container file are identified based at least in part on the analysis of the virtual machine container file. The identified one or more differences corresponding to the virtual machine file system metadata portion of the virtual machine file are utilized to identify one or more changes from the content files included in the first version of the virtual machine container file to content files included in the second version of the virtual machine container file.

Growing community of inventors

Santa Clara, CA, United States of America

Zhihuan Qiu