International Business Machines Corporation

Dell Products Inc.

Emc IP Holding Company LLC

Yosef Shatsky

Doron Tal

Asaf Porat-Stoler

Danny Harnik

Sergey Marenkov

Ben Sasson

Jonathan Fischer-Toubol

Rivka Mayraz Matosevich

Ety Khaitzin

Afief Halumi

David Darden Chambliss

Chaim Koifman

Joseph S Glider

Aviv Caro

Reut Cohen

Tom Sivan

Avraham Bab-Dinitz

Jonathan Amit

Dmitry Sotnikov

Ori Shalev

Yifat Kuttner

Dorit Hakmon

Yuval Berger

Amitai Alkalay

Ofer Leneman

Ali Aiouaz

Michael Moser

Shai Ishaya Tahar

Moshik Hershcovitch

Omer Haklay

Dmitry Sotniknov

Ury Matarazzo

A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Performance of dispersed location-based deduplication

An apparatus comprises at least one processing device configured to receive unmap requests for freeing up data previously written to one or more storage regions of at least one storage device of a distributed storage system, the unmap requests being received from two or more write cache instances of two or more storage nodes of the distributed storage system. The processing device is also configured to identify at least a subset of the unmap requests which are directed to a given storage region of the at least one storage device. The processing device is further configured to provide the subset of the unmap requests to at least one storage controller associated with the at least one storage device of the distributed storage system responsive to determining that at least one designated unmap condition has been met.

Orchestration of unmap requests received from write cache instances of a distributed storage system

An apparatus comprises at least one processing device configured to determine a given portion of metadata associated with a given one of a plurality of processing groups, the given processing group being assigned to a given one of a plurality of storage nodes of a data storage system. The at least one processing device is also configured to generate two or more copies of the given portion of the metadata, and to select two or more of the plurality of storage nodes of the data storage system on which to store the two or more copies, the two or more storage nodes being selected based at least in part on the assignment of the given processing group to the given storage node. The at least one processing device is further configured to store the two or more copies of on the selected two or more storage nodes.

Distribution of copies of metadata associated with processing groups among storage nodes of a data storage system

Techniques are provided to facilitate access to fragmented snapshot data. For example, a storage control system generates a snapshot data structure of a storage volume. The snapshot data structure comprises plurality of nodes comprising a volume node and one or more snapshot nodes, the volume node is configured to store new and updated data that is written to logical offsets of the storage volume, and the one or more snapshot nodes comprise point-in-time copies of data of the storage volume taken at different times. The storage control system adds an entry for the volume node in a metadata structure associated with the snapshot data structure, wherein the entry comprises a reference to a physical location of data held by a given one of the snapshot nodes at a given logical offset of the storage volume.

Facilitating access to fragmented snapshot data

Techniques are provided for managing a dynamic reserve capacity of a block storage system. A storage control system partitions a storage capacity of at least one storage device into a plurality of blocks for storing data. A data access operation is performed by the storage control system which results in invalidating at least a portion of data stored in a given block. In response to the invalidating, the storage control system determines a degree of invalid data in the given block, and compares the determined degree of invalid data to a utilization threshold. The storage control system designates the given block as fully utilized in response to determining that the degree of invalid data does not exceed the utilization threshold, and designates a capacity of the given block that is occupied by the invalid data to be part of a reserve capacity of the at least one storage device.

Dynamic reserve capacity in storage systems

A data deduplication process is performed in a storage system which includes storage nodes, and storage control nodes which can access data directly from each storage node. A first storage control node sends a message to a second storage control node to initiate a deduplication process with respect to a given data block and an original data block owned by the second storage control node. The second storage control node increments a reference counter associated with the original data block, and sends a message to the first storage control which includes metadata. The first storage control node uses the metadata to read the original data block from a given storage node, performs a data compare process to determine whether the given data block matches the original data block, and creates a reference to the original data block, if the given data block matches the original data block.

Data deduplication in a disaggregated storage system

An apparatus comprises a processing device configured to receive a request to perform merging in a snapshot mapping structure comprising objects representing a given storage volume and its snapshots. The objects are organized in a tree structure representing data dependencies between the given storage volume and its snapshots. The processing device is also configured to analyze the snapshot mapping structure to identify ones of the objects that are candidates for deletion and to determine merge groups. A given one of the merge groups includes (i) at least a subset of the objects identified as candidates for deletion and (ii) a given object that is a child object of at least one of the subset of the objects identified as candidates for deletion. The processing device is further configured to perform merging in the snapshot mapping structure by merging data items in the given merge group to one of its objects.

Merging objects in a snapshot mapping structure characterizing data dependencies between a storage volume and its snapshots

An apparatus comprises a processing device configured to receive a write request to write a given portion of data to a storage system comprising a multiple-instance write cache, the multiple-instance write cache comprising a first write cache instance that utilizes replica-based data protection and a second write cache instance that utilizes data striping-based data protection, and to determine a size of the given data portion and to compare the size of the given data portion to at least one size threshold. The processing device is also configured, responsive to a first comparison result, to write the given data portion to the first write cache instance. The processing device is further configured, responsive to a second comparison result different than the first comparison result, to write at least part of the given data portion to the second write cache instance.

Multiple-instance write cache for a storage system

Techniques are provided for implementing and managing a multi-modal write cache for a data storage system. For example, a storage control system is configured to perform a write caching method which comprises the storage control system receiving an input/output (I/O) write request from a client application to write data to a primary storage volume, comparing a current I/O workload associated with the client application to an I/O workload threshold, and writing the data of the I/O write request to one of (i) a persistent write cache in a persistent storage volume and (ii) a non-persistent write cache in a non-persistent storage volume, based at least in part on a result of comparing the current I/O workload to the I/O workload threshold.

Multi-modal write cache for data storage system

Techniques are provided for managing metadata of a storage system. A storage control system manages a storage metadata structure which comprises metadata items for accessing data items in a storage system comprising a persistent write cache and a primary storage. The storage metadata structure comprises a first metadata structure that is configured to organize metadata items received from the write cache at a first granularity, and a second metadata structure that is configured to organize metadata items received from the first metadata structure at a second granularity, wherein the second granularity is greater than the first granularity. The storage control system utilizes the storage metadata structure to access data items in the persistent write cache and the primary storage.

Managing granularity of a metadata structure for a storage system

A storage control system receives a request to access target data and searches a primary metadata structure to find an associated metadata entry. The primary metadata structure comprises a log-structured merge tree comprising a first level of segments which comprises a group of segments having an associated group filter, and individual filters associated with respective segments of the group of segments. The storage control system searches the primary metadata structure by performing a lookup operation using the group filter to determine whether the associated metadata entry is potentially present in a given segment of the group of segments, and performing a lookup operation using the individual filters to identify the given segment of the group of segments which potentially includes the metadata entry, when the metadata entry is determined to be potentially present in a given segment of group of segments, using the group filter.

Managing lookup operations of a metadata structure for a storage system

A storage control system manages a storage metadata structure which comprises first and second tree structures. The first tree structure is configured to accumulate metadata entries associated with newly written data items, and sort the accumulated metadata entries by index keys. The second tree structure is configured to organize metadata entries using an index structure that enables random-access to the metadata entries using the index keys. The storage control system performs a merging process to merge metadata entries in leaf levels of the first and second tree structures, and performs a tree construction process to construct a third tree structure by populating a leaf level of the third tree structure with merged metadata entries from the leaf levels of the first and second tree structures. The storage metadata structure is updated to comprise the first tree structure, and the third tree structure in place of the second tree structure.

Managing insert operations of a metadata structure for a storage system

A garbage collection process is performed in a storage system which comprises a storage control node, and storage nodes which implement a striped volume comprising a plurality of stripes having strips that are distributed over the storage nodes. The storage control node selects a victim stripe for garbage collection, and an empty stripe in the striped volume. The storage control node determines a data strip of the victim stripe having predominantly valid data based on a specified threshold, and sends a copy command to a target storage node which comprises the predominantly valid data strip, to cause the target storage node to copy the predominantly valid data strip to a data strip of the empty stripe which resides on the target storage node. The storage control node writes valid data blocks of the victim stripe to remaining data strips of the empty stripe, and releases the victim stripe for reuse.

Optimization for garbage collection in a storage system

An apparatus comprises a processing device configured to maintain a snapshot mapping structure for a storage volume. The snapshot mapping structure comprises objects representing the storage volume and snapshots thereof. The objects are organized in a tree structure representing data dependencies between the storage volume and its snapshots. Each of the objects comprises a dynamic bitmap indicating whether that object contains new data for different offset positions relative to its ancestor objects in the tree structure. The processing device is further configured to detect operations directed to a specified one of the storage volume or one of its snapshots, to parse the snapshot mapping structure to identify objects to utilize for processing the operations, and to processing the operations utilizing the dynamic bitmaps of the identified objects.

Snapshot mapping structure characterizing data dependencies between a storage volume and its snapshots

A storage control system receives a first data block to be written to a primary storage, and generates a content signature for the first data block. The storage control system adds a first entry for the first data block into a persistent deduplication database. The first entry comprises a key which comprises the content signature for the first data block. The persistent deduplication database comprises a tree data structure which comprises elements that are configured to store entries for data blocks. The storage control system merges the entries of at least two elements of the tree data structure to generate a set of merged entries which comprises the first entry for the first data block, and a second entry for a second data block, and commences a deduplication process in response to determining that the first entry and the second entry in the set of merged entries have matching keys.

Growing community of inventors

Karnei Shomron, Israel

Yosef Shatsky