International Business Machines Corporation

Micro Technology, Inc.

Mti Technology Corporation

Emc Corporation

Sf2 Corporation

MIT Technology Corp.

Joseph S Glider

David Darden Chambliss

Thomas E Idleman

Mihail Corneliu Constantinescu

David H Jaffe

David T Powers

Alessandro Sorniotti

Maohua Lu

Danny Harnik

Elli Androulaki

Nathalie Baracaldo

Wendy Ann Belluomini

Ety Khaitzin

Himabindu Pucha

Larry P Henson

Jonathan Amit

Yosef Shatsky

Kumar Gajjar

Kaushik S Shah

Edward E Asato

Randy H Katz

Luis Alfonso Lastras-Montano

Bulent Abali

Michael Edward Factor

Rui Zhang

Chaim Koifman

Elliot Karl Kolodner

Bhushan Pradip Jain

Aviv Caro

Dalit Naor

Cameron James Mcallister

Demyn Lee Plantenberg

Hoke S Johnson, Iii

Robert S Koontz

Dilip N Simha

Jorge Guerra Delgado

Carlos Francisco Fuente

Henry Esmond Butterworth

Paula Kim Ta-Shma

James C Chen

Sivan Tal

Jonathan Fischer-Toubol

Michael Moser

Juan C Gomez

Eran Rom

Lloyd R Shipman, Jr

Shai Ishaya Tahar

Yifat Kuttner

Jesse I Stamness

Lawrence Yium-Chee Chiu

Corneliu Mihail Constantinescu

Duc H Trang

Hoke S Johnson Iii

Tal Sivan

Computer program products, as well as corresponding systems and methods are configured for performing deduplication in conjunction with random read and write operations, and include: computing a fingerprint of data included in a write request; determining whether a short term dictionary comprises an entry corresponding to the fingerprint; in response to determining the short term dictionary comprises the entry corresponding to the fingerprint, writing the data to a data store in a deduplicating manner; in response to determining the short term dictionary does not comprise the entry, determining whether a long term dictionary corresponding to the namespace comprises the entry; in response to determining the long term dictionary comprises the entry, writing the data to the data store in the deduplicating manner; and in response to determining the long term dictionary does not comprise the entry, writing the data to the data store in a non-deduplicating manner.

Region-integrated data deduplication implementing a multi-lifetime duplicate finder

In one embodiment, a deduplicating storage system includes a processor and logic integrated with and/or executable by the processor. The logic is configured to cause the processor to perform a method which includes: computing a fingerprint of a data chunk, and determining whether a short term dictionary corresponding to the namespace comprises an entry corresponding to the fingerprint. In response to determining the short term dictionary does not comprise the entry, a determination is made whether a long term dictionary corresponding to the namespace comprises the entry. In response to determining the long term dictionary comprises the entry: the data chunk is written to the data store in the deduplicating manner, and the short term dictionary is repopulated with the entry. Moreover, in response to determining the long term dictionary does not comprise the entry, the data chunk is written to the data store in a non-deduplicating manner.

Embodiments relate to processing streams of encrypted data received from multiple users. A received encrypted data stream is separated into one or more encrypted data chunks, placing the data chunk(s) into a sub-stream, and decrypting the data chunks into plaintext. One or more advanced data functions are applied to the plaintext, thereby effectively transforming the plaintext. The transformed plaintext is organized into one or more data units, and each data unit is encrypted with a wrapped encryption key. The aspect of encrypting the data unit includes creating a fixed size encryption unit, whereby the wrapped encryption key comprises a master key and a private key.

Receipt, data reduction, and storage of encrypted data

Computer program products, as well as corresponding systems and methods are configured for performing deduplication in conjunction with random read and write operations, and include: receiving a write request comprising data; computing a fingerprint of the data; determining whether a short term dictionary comprises an entry corresponding to the fingerprint; in response to determining the short term dictionary comprises the entry corresponding to the fingerprint, writing the data to a data store in a deduplicating manner; in response to determining the short term dictionary does not comprise the entry, determining whether a long term dictionary corresponding to the namespace comprises the entry; in response to determining the long term dictionary comprises the entry, writing the data to the data store in the deduplicating manner; and in response to determining the long term dictionary does not comprise the entry, writing the data to the data store in a non-deduplicating manner.

In one embodiment, a computer-implemented method for performing deduplication in conjunction with random read and write operations across a namespace that is divided into a plurality of disjoint regions is disclosed. The method includes: maintaining a metadata structure for each of the plurality of disjoint regions via the respective region manager, each metadata structure comprising metadata indicating a physical storage location of one or more data chunks associated with the respective region; and performing a deduplicated write operation. The deduplicated write operation includes: determining whether the disjoint regions comprise a second data chunk identical in content to a first data chunk; and in response to determining one or more of the plurality of disjoint regions other than the first region includes the second data chunk, establishing a reference from the first data chunk to the second data chunk. Corresponding systems and computer program products are also disclosed.

Region-integrated data deduplication

Embodiments relate to avoiding out-of-space conditions in storage controllers operating with efficiency capabilities between virtual space in a data container and real space in a storage container. Both the real space and the virtual space are monitored. The real space usage is compared to a threshold to provide information about occupancy of the real space. A virtual size of the virtual object is adjusted responsive to the real space usage meeting or exceeding the threshold.

Avoiding out-of-space conditions in storage controllers

In one embodiment, a computer program product is configured for performing deduplication in conjunction with random read and write operations across a namespace divided into a plurality of disjoint regions. The computer program product includes a computer readable storage medium having program instructions embodied therewith, where the computer readable storage medium is not a transitory signal per se. The program instructions are executable by a computer to cause the computer to perform a method including: maintaining a metadata structure for each of the plurality of disjoint regions via the respective region manager(s) of the plurality of disjoint regions, each metadata structure comprising metadata indicating a physical storage location of one or more data chunks associated with the respective region; and performing, by the computer, a deduplicated write operation of a first data chunk in a first region of the plurality of disjoint regions.

Embodiments relate to deduplication and compression on data performed downstream from where the data is encrypted. Confidentiality of data is maintained, and the ability of storage systems to perform data reduction functions is supported. Encrypted data to be written to a storage system is separated into one or more data chunks. For a data chunk, a master encryption key for an owning entity associated with the data chunk is retrieved. The data chunk is decrypted into plaintext, and the plaintext is transformed by performing one or more advanced data functions. A private key is created and used to encrypt the transformed plaintext, which is stored as a first encryption unit. A wrapped key is created by encrypting the private key with the master key, limits data access to the owning entity, and is stored as metadata for the encryption unit.

Shared data encryption and confidentiality

Various embodiments for identifying data in a data deduplication system, by a processor device, are provided. In one embodiment, a method comprises efficiently identifying duplicate data in the data deduplication system by identifying fingerprint matches using a direct inter-region fingerprint lookup to search for the fingerprint matches in at least one of a plurality of metadata regions, the direct inter-region fingerprint lookup supplementing a central fingerprint index.

Direct lookup for identifying duplicate data in a data deduplication system

Embodiments of the invention relates to avoiding out-of-space conditions in storage controllers operating with efficiency capabilities between virtual space in a data container and real space in a storage container. Both the real space and the virtual space are monitored and their respective usage is compared to provide information about occupancy of the real space to the virtual space. Usage of the containers is balanced by employing a virtual file associated with a reserved portion of free capacity in the virtual space.

Balancing usage of real and virtual space to avoid out-of-space conditions in storage controllers

Detecting data duplication includes maintaining a fingerprint directory including one or more entries. Each entry includes a data fingerprint and a data location for a data chunk. A shadow list including a record of fingerprint values not contained in the fingerprint directory is maintained. Each entry is associated with a seen-count attribute, which is an indication of how often a data fingerprint has been seen in arriving data chunks to be written in a storage system, and distinguishes multiply-seen entries for data fingerprints present in at least two data chunks from once-seen entries for data fingerprints present in no more than a single data chunk. Each entry retrieved from the shadow list relates to twice-seen fingerprints.

Deduplicating storage with enhanced frequent-block detection

Embodiments relate to processing streams of encrypted data received from multiple users. As a stream is received, smaller partitions in the form of data chunks, including a first data chunk, are created and subject to individual decryption. The first data chunk is placed into sub-stream according to a first master key associated with a first owning entity. Prior to processing, the first data chunk is decrypted into plaintext, and the plaintext is transformed by applying one or more advanced data functions. The transformed plaintext is organized into a first data unit, and a first encryption unit is created from the first data unit. The first encryption unit has a space allocation in persistent storage. Accordingly, confidentiality of data is maintained, and the ability of storage systems to perform data reduction functions is supported.

Data reduction in a storage system comprises determining attributes of data for storage in the storage system and determining expected data reduction effectiveness for the data based on said attributes. Said effectiveness indicates the benefit that data reduction is expected to provide for the data based on said attributes. The data reduction further comprises applying data reduction to the data based on the expected data reduction effectiveness and performance impact, to improve resource usage efficiency.

Enhancing performance-cost ratio of a primary storage adaptive data reduction system

Embodiments relate to deduplication and compression on data performed downstream from where the data is encrypted. Confidentiality of data is maintained, and the ability of storage systems to perform data reduction functions is supported. Encrypted data to be written to a storage system is separated into one or more data chunks. For a data chunk, a master encryption key for an owning entity associated with the data chunk is retrieved. The data chunk is decrypted into plaintext, and the plaintext is transformed by performing one or more advanced data functions. A private key is created and used to encrypt the transformed plaintext, which is stored as a first encryption unit. A wrapped key is created by encrypting the private key with the master key, and is stored as metadata for the encryption unit to limit data access to the owning entity.

Embodiments of the invention relate to efficiently storing encrypted data in persistent storage or passing to another data processing component. A downstream decrypter is utilized to act within the data path between a data generator and a storage server. The decrypter fetches an encryption key and any other necessary auxiliary information necessary to decrypt received data. Following decryption of the data, the decrypter has the ability to operate directly on plaintext and perform storage efficiency functions on the decrypted data. The decrypter re-encrypts the data prior to the data leaving the decrypter for persistent storage to maintain the security of the encrypted data.

Achieving storage efficiency in presence of end-to-end encryption using downstream decrypters

Growing community of inventors

Palo Alto, CA, United States of America

Joseph S Glider