Other

Quantum Corporation

Andrew Leppard

Timothy Stoakes

Alan Kennington

Apparatus, methods, and other embodiments associated with object synthesis are described. One example apparatus includes logic for identifying a block in a data de-duplication repository and for identifying a reference to the block. The apparatus also includes logic for representing a source object using a first named, organized collection of references to blocks in the data de-duplication repository and logic for representing a target object using a second named, organized collection of references. The apparatus is configured to synthesize the target object from the source object. Since synthesis may be complicated by edge cases, the apparatus is configured to account for conditions including a block in the target object needing less than all the data in a source object block, data to be used to synthesize the target object residing in a sparse hole in a data stream, and the target object needing data not present in the source object.

Object synthesis

Methods and systems for protecting de-duplication repositories against a malicious attack are disclosed. One method receives at least one block of data to store in a data storage system. A de-duplication engine comprising a secret key is utilized to generate a secret key hash of the at least one block of data. A comparison of the secret key hash of the at least one block of data with a secret key hash table of previously stored data on the data storage system to identify duplicated data, the secret key hash comparing protecting the data storage system against a malicious attack.

Protecting de-duplication repositories against a malicious attack

Example systems and methods concern a sub-block parser that is configured with a variable sized window whose size varies as a function of the actual or expected entropy of data to be parsed by the sub-block parser. Example systems and methods also concern a sub-block parser configured to compress a data sequence to be parsed before parsing the data sequence. One example method facilitates either actually changing the window size or effectively changing the window size by manipulating the data before it is parsed. The example method includes selectively reconfiguring a data set to be parsed by a data-dependent parser based, at least in part, on the entropy level of the data set, selectively reconfiguring the data-dependent parser, based, at least in part, on the entropy level of the data set, and parsing the data set.

Manipulating the actual or effective window size in a data-dependant variable-length sub-block parser

Apparatus, methods, and other embodiments associated with blocklet pattern identification are described. One example method includes accessing a blocklet produced by a computerized data de-duplication parsing process before providing the blocklet to a duplicate blocklet determiner. The example method also includes hashing a portion of the blocklet to produce a pattern indicating hash and then identifying the blocklet as a pattern blocklet if the pattern indicating hash matches a pre-determined pattern indicating hash. To improve efficiency in a data de-duplication process, the blocklet pattern identifying may be performed independently from a data structure and process used by the duplicate blocklet determiner. If the blocklet is a pattern blocklet, then the method includes selectively controlling the duplicate blocklet determiner to not process the pattern blocklet. The duplicate determination is not needed because a pattern determination has already been made.

Blocklet pattern identification

An embodiment for minimizing damage caused by corruption of de-duplicated data is disclosed. One embodiment counts the number of references to a data segment indexed by a de-duplication index. When a reference count for the data segment reaches a first threshold a duplicate of the data segment is generated. In addition, the duplicate of the data segment is also stored to minimize damage of the de-duplicated data caused by a corruption of the data segment.

Minimize damage caused by corruption of de-duplicated data

A method of identifying application metadata in a backup stream comprises finding a known application metadata signature within a backup stream. A source application of user data in the backup stream is identified based upon the known application metadata signature. Application metadata, which has been generated by the source application, is removed from consideration by a data de-duplicator which is used to perform data de-duplication on the backup stream.

Identifying application metadata in a backup stream

In one embodiment, a method includes accessing data; partitioning the data into sub-blocks; determining whether a first one of the sub-blocks is identical to another one of the sub-blocks or similar to another one of the sub-blocks; if the first one of the sub-blocks is identical to another one of the sub-blocks, applying by the one or more computer systems hash-based deduplication to storage of the first one of the sub-blocks with respect to the other one of the sub-blocks; and, if the first one of the sub-blocks is similar to another one of the sub-blocks, applying by the one or more computer systems sub-block differencing to storage of the first one of the sub-blocks with respect to the other one of the sub-blocks.

Combining hash-based duplication with sub-block differencing to deduplicate data

Example apparatus, methods, and computers are configured to consider multiple lookups when making decisions concerning whether a probabilistic data structure indicates that an item is or is not present. One example method includes receiving a first response from a probabilistic data structure, where the first response indicates whether a first element is a member of a set of stored elements. The example method also includes receiving a set of second responses from the probabilistic data structure, where the set of second responses indicate whether members of a corresponding set of second elements are members of the set of stored elements. The method then provides a present/absent signal concerning whether the first element is a member of the set of stored elements. The signal is computed as a function of the first response and the set of second responses rather than merely as a function of the first response.

Considering multiple lookups in bloom filter decision making

A computer-enabled method of storing an input dataset in a storage medium includes storing a copy for each of a plurality of repeatable blocks of data in an input dataset in a storage medium. The process further includes finding a location in the storage medium of the copy of a block of data in the input dataset. Finding the location includes determining a most likely location in the storage medium of the copy of the block of data from one or more blocks of data preceding the block of data based on statistics of past stored data. Finding the location further includes if the determined most likely location contains a block of data that matches with the actual block of data, retrieving the location in the storage medium of the copy of the block of data. The process also includes storing the location of the copy of the block of data.

Data de-duplication by predicting the locations of sub-blocks within the repository

In a method for repairing a corrupt data segment used by a de-duplication engine, a data segment indexed by a de-duplication index is identified as being as a corrupt data segment. A search is made to find a duplicate data segment with which to replace the corrupt data segment, where the duplicate data segment comprises an uncorrupted version of the data segment. The corrupt data segment is replaced with the duplicate data segment.

Growing community of inventors

Unley, Australia

Andrew Leppard