Symantec Corporation

Western Digital Technologies, Inc.

Mark Christiaens

Koen De Keyser

Frederik Jacqueline Luc De Schrijver

Stijn Blyweert

Weibao Wu

Fanglu Guo

Bastiaan Stougie

Mark Vertongen

Cyril Guyot

Robert Eugeniu Mateescu

Adam Manzanares

Lluis Pamies-Juarez

Nick Cremelie

A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.

Reduced storage of metadata in a distributed encoded storage system

A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.

Fast read operation utilizing reduced storage of metadata in a distributed encoded storage system

A set of encoded data fragments is grouped into a container object in sequential order. Each encoded data fragment is a specific fragment size, and the container object is a specific container object size. The sequential order of the set of encoded data fragments can be tracked in a log in memory, such that the location of any one of the data fragments in the container object can be determined. The container object can be stored directly on a specific backend storage element, without using a file system. A corresponding container object identifier identifies the physical storage location of the container object on the backend storage element. The container object identifier is tracked in the log in memory, such that the physical location on the backend storage element of any specific one of the set of encoded data fragments in the container object can be determined.

Staging of write operations for container-based storage for sequential media

Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

Storage cluster and method that efficiently store small objects with erasure codes

Various methods and systems for using adaptive data compression in a backup system are disclosed. One method involves detecting whether to compress a unit of storage that is to be backed up. The detecting involves attempting to compress a portion of the unit of storage. If the attempt to compress the portion of the unit of storage meets a specified compression performance threshold, i.e., if the unit of storage is compressible, the unit of storage is compressed. Otherwise the unit of storage is not compressed.

Adaptive compression using a sampling based heuristic

Analyzing backup objects maintained by a de-duplication server. A plurality of first objects may be maintained. Each first object may refer to second object(s) and each second object may refer back to at least one first object. For each respective first object, the respective first object may be analyzed to determine the one or more second objects referred to by the respective first object. Correspondingly, a command may be generated for each respective second object of the determined second object(s), thereby generating a plurality of commands. Each command may be used to verify that the respective second object refers back to the respective first object. The plurality of commands may be sorted into a disk access order. The commands may be used to verify that each second object refers back to first objects that refer to the second object.

Analyzing backup objects maintained by a de-duplication storage system

A computer-implemented method for removing unreferenced data segments from deduplicated data systems may include: 1) identifying a deduplicated data system that contains a plurality of data objects, 2) dividing the data objects within the deduplicated data system into a plurality of data object groups, 3) identifying, within the data object groups, at least one data object group that has changed subsequent to a prior garbage-collection operation that removed data segments that were not referenced by data objects within the deduplicated data system, 4) identifying at least one container within the deduplicated data system that contains data segments referenced by data objects within the changed data object group, and then, for each identified container, 5) removing data segments from the identified container that are not referenced by data objects within the deduplicated data system. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for removing unreferenced data segments from deduplicated data systems

A computer-implemented method for removing unreferenced data segments from deduplicated data systems may include: 1) identifying a deduplicated data system that contains a plurality of data segments, 2) identifying a plurality of containers within the deduplicated data system, with each container containing a subset of the data segments within the deduplicated data system, 3) identifying at least one container within the plurality of containers that is likely to include a large proportion of data segments that are not referenced by data objects within the deduplicated data system, and then, for each identified container, 4) searching for unreferenced data segments within the identified container and 5) removing the unreferenced data segments from the identified container. Various other methods, systems, and computer-readable media are also disclosed.

A backup cache on a client is prepopulated with fingerprints corresponding to data stored on a backup server. A plurality of fingerprints of data are received from the client, each received fingerprint corresponding to data present on the client which is subject to backup to the server. For each received fingerprint, an attempt is made to locate data corresponding to the received fingerprint stored on the backup server. Responsive to locating data corresponding to a received fingerprint stored on the backup server, data stored on the backup server considered to be 'in the neighborhood' of the located data is identified. Fingerprints (e.g., hashes) identifying data stored on the backup server in the neighborhood of the located data are created and transmitted to the client, for prepopulating the backup cache on the client.

Optimized prepopulation of a client side cache in a deduplication environment

A computer-implemented method for managing deduplicated data using unilateral referencing may comprise: 1) identifying each file in the deduplicated data system, 2) identifying each data segment in the deduplicated data system that is referenced by at least one file in the deduplicated data system, and then 3) creating an active-data-segments set that identifies or references each data segment that is referenced by at least one file in the system. Data segments in the system that are not identified or referenced in the active-data-segments set may be removed. Corresponding systems and methods are also disclosed.

Growing community of inventors

Aalter, Belgium

Mark Christiaens