Huawei Technologies Co., Limited

Zongquan Zhang

Chengwei Zhang

Jun You

Chuanshuai Yu

Yanhui Zhong

Yuguo Li

A data deduplication method includes receiving an overwrite request sent by an external device, where the overwrite request carries a data block and a first address into which the data block is to be stored, determining whether an overwrite quantity of the first address exceeds a first threshold within a time period [t, t], where both tand tare time points, and tis later than t, and when the overwrite quantity of the first address exceeds the first threshold within the time period [t, t], skipping performing a deduplication operation on the data block or when the overwrite quantity of the first address does not exceed the first threshold within the time period [t, t], performing a deduplication operation on the data block.

Data deduplication method and apparatus

Embodiments of the present invention provide a data processing method and apparatus. According to the embodiments of the present invention, when it is found that a data hash value in a currently received data stream exceeds a preset first threshold, a part or all of data in the data stream is not deduplicated, and is directly stored, so as to prevent the data in the data stream from being dispersedly stored into a plurality of storage areas; instead, the part or all of the data is stored into a storage area in a centralized manner, so that a deduplication rate is effectively improved on the whole, particularly in a scenario of large data storage amount.

Data processing method and apparatus

The present disclosure directs to solutions for performing deduplication by a storage device. In the solutions, according to a duplicate data locality principle, non-duplicate data blocks whose logical addresses are contiguous are stored in contiguous physical addresses in a sequence of the logical addresses, and fingerprints of the non-duplicate data blocks whose logical addresses are contiguous are also stored in contiguous physical addresses in the sequence of the logical addresses, and in addition, a mapping from a logical address, which is of one data block in the non-duplicate data blocks whose logical addresses are contiguous, to an aggregation address is established.

Method and storage device for reducing data duplication

A data deduplication method and apparatus are provided. The method includes receiving an overwrite request sent by an external device, where the overwrite request carries a data block and a first address into which the data block is to be stored; determining whether an overwrite quantity of the first address exceeds a first threshold within a time period [t, t], where both tand tare time points, and tis later than t; and when the overwrite quantity of the first address exceeds the first threshold within the time period [t, t], skipping performing a deduplication operation on the data block; or when the overwrite quantity of the first address does not exceed the first threshold within the time period [t, t], performing a deduplication operation on the data block. Therefore, a large quantity of computing resources of a storage server are saved, and impact brought by a deduplication operation on storage server performance is also decreased.

Embodiments of the present disclosure disclose a solution for data backup and recovery in a storage system. When a source device in the storage system backs up, to a backup-end device, a data block that is written after a snapshot S, the source device performs a logical operation such as an exclusive-NOR or exclusive-OR operation on the written data block and an original data block, which is recorded in the snapshot S, of the written data block, and then compresses a data block obtained after the logical operation, which improves a compression ratio of a data block, thereby reducing an amount of data that is sent to the backup-end device, and saving transmission bandwidth. The solution may be further applied to a scenario of data recovery in a storage system.

Growing community of inventors

Chengdu, China

Zongquan Zhang