International Business Machines Corporation

Ehud Meiri

Yariv Bachar

Lior Aronovich

Michael Hirsch

Ron Asher

Yair Toaff

Shmuel Tomi Klein

Oded Sonin

Ilai Harsgor-Hendin

Shay Haim Akirav

Asaf Levy

Ron Edelstein

Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each small data chunk, a signature is generated based on a combination of a representation of characters used in selecting data to be deduplicated. A c-spectrum of the small data chunk being a sequence of representations of different characters ordered by a frequency of occurrence in the small data chunk, and an f-spectrum of the small data chunk being a corresponding sequence of frequencies of the different characters in the small data chunk.

Scalable deduplication system with small blocks

A strictly increasing function is implemented to generate a plurality of unique creation stamps, each of the plurality of unique creation stamps increasing over time pursuant to the strictly increasing function. A new task to be placed with the plurality of tasks is labeled with a new unique creation stamp of the plurality of unique creation stamps. The one of the list of dismissal rules holds a minimal valid creation (MVC) stamp, which is updated when a dismissal action for the one of the list of dismissal rules is executed.

Selective constant complexity dismissal in task scheduling

Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each of the small data chunk, a signature is generated based on a combination of a representation of characters that appear in the small data chunk with a representation of frequencies of the small data chunk. A signature is generated based on a combination of a representation of characters that appear. The signature is used to help in selecting the data to be deduplicated. Additional system and computer program product embodiments are disclosed and provide related advantages.

A strictly increasing function is implemented to generate a plurality of unique creation stamps, each of the plurality of unique creation stamps increasing over time pursuant to the strictly increasing function. A new task to be placed with the plurality of tasks is labeled with a new unique creation stamp of the plurality of unique creation stamps. The one of the list of dismissal rules holds a minimal valid creation (MVC) stamp, which is updated when a dismissal action for the one of the list of dismissal rules is executed. The dismissal action acts to dismiss a selection of tasks over time due to continuous dispatch.

Various embodiments for deduplicated data processing congestion control in a computing environment are provided. In one such embodiment, a congestion target setpoint is calculated using one of a proportional constant, an integral constant, and a derivative constant, wherein the congestion target setpoint is a virtual dimension setpoint. A single congestion metric is determined from a sampling of a plurality of combined deduplicated data processing congestion statistics in a number of active deduplicated data processes. A congestion limit is calculated from a comparison of the single congestion metric to the congestion target setpoint, the congestion limit being a manipulated variable. The congestion limit is compared to the number of active deduplicated data processes. If the number of active deduplicated data processes are less than the congestion limit, a new deduplicated data process of the number of active deduplicated data processes is spawned.

Deduplicated data processing congestion control

A plurality of log processes are synchronized. Each is independently performed in parallel with one another, into a single set of log files. A line buffering mechanism of an operating system (OS) of the computing environment forecloses interleaving of the log processes. Log management operations are concurrently performed by a single process protected by a file-system lock of the OS. The log management operations include at least one of a log compression, log retention, and log rotation operation.

Coordination of event logging operations and log management

For scalable data deduplication working with small data chunks in a computing environment, for each of the small data chunks, a signature is generated based on a combination of a representation of characters that appear in the small data chunks with a representation of frequencies of the small data chunks. The signature is used to help in selecting the data to be deduplicated.

Various embodiments for deduplicated data processing congestion control in a computing environment are provided. In one such embodiment, a single congestion metric is determined from a sampling of a plurality of combined deduplicated data processing congestion statistics in a number of active deduplicated data processes. The congestion limit is calculated from a comparison of the single congestion metric to a congestion target setpoint, the congestion target setpoint being a virtual dimension setpoint and the congestion limit being a manipulated variable. The number of active deduplicated data processes is compared to the congestion limit. If the number of active deduplicated data processes is less than the congestion limit, a new deduplicated data process is spawned.

Growing community of inventors

Tel Aviv, Israel

Ehud Meiri