International Business Machines Corporation

Akamai Technologies, Inc.

Globalfoundries Inc.

Dmitry Sotnikov

Danny Harnik

Sergey Marenkov

Ety Khaitzin

Jonathan Amit

Chaim Koifman

Elliot Karl Kolodner

Nir Halowani

Paula Kim Ta-Shma

George Goldberg

Lilia Demidov

Ronen Itshak Kat

Oded Margalit

Michael Edward Factor

Michael Hakimi

Yosef Shatsky

Ben Sasson

David Hadas

Amir Epstein

Shai Ishaya Tahar

Gil Vernik

Brian J Mancuso

Shai Kedem

Omri Guttman

Kfir Zigdon

Kat I Ronen

David Darden Chambliss

Andrzej P Kochut

Alexei A Karve

Ron Asher

Ilias Iliadis

Aviv Caro

Vinodh Venkatesan

Dalit Naor

Yosef Moatti

Jonathan Fischer-Toubol

Yaron Weinsberg

Mangesh Kasbekar

Eran Rom

Avishay Traeger

Lior Shlomov

Gil Shapira

Nir Friedman

Shai Taharlev

Denis Emelyanov

Arnon Shoshany

Dvir Tuberg

Improved technology for managing the caching of objects that are rarely requested by clients. A cache system can be configured to assess a class of objects (such as objects associated with a particular domain) for cacheability, based on traffic observations. If the maximum possible cache offloading for the class of objects falls below a threshold level, which indicates a high proportion of non-cacheable or 'single-hitter' content, then cache admission logic is configured to admit objects only after multiple clients requests during a time period (usually the object's time in cache, or eviction age). Otherwise, the cache admission logic may operate to admit objects to the cache after the first client request, assuming the object meets cacheability criteria. The technological improvements disclosed herein can be used to improve cache utilization, for example by preventing single-hitter objects from pushing out multi-hit objects (the objects that get hits after being added to cache).

Systems and methods for preventing the caching of rarely requested objects

Improvements to internet cache protocols are disclosed. In certain embodiments, a client-facing proxy server can query peer servers to determine whether they have a copy of an object that the proxy server needs. The peer servers can respond based on object information that the peer servers stored about objects they have in cache, where the peers recorded such object information previously when ingesting the objects into their cache and stored it separately from the objects for fast access (e.g. in RAM vs. on disk). This information can be expressed in a compact way using just a few object flags, and enables the peer server to quickly respond and with detail about the status of objects they hold. The proxy server can make an intelligent decision about which peer to use, and indeed whether to use a peer at all.

Internet caches with object hints

Origin offload is a key performance indicator of a content delivery network (CDN). This patent document presents unique methods and systems for measuring origin offload and applying those measurements to improve the offload. The techniques presented herein enable resource-efficient measurement of origin offload by individual servers and aggregation and analysis of such measurements to produce significant insights. The teachings hereof can be used to better identify root causes of suboptimal offload performance, to tune CDN settings and configurations, and to modify network operations, deployment and/or capacity planning. In addition, discussed herein are improved metrics showing offload in relation to the maximum achievable offload for the particular traffic being served.

Measuring and improving origin offload and resource utilization in caching systems

A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Identification of high deduplication data

Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Low memory sampling-based estimation of distinct elements and deduplication

Methods, computing systems and computer program products implement embodiments of the present invention that include configuring a storage system to store multiple storage entities for access by one or more host computers in communication with the storage system, and specifying a compression condition including a minimum compression ratio. The storage system can then estimate an expected compression ratio for a given storage entity, compress the given storage entity upon the expected compression ratio meeting the compression condition, and provide, to a given host computer, access to the compressed given storage entity.

Storage data reduction analysis and forecast

Various embodiments for data management in a replicated storage environment, by a processor device, are provided. In one embodiment, a method comprises storing a plurality of data replicas under a plurality of heterogeneous compression algorithms, wherein one of the data replicas is optimized for a data operation.

Heterogeneous compression in replicated storage

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Real-time identification of data candidates for classification based compression

Methods and systems for data transfer include adding a data chunks to a priority queue in an order based on utilization priority. A reducibility score for the data chunks is determined. A data reduction operation is performed on a data chunk having a highest reducibility in the priority queue using a processor if sufficient resources are available. The data chunk having the lowest reducibility score is moved from the priority queue to a transfer queue for transmission if the transfer queue is not full.

Network utilization improvement by data reduction based migration prioritization

A globally distributed scan list is determined. A determination is made whether the first data replica in the first plurality of data stored on a first device is in sync with a second data replica in the second plurality of data on a second device. In response to determining that the first data replica is not in sync with the second data replica, the first data replica is added to an unsynced queue. The neighbor data of the first plurality of data is added to a suspect queue. The priority to check the neighbor data is increased if the neighbor data is already in the suspect queue. Unsynced neighbor data is added to the unsynced queue. The priority for recovery of the data in the unsynced queue is determined. The priority is based on the vulnerability of the data. A data replica in the unsynced queue is recovered.

Fully distributed intelligent rebuild

Machines, systems and methods for enhancing data recovery in a data storage system, the method comprising determining whether one or more data storage mediums in a data storage system are unavailable; determining data that are at a risk of loss, due to said one or more data storage mediums being unavailable; from among the data that is determined to be at the risk of loss, identifying data that is highly vulnerable to loss; and creating one or more temporary replicas of the data that is highly vulnerable to loss.

Reliability enhancement in a distributed storage system

Growing community of inventors

Givataim, Israel

Dmitry Sotnikov