International Business Machines Corporation

Globalfoundries Inc.

Lawrence Yiumchee Chiu

Paul Henri Muench

Sangeetha Seshadri

Yu-Cheng Hsu

Yang Liu

Hyojun Kim

David Darden Chambliss

Chiahong Chen

Mei Mei

Chao G Li

Matthew J Kalos

Harry M Yudenfriend

Gail Andrea Spear

Prasenjit Sarkar

Joseph S Hyde, Ii

Cheng-Chung Song

James Alan Ruddy

Aayush Gupta

Mohit Saxena

Yong Guo

Clement Lambert Dickey

Shan Fan

Timothy Lee Toohey

Erci Xu

Lokesh M Gupta

Michael T Benhase

Evangelos Stavros Eleftheriou

Roman Alexander Pletka

Kaladhar Voruganti

Ioannis Koltsidas

Robert Haas

Nicolas Marc Clayton

Stephen LaRoux Blinick

Maohua Lu

Krishnakumar Rao Surugucchi

Xiao-yu Hu

Patrick Cozzi

Yang Zhou

Nicholas Lloyd Armstrong-Crews

Patrick Randolph Eaton

Surugucchi Krishnakumar Rao

Kaladhar Vorunganti

Provided are a computer program product, system, and method for managing data in a storage. Each index entry in and index identifies a log address in the log of one of the records. Each record in the log includes a key and value, and the log addresses map to physical addresses in the storage. A plurality of processed log addresses in the log are processed by determining whether the index indicates that the processed log address includes a current version of the record in the log; adding the record at the processed log address to a selected log address in the log in response to determining that the index indicates that the processed log address comprises the current version of the record; and deleting a record at the physical address in the storage to which the processed log address maps.

Managing data in storage according to a log structure

In accordance with one aspect of the present description, in response to a detection by a storage controller, of an operation by a host relating to migration of input/output operations from one host to another, a cache server of a storage controller, transmits to a target cache client of the target host, a cache map of the source cache of the source host wherein the cache map identifies locations of a portion of the storage cached in the source cache. In response, the cache client of the target host, may populate the target cache of the target host with data from the locations of the portion of the storage, as identified by the cache map transmitted by the cache server, which may reduce cache warming time. Other features or advantages may be realized in addition to or instead of those described herein, depending upon the particular application.

Cache migration management in a virtualized distributed computing system

In various embodiments, computer-implemented techniques for improving function of in-memory processing systems via adaptively caching datasets include: identifying data stored in a distributed filesystem, the data including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Corresponding systems and computer-program products are also disclosed.

Adaptive caching and dynamic delay scheduling for in-memory data analytics

One embodiment provides a method for retrospective snapshot creation including creating, by a processor, a first snapshot that captures logical state of a data store at a first key. Creation of the first snapshot is based on determining a log offset corresponding to the first key, determining existence of a second snapshot that captures logical state of the data store and recording a retrospective snapshot at a last valid log address offset prior to the first key upon a determination that the second snapshot exists based on determining at least one of: whether log address offsets from a first log entry of a log to a log entry of the log at the first key are contiguous and whether log address offsets from the second snapshot to the first key are contiguous.

Retrospective snapshots in log-structured storage systems

Provided are a computer program product, system, and method for handling failure of a command to add a record to a log. The log has records at log addresses. A command is received to add a record to the log indicating a current log address at which the record is stored in the log. A determination is made as to whether the command has a resend parameter set, wherein the resend parameter indicates whether the command is being resent to add the record. A determination is made as to whether there is a log address in the log having the record following the current log address in response to determining that the resend parameter indicates that the command is being resent. The determined log address in the log is returned.

Handling failure of a command to add a record to a log

Provided are a computer program product, system, and method for providing access information to a storage controller to determine a storage tier for storing data. Access information is maintained for each data record in a data store, wherein the access information indicates a level of access to each data record in the data store. A write request directed to a data record in the data store is received. A command is generated identifying the data record and including the access information for the data record. The command is transmitted to the storage controller, wherein the storage controller uses the access information to determine one of the plurality of storage tiers on which to store the data record.

Providing access information to a storage controller to determine a storage tier for storing data

One embodiment provides a method for retrospective snapshot creation including creating, by a processor, a first snapshot that captures logical state of a data store at a first time in a time range. Creation of the first snapshot is based on determining existence of a second snapshot that captures logical state of the data store and recording a retrospective snapshot at a last valid log address offset prior to the first time upon a determination that the second snapshot exists based on determining at least one of: whether log address offsets from a first log entry of a log to a log entry of the log at the first time are contiguous and whether log address offsets from the second snapshot to the first time are contiguous.

Retrospective snapshots in log structured storage systems

A method to efficiently checkpoint and reconstruct an in-memory index associated with a log-structured object store includes enabling asynchronous write operations to occur to a log-structured object store. The log-structured object store utilizes an in-memory index to access objects therein. The method further enables checkpoint operations to occur to the log-structured object store without pausing the asynchronous write operations. When initiating checkpoint operations, the method establishes a 'begin checkpoint' marker on the log-structured object store. This 'begin checkpoint' marker is configured to point to an oldest known log location recorded in the in-memory index. In the event the in-memory index is lost, the method reconstructs the in-memory index by analyzing the log-structured object store starting from the oldest known log location. A corresponding system and computer program product are also disclosed and claimed herein.

Efficient index checkpointing in log-structured object stores

A method to efficiently checkpoint and reconstruct an in-memory index associated with a log-structured object store includes enabling asynchronous write operations to occur to a log-structured object store. The log-structured object store utilizes an in-memory index to access objects therein. The method further enables checkpoint operations to occur to the log-structured object store without pausing the asynchronous write operations. When initiating checkpoint operations, the method establishes a 'begin checkpoint' marker on the log-structured object store. This 'begin checkpoint' marker is configured to point to an earliest address in the log-structured object store that is uncommitted to the in-memory index. In the event the in-memory index is lost, the method reconstructs the in-memory index by analyzing the log-structured object store starting from the earliest address uncommitted to the in-memory index. A corresponding system and computer program product are also disclosed.

Efficient index recovery in log-structured object stores

In various embodiments, techniques for improving function of in-memory processing systems include identifying data stored in a distributed filesystem, including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Techniques may also include determining an average remote execution time of the in-memory processing system; tracking a waiting time of a job pending processing on a local node of the in-memory processing system; comparing the tracked waiting time of the job pending processing; and demoting a locality of the job if waiting time of the job exceeds the average remote execution time of the in-memory processing system. Corresponding systems and computer-program products are also disclosed.

Provided are a method, system, and computer program product for processing read and write requests in a storage controller. A host adaptor in the storage controller receives a write request from a host system for a storage address in a storage device. The host adaptor sends write information indicating the storage address updated by the write request to a device adaptor in the storage controller. The host adaptor writes the write data to a cache in the storage controller. The device adaptor indicates the storage address indicated in the write information to a modified storage address list stored in the device adaptor, wherein the modified storage address list indicates modified data in the cache for storage addresses in the storage device.

Processing read and write requests

For a high availability cache, a cache module obtains permission to manage the cache in response to a failover event in a server cluster by communicating a cache coherency token. An update module rebuilds a cache directory from data stored in the cache and accesses the cache without reloading the data stored in the cache.

High availability cache in server cluster

Growing community of inventors

Saratoga, CA, United States of America

Lawrence Yiumchee Chiu