International Business Machines Corporation

Wayne A Sawdon

Frank Bernhard Schmuck

Deepavali M Bhagwat

Dean Hildebrand

Marc Adam Kaplan

Marc M Eshel

Joseph W Dain

James Christopher Wyllie

Roger Lee Haskin

M Corneliu Constantinescu

Renu Tewari

Manoj Premanand Naik

James Patrick Smith

Dominic Mueller-Wicke

Nilesh Prabhakar Bhosale

James Hewitt

Nils Haustein

Vasily Tarasov

Stefan Bender

Umesh Deshpande

Leo Shyh-Wei Luan

Sven Oehme

Steven M Berman

Constantine Arnold

Robert J Curran

Lukas Rupprecht

Subashini Balachandran

Leo S Luan

Gero Friedrich Wolf Schmidt

Robert Beverley Basham

Rui Zhang

Alessandro Sorniotti

Elli Androulaki

Christian Mueller

Neeta Garimella

Wayne Curtis Hineman

Owen Theodore Anderson

James C Davis

Frank N Lee

Douglas Stephen Noddings

Harold Joseph Roberson, Ii

Martin Petermann

Dimitrios Skourtis

Daniel Lloyd McNabb

Dietmar Fischer

Carsten Brixius

Dean S Daniels

Nikola Knezevic

Frank H B Schmuck

Demetrios K Michalaros

Provided are a computer program product, integrated cache manager, and method for maintaining data in a first level memory and buckets representing regions of memory devices to extend data cache. A plurality of buckets represent distinct regions of memory devices. The buckets are associated with different threshold access count ranges. Data having an access count is stored in one of the buckets associated with a threshold access count range including the access count of the data to store. Data evicted from a first level memory is copied to an initial bucket comprising one of the buckets. Data is moved from a source bucket comprising one of the buckets, including the initial bucket, to a target bucket of the buckets having a target threshold access count range including an access count of the data to move.

Maintaining data in a first level memory and buckets representing regions of memory devices to extend data cache

A computer-implemented method according to one embodiment includes monitoring for occurrence of trigger events corresponding to creation of metadata in a computing environment, and detecting occurrence of a trigger event. The method further includes loading a rule that is associated with the detected trigger event in response to detecting occurrence of the trigger event. The rule includes a condition and an action, and the action defines a subset of metadata associated with processing a job in the computing environment. The method further includes collecting the defined subset of metadata in accordance with the action in response to the condition of the rule being met, and storing the collected subset of metadata. A system according to another embodiment includes a processor, and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor. The logic is configured to perform the foregoing method.

Rule-based collections of subset(s) of metadata in response to a trigger event occurring

Embodiments for storage resource utilization analytics using metadata tags by a processor. Storage capacity utilization in the plurality of heterogeneous storage systems may be determined using one or more events indexed into a centralized search index.

Storage resource utilization analytics in a heterogeneous storage system environment using metadata tags

One embodiment provides a method for storage management in a hierarchical file system that includes creating a structure for the hierarchical file system including a top-file system and sub-file systems that each include files and directories. Directories for the sub-file systems of the hierarchical file system are tracked. A request for creation of a unique directory in the hierarchical file system is received. Upon a request by a particular sub-file system for more inodes without any inodes available, inode numbers are revoked from at least one other sub-file system having unused inodes.

Global namespace for a hierarchical set of file systems

One embodiment provides a method for optimizing data read-ahead for machine learning applications including obtaining, by a processor, next file information from a workflow scheduler for next input files for a next processing stage that are to be accessed by a machine learning application. Data for the next processing stage for the machine learning application and at least one system job is prefetched. The next input files are prefetched as the prefetching data reaches an end of current inputs. A schedule is caused to chain the next input files for the next processing stage to current input files of a current processing stage.

Prefetch chaining for optimizing data read-ahead for workflow and analytics

In one embodiment, a computer-implemented method for namespace and content manipulation within a storage system includes: defining policy rule(s), each policy rule dictating: how namespaces within the storage system may be transformed; and/or how content stored within the storage system may be transformed; receiving a request for access to a particular data set stored within the storage system; parsing the request to determine one or more transformations requested therein and/or required thereby; determining whether the particular data set is compliant with the one or more transformations; in response to determining the particular data set is compliant with the one or more transformations, providing access to the particular data set; and in response to determining the particular data set is not compliant with the one or more transformations, performing the one or more transformations to generate a transformed data set, and providing access to the transformed data set.

Dynamic, transparent manipulation of content and/or namespaces within data storage systems

Embodiments relate to policy-based, multi-scheme data reduction for a computer memory. An aspect includes receiving a plurality of policy rules by a policy engine of the computer memory, wherein a first policy rule specifies applying a first data reduction scheme to data in the computer memory based on the data matching first characteristics, wherein a second policy rule specifies applying a second data reduction scheme to data in the computer memory based on the data matching second characteristics, wherein the first data reduction scheme is different from the second data reduction scheme. Another aspect includes determining, by the policy engine, that first data in the computer memory matches the first characteristics, and that second data in the computer memory matches the second characteristics. Yet another aspect includes applying the first data reduction scheme to the first data, and applying the second data reduction scheme to the second data.

Policy-based, multi-scheme data reduction for computer memory

Embodiments for namespace management between storage systems by a processor. Events, containing metadata associated with an operation, from a plurality of heterogeneous storage systems may be received such that the events are indexed into a common repository according to the metadata to maintain a single namespace for data in the plurality of heterogeneous storage systems.

Global namespace in a heterogeneous storage system environment

A computer-implemented method according to one embodiment includes identifying an event associated with a high speed storage tier of a data storage system, determining a policy rule that is triggered in response to the event, and implementing one or more data management actions associated with the high speed storage tier according to the policy rule.

Dynamically managing a high speed storage tier of a data storage system

One embodiment provides a method for optimizing data read-ahead for workflow and analytics applications including obtaining, by a processor, next file information from a workflow scheduler for next files for a next processing stage that are to be accessed by a process. Data for the next processing stage for at least one application and at least one system job is prefetched. The next files are prefetched as the prefetching data reaches an end of current inputs.

In an approach to maintain data integrity in a computer data storage system, a clone management table is generated for a clone file tree that includes a relationship between at least one parent file and a clone file. The parent file is restored by: (i) determining that the parent-child relationship exists between the parent file and the clone file based on an entry in the clone management table, (ii) restoring data blocks in the parent file to the file system based on an analysis of an entry in the clone management table for the parent file and an identification of data blocks included in the clone file, and (iii) generating a new instance of the parent file based on the data blocks included in the file system and the entry in the clone management table for the clone file tree.

Clone file backup and restore

A computer-implemented method according to one embodiment includes identifying an event within a system; applying a model to the event to create an aggregated event; and storing the aggregated event.

Performing hierarchical provenance collection

Embodiments for triggering custom metadata extraction by a processor. Information may be extracted from an event so as to access data across a plurality of heterogeneous storage systems. Metadata may be extracted from the data that is accessed such that the metadata is assigned as custom metadata and indexed into a centralized search index, wherein the custom metadata is correlated to existing metadata associated with the data in the centralized search index.

Custom metadata extraction across a heterogeneous storage system environment

A computer-implemented method according to one embodiment includes identifying an accessing of a file within an operating system, checking an in-memory cache for path information associated with the file, checking an external cache for the path information associated with the file, conditionally retrieving the path information associated with the file by performing a file system lookup and adding the path information associated with the file to the in-memory cache and the external cache, and returning the path information.

Implementing a secondary storage dentry cache

The embodiments described herein relate to managing compressed data to optimize file compression. A compression is performed on a first set of data to create a first set of compressed data partitions in a compression group. A partition table is constructed, and partition entries for the first data block are added to the table in conjunction with the first set. A current size of the compression group is assessed. In response to a compression ratio being greater than a target compression ratio and internal fragmentation of the compression group being smaller than a threshold, the compression group is dynamically completed. The dynamic completion decides a size for the compression group. The partition table is added to the compression group by assigning space within the first compression group for the table. The compression group is written to persistent storage.

Compressed data layout with variable group size

Growing community of inventors

San Jose, CA, United States of America

Wayne A Sawdon