International Business Machines Corporation

Mohit Saxena

James Lee Hafner

Aayush Gupta

David Allan Pease

Mario Blaum

Prasenjit Sarkar

Nagapramod S Mandagere

Mingyuan Xia

Lawrence Yiumchee Chiu

Erci Xu

Dean Hildebrand

Rui Zhang

Sangeetha Seshadri

Pin Zhou

Paul Henri Muench

Reshu Jain

Umesh Deshpande

Cognitive compression with varying structural granularities in a NoSQL database by establishing a data training set for compressing and decompressing data stored within the NoSQL database. The data training set includes received user policy goals, compression parameters, and metered feedback associated with data usage and workload characteristics. A compression parameter model is dynamically implemented in real-time for the data selected according to the established data training set to compress and decompress the data at a given structural granularity.

Cognitive compression with varying structural granularities in NoSQL databases

In one embodiment, a computer-implemented method for object-granular policy assignment in an object storage environment includes: automatically assigning a plurality of objects stored within the object storage environment to one or more policies; and locating one or more of the objects among a plurality of policy rings within the object storage environment based on semantics of the one or more policies. A virtual namespace maps each object to at least one of the policy rings based on the one or more policies assigned to the object; and the virtual namespace is scalable with a number of the plurality of objects stored within the object storage environment. Corresponding systems and computer program products are also disclosed.

Scalable virtual namespace and semantic policy management for object storage

A method for recovering data for a file system includes a journal-less recovery process that detects one or more inconsistencies for file system blocks upon a system failure based on one or more comparisons of information for the file system blocks.

Journal-less recovery for nested crash-consistent storage systems

The embodiments relate to performing a write operation in a filesystem. Replica files embedded with respective sets of data are stored are on persistent storage local to replica nodes of the filesystem. The storage includes atomically updating the replica files to the replica nodes. Each set of data has information for self-characterizing each replica file. A map stored at persistent storage local to a remote node is updated. The map maintains information for identifying each replica file stored at each replica node.

Fast recovery using self-describing replica files in a distributed storage system

In various embodiments, computer-implemented techniques for improving function of in-memory processing systems via adaptively caching datasets include: identifying data stored in a distributed filesystem, the data including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Corresponding systems and computer-program products are also disclosed.

Adaptive caching and dynamic delay scheduling for in-memory data analytics

In one embodiment, a computer-implemented method includes selecting a cache block descriptor (CBD) from amongst a plurality of CBDs stored to a cache storage device to defragment based on a determination of utilization of a particular fine block descriptor (FBD) having a first size that is allocated to the selected CBD. The cache storage device includes a free pool of FBDs having various sizes that is available for use in the plurality of CBDs. Also, the particular FBD having the first size has a lowest availability in the free pool of FBDs. Other methods, systems, and computer program products are described in accordance with additional embodiments.

Reducing defragmentation in a multi-grained writeback cache

Embodiments relate to a system with multiple erasure codes, and selecting and encoding for a write file with one of the codes to mitigate costs associated with storage recovery. The codes include a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data. State data is tracked to ascertain frequency of access to the file. One of the erasure codes is dynamically selected based on the tracked data, with the focus of the code select to lower recovery costs, and the data is encoded with the selected erasure code. Accordingly, the original coding of the write file is subject to change based on the tracked state data.

Multiple erasure codes for distributed storage

In one embodiment, a method includes determining a data record partition size based on resources used for transferring data from a higher storage tier to one or more lower storage tiers. The method also includes determining which data records stored to the higher storage tier are suitable for export to the one or more lower storage tiers, determining a distribution mapping of the first memory, the distribution mapping indicating a relative distribution of storage locations for all of the data records that are stored to the higher storage tier, identifying all sets of contiguously stored data records on the higher storage tier that are suitable for export and greater in size than the data record partition size, logically sorting, in a descending order of size, the sets of contiguously stored data records, and sending a list of logically sorted sets of contiguously stored data records to an exporter.

Efficient export scheduling using index and schema to determine data distribution

A filesystem includes a pool of data blocks and a pool of index blocks. Non-overlapping subpools of data blocks, including a first subpool of data blocks and a second subpool of data blocks, are identified within the pool of data blocks. Likewise, non-overlapping subpools of index blocks, including a first subpool of index blocks and a second subpool of index blocks, are identified within the pool of index blocks. A first file group is mounted within a global namespace on a first mount point on the filesystem. A second file group is also mounted within the global namespace on a second mount point on the filesystem. The first subpool of data blocks and the first subpool of index blocks are allocated to the first mounted file group, and the second subpool of data blocks and the second subpool of index blocks are allocated to the second mounted file group.

Filesystem with isolated independent filesets

A method for storing and recovering data for a file system includes a write process that performs a write instruction for data that includes atomically updating corresponding data blocks and metadata-links on a storage medium. The metadata-links contain links from data blocks to parent metadata blocks and links from metadata blocks to the parent metadata blocks. A journal-less recovery process that detects one or more inconsistencies for file system blocks comprising data blocks or metadata blocks upon a system failure based on one or more comparisons of information for the file system blocks.

In one general embodiment, a computer-implemented method includes creating multiple pools of micro services. Each of the pools includes a specific configuration set and resource properties. Also, the computer-implemented method includes receiving incoming workloads. Moreover, the computer-implemented method includes, for each of the incoming workloads, dynamically mapping the incoming workload, based on characteristics of the incoming workload, to an access path traversing a combination of a subset of the pools of micro services.

Smart partitioning of storage access paths in shared storage services

In one embodiment, a system includes a cache storage device, a back-end storage device, and a processor and logic integrated with and/or executable by the processor. The logic is configured to receive indication of failure of a primary cache server at a secondary cache server, the primary and secondary cache servers being configured to manage read requests and write requests for the back-end storage device. The logic is also configured to set the secondary cache server to a by-pass mode for read requests directed to any portions of the back-end storage device managed by the primary cache server prior to the failure. Moreover, the logic is configured to read an index of cache block descriptors (CBDs) managed by the primary cache server prior to the failure into a memory of the secondary cache server.

Instant recovery in a multi-grained caching framework

In various embodiments, techniques for improving function of in-memory processing systems include identifying data stored in a distributed filesystem, including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Techniques may also include determining an average remote execution time of the in-memory processing system; tracking a waiting time of a job pending processing on a local node of the in-memory processing system; comparing the tracked waiting time of the job pending processing; and demoting a locality of the job if waiting time of the job exceeds the average remote execution time of the in-memory processing system. Corresponding systems and computer-program products are also disclosed.

Growing community of inventors

San Jose, CA, United States of America

Mohit Saxena