International Business Machines Corporation

William W Owen

Dean Hildebrand

Sandeep R Patil

Sasikanth Eda

Smita J Raut

Simon Lorenz

Brian Nelson

Rainer Wolafka

Varun Mittal

Rui Zhang

Nagapramod S Mandagere

Aayush Gupta

Shripad Jayant Nadgowda

Trishali Satish Nayar

Sanjay Vipin Gandhi

Ashutosh V Mate

Heiko Hary Ludwig

Timothy A Johnson

Mathias Dietz

Erik Rueger

Craig Anthony Klein

Louie Arthur Dickens

Michael Edward Factor

Robert Haas

Robert Beverley Basham

Abhishek Jain

Nathalie Baracaldo Angel

Christof Schmitt

Nilesh Prabhakar Bhosale

Joshua M Rhoades

Thomas D Weigold

Slavisa Sarafijanovic

Harald Seipp

Ashish Pandey

Nathalie Baracaldo-Angel

Sandeep Naik

Robert L Valdez

Marina M Ruíz

Kumaran Rajaram

Lukas L Rupprecht

Winnie Hiu-Tung Tsang

Corinna J Sheret

A method for more efficiently storing genomic includes designating multiple different data storage techniques for storing genomic data generated by a genomic pipeline. The method further identifies a file, made up of multiple blocks, generated by the genomic pipeline. The method determines which data storage technique is most optimal to store each block of the file. In doing so, the method may consider the type of the file, the stage of the genomic pipeline that generated the file, the access frequency for blocks of the file, the most accessed blocks of the file, and the like. The method stores each block using the data storage technique determined to be most optimal after completion of a designated stage of the genomic pipeline, such that blocks of the file are stored using several different data storage techniques. A corresponding system and computer program product are also disclosed.

Hybridized storage optimization for genomic workloads

A computer-implemented method according to one embodiment includes in response to identifying a match between a file and at least one policy, updating details of a container in an object namespace of a file system, where the object namespace of the file system stores object-based representations of files within the file system; creating a symbolic link to the file in the object namespace of the file system; and updating a database to identify the symbolic link and the details of the container.

System and methods for dynamic generation of object storage datasets from existing file datasets

A processor may identify a first directory in the storage system. The first directory may include one or more subdirectories in one or more levels under the first directory. The one or more subdirectories may include a second directory that has includes one or more objects. The first directory may be associated with a first inode, and the second directory may be associated with a second inode. The processor may perform a stat call on the second directory to determine metadata attributes for the one or more objects that are stored in the second directory. The metadata attributes for the one or more objects may be stored in the second inode. The processor may add the metadata attributes for the one or more objects to the first inode.

Unified file and object storage architecture for clustered file systems

A computer-implemented method according to one embodiment includes identifying a plurality of segment files within an object storage system, determining all data blocks associated with the plurality of segment files within the object storage system, and mapping all the data blocks associated with the plurality of segment files to a single new file within the object storage system.

Performing object consolidation within a storage system

A computed-implemented method is provided for generating object storage datasets from file datasets. The computed-implemented method includes analyzing entries of a file system to identify a file and checking the file against at least one policy. In response to identifying a match between the file and the at least one policy, details of a container are updated. A symbolic link to the file is created to allow access of the file by an object-based client. Updating the details of the container includes updating a database such that the database identifies at least one of the created symbolic link and the details of the container.

In one general embodiment, a computer-implemented method includes analyzing a plurality of data management factors within an object-based storage system, including tracking which of a plurality of proxy servers within the object-based storage system currently holds a lock; determining a data management policy for predetermined data within the object-based storage system, based on the analyzing, including identifying a current location of the lock within one of the plurality of proxy servers within the object-based storage system; and sharing the data management policy for the predetermined data, including the current location of the lock within the object-based storage system, with a load balancer associated with the object-based storage system.

Sharing a data management policy with a load balancer

A computer-implemented method, according to one embodiment, includes: receiving a file operation, and determining a transaction identification which corresponds to the file operation. The transaction identification is used to create a unique queue which corresponds to the file operation. A number of fields included in the unique queue is equal to a number of sub-operations included in a predefined template associated with the file operation. Moreover, the number of fields included in the unique queue are populated as each of the respective number of sub-operations are performed. A determination is made as to whether each of the number of fields in the unique queue have been populated, and the unique queue is sent to a global queue in response to determining that each of the number of fields in the unique queue have been populated.

Enabling consistency in push order for network caching

A processor may identify a first directory in the UFO storage system. The first directory may include one or more subdirectories in one or more levels under the first directory. The one or more subdirectories may include a second directory that has includes one or more objects. The first directory may be associated with a first inode, and the second directory may be associated with a second inode. The processor may perform a stat call on the second directory to determine metadata attributes for the one or more objects that are stored in the second directory. The metadata attributes for the one or more objects may be stored in the second inode. The processor may add the metadata attributes for the one or more objects to the first inode.

A computer-implemented method, according to one embodiment, includes: sending management data to a designated common cell in a namespace of a central storage location from one or more remote storage locations. The management data corresponds to object data stored at the one or more remote storage locations. the object data stored at each of the one or more remote storage locations is further transferred to a respective cell in the namespace of the central storage location. Moreover, a modified consistency hashing algorithm configuration is implemented. The modified consistency hashing algorithm configuration routes updates of existing object data in the one or more remote storage locations to the respective cells in the central storage location. Other systems, methods, and computer program products are described in additional embodiments.

Consistent hashing configurations supporting multi-site replication

A computer-implemented method, according to one embodiment, includes: dividing a namespace in a central storage location into more than one cell. A first of the more than one cell is a common cell, which is configured to store management data received from remote storage locations coupled to the central storage location. Each of the remaining cells are configured to store object data received from a respective one of the remote storage locations. Management data received from the remote storage locations is stored in the common cell of the central storage location. Object data is also received, where the object data corresponds to the management data received from the remote storage locations. Furthermore, the received object data is stored in the respective cells of the central storage location. Other systems, methods, and computer program products are described in additional embodiments.

In one general embodiment, a computer program product for sharing a data management policy with a load balancer comprises a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se. Additionally, the program instructions are executable by a processor to cause the processor to perform a method comprising analyzing, by the processor, a plurality of data management factors within an object-based storage system, determining, by the processor, a data management policy for predetermined data within the object-based storage system, based on the analyzing, and sharing, by the processor, the data management policy for the predetermined data with a load balancer associated with the object-based storage system.

An object-based data storage system includes a memory and a processor for executing machine executable instructions configured for implementing logical containers for data objects each having a global identifier. The containers are configured for storing metadata including a first parameterization value descriptive of a number of storage nodes and a second parameterization value descriptive of a classification of the data objects. The machine executable instructions are further configured for implementing a first object storage ring for addressing storage locations across the multiple storage nodes using a surjective function. Execution of the instructions causes the processor to: instantiate the first object storage ring, receive the global identifier and the metadata by the first object storage ring, and generate a storage address by the first object storage ring for the data object using the global identifier, the first parameterization value and the second parameterization value as input to the surjective function.

Data object storage across multiple storage nodes

A computer-implemented method, according to one embodiment, includes: creating a namespace in a central storage location, and dividing the namespace into more than one cell. A first cell is a common cell configured to store management data received from remote storage locations coupled to the central storage location. Each of the remaining cells are configured to store object data received from a respective one of the remote storage locations. The computer-implemented method further includes: receiving management data from the remote storage locations, storing the received management data in the common cell of the central storage location, receiving object data corresponding to the management data received from the remote storage locations, and storing the received object data in the respective cells of the central storage location. The management data corresponds to object data stored in the remote storage locations. Other systems, methods, and computer program products are described in additional embodiments.

Techniques are disclosed herein for controlling object placement in object storage. A placement component of a storage application receives a request to store a first object in an object store having multiple nodes. The object store determines a placement of the first object to one of the nodes based on an object namespace including a numerical namespace and a lexicographical namespace. Each node is assigned a corresponding subspace of the object namespace for storing objects. The first object includes a numerical namespace value and a lexicographical namespace value. A second object (a replica of the first object) is generated. The first object is stored to a first node based on the lexicographical namespace value. The second object is stored to a second node based on the numerical namespace value.

Growing community of inventors

Tucson, AZ, United States of America

William W Owen