International Business Machines Corporation

Randal Chilton Burns

Robert Michael Rees

Darrell D E Long

Darrell D Long

Atul Goel

Benjamin Clay Reed

Jason Christopher Young

Wayne Curtis Hineman

Rajagopal Ananthanarayanan

Ralph Attila Becker-Szendy

Edward Gustav Chron

David Macnaughton Wolfe

Inderpal Singh Narang

Ronald Fagin

James J Seeger

Miklos Ajtai

Larry Joseph Stockmeyer

Jujjuri Venkateswararao

A system and method for managing logical versions of a filesystem made through a near-instantaneous copy process is provided. Each logical version of the filesystem, including the primary version, is assigned a unique epoch number or other identifying value, and each filesystem object is assigned two epoch numbers that define the space of logical versions for which the object exists. A list of all valid epoch numbers are maintained in a version table. The near-instantaneous creation of a logical version of the filesystem requires inserting a unique epoch number entry in the version table () and the next epoch number to the new logical version. An original state of the system is preserved through the use of a copy-on-write procedure for filesystem objects that are reference by that logical filesystem version (). Accordingly, filesystem versions are maintained and original states of previous states of the logical filesystem versions are preserved.

Managing filesystem versions

Various copy-on-write implementations for a computing environment are presented. One copy-on-write implementation includes employing a read mapping table to perform a first virtual block to physical block mapping for use in reading a block of data of a file to be copy-on-written from physical storage for modification; and employing a different, write mapping table to perform a second virtual block to physical block mapping for use in writing a modified block of the file data to physical storage, wherein copy-on-write of the block of data is achieved using a single write operation. In another implementation, a distributed copy-on-write of a file for a client server environment is presented. This distributed copy-on-write includes performing, by a first client, copy-on-write of at least one block of data of the file to be copy-on-written, and performing, by a second client, copy-on-write of at least one other block of data of the file, wherein multiple clients perform the copy-on-write of the file.

Method, system and computer program product for implementing copy-on-write of a file

'A system and associated method restore a file system from incremental backups in the presence of deletion, without restoring deleted files. This is accomplished by logging deletions in a deletion log. Using the deletion log, the system re-deletes the deleted files upon a restore operation to an incremental backup epoch. The system the logs deletions so that the files that were deleted and survived a backup epoch have their records logged. The activity log enables an incremental backup from a prior snapshot, and represents a point-in-time version of the file system. The system can either not restore the deleted files, or, alternatively, it can restore the deleted files and then re-deletes them.'

System and method for restoring a file system from backups in the presence of deletions

In a distributed file system the distributed storage management is made useful to a variety of applications. Multiple quality of service options are provided through locking. Three locking systems are provided. The system offers a locking system designed for sequential consistency with write-back caching, typical of distributed file systems. A second locking system is provided for sequential consistency with no caching for applications that manage their own caches. Finally, a locking system that implements a weaker consistency model with write-back caching, designed for efficient replication and distribution of data is included. Locks for replication are suitable for serving dynamic data on the Internet and other highly-concurrent applications. The selection of the appropriate lock protocol for each file is set using the file metadata. Further, a novel locking system is provided for the lock system implementing a weak consistency model with write back caching. This system is implemented utilizing two whole file locks: a producer lock P and a consumer lock C. Any client can hold a consumer lock and when holding a consumer lock can read data and cache data for read. The producer lock is only held by a single writer and a writer holding a producer lock can write data, allocate and cache data for writing. When a writer performs a write, the write is performed as an out-of-place write. An out-of-place write writes the data to a different physical storage location than from which it was read. By performing an out-of-place write the old data still exists and is available to clients. Once the writer completes the write and releases the producer lock the previous data is invalidated and the clients are informed of the new location of the data. Clients can then read the new data from storage when needed and the server reclaims the old data blocks.

Producer/consumer locking system for efficient replication of file data

A system and method for managing access to assets in a distributed data storage system includes requesting, from a client computer, a lock from a server computer. The lock is evaluated using a bitwise comparison of the protected access modes and restricted access modes defined by the lock with the protected and restricted modes defined by the currently outstanding locks using an algorithm, such that a potentially large compatibility table is not needed.

System for dynamically evaluating locks in a distributed data storage system

A data management system for storage area networks (SANs) includes an authentication and directory service that integrates the management of cache coherency and synchronization into an authentication mechanism, referred to herein as a ticket. The data in the system is notionally classified into colors. When a principal computer requests a read or a write access to data of a particular &ldquo;color&rdquo;, the service determines whether the principal is authorized for the requested access, and if so, the service issues a ticket to the principal that specifies a data color that the bearer of the ticket is authorized to access using the type of access mechanism specified on the ticket. A hidden &ldquo;hue&rdquo; can also be specified on the ticket. The service can variously inform principals to cancel tickets based on color or time of issue and can also variously inform storage devices not to honor tickets of a certain hue or time of issue as appropriate to thereby maintain cache coherency and data consistency.

System and method for managing authentication and coherency in a storage area network

A system, method, and computer program product for a lease-based timeout scheme that addresses fencing's shortcomings. Unlike fencing, this scheme (or protocol) enables an isolated computer to realize it is disconnected from the distributed system and write its dirty data out to storage before its locks are stolen. In accordance with the invention, data consistency during a partition in a distributed system is ensured by establishing a lease based protocol between in the distributed system wherein a client can hold a lease with a server. The lease represents a contract between a client and a server in which the server promises to respect the client for a period of time. The server respects the contract even when it detects a partition between the client and itself.

Lease based safety protocol for distributed system with multiple networks

A data storage system randomly determines a start offset at which to write objects to a storage medium. For updated blocks of the object, e.g., for blocks written during copy-on-write as part of a point-in-time snapshot, the updated block is written in the region of the original file or as close thereto as possible to achieve &ldquo;virtual contiguity&rdquo;. Subsequent reads of the object read entire region containing both the object and, potentially, &ldquo;chaff&rdquo; data other than the object. The &ldquo;chaff&rdquo; data is discarded by the I/O system or file system using, e.g., a bit mask, subsequent to the read.

Data placement and allocation using virtual contiguity

A system and method for managing access to assets in a distributed data storage system includes requesting, from a client computer, a semi-preemptible access lock from a server computer. The semi-preemptible lock, if granted, is held by the client computer as long as the server does not demand it back, with the client computer granting open instances under non-preemptible file locks for the asset to which the locks pertain as long as the client computer holds the semi-preemptible lock. When another client computer requests the semi-preemptible lock, the server can demand the lock from the holding client, which relinquishes the lock if no open instances are protected by the lock. Otherwise, the holding client computer first attempts to downgrade its lock to meet the request, and if compatibility is not achieved thereby, the holding client refuses to relinquish the lock.

System for managing asset access in a distributed storage system

A bit-parallel system and method for allocating storage space for data objects using a bitmap. It is determined whether a suffix of free space bits in a previous bitmap word can be used with a prefix of free space bits in a current word that is contiguous to the previous word. If so, this renders a string of free space bits spanning multiple words, and it is determined whether the string represents a sufficiently large number of contiguous blocks (&ldquo;the target&rdquo;) to store the data object. If not, it is determined whether sufficient contiguous free space bits in the current word exist to fulfill the target. If the target still can't be achieved, the longest suffix of free space bits in the current word is found for possible use with the prefix of the next contiguous word, and the next word is then retrieved and processed. The algorithm for finding the suffix preferably is undertaken by considering bits in parallel.

System and method for allocating storage space using bit-parallel search of bitmap

A decentralized file system based on a network of remotely encrypted storage devices is disclosed. The file system includes a network to which a network client, a secure remotely encrypted storage device, a key manager, and a lock manager are attached. The system organizes data as files and directories. Files or directories are composed of one or more streams, which logically partition the data associated with the files or directories. The device serves as a repository of the system's data. The key manager controls data access keys while the lock manager handles consistency of the files. A network user may have read or write access to a file. Access is controlled using keys and access lists maintained by the key manager.

Decentralized remotely encrypted file system

A method and a system are presented for generating differentially compressed output from binary sources. Given two versions of the same file as input streams, a compact encoding of one of the input streams is generated, by representing it as a set of changes with respect to the other input stream. Algorithms for differencing files requiring time linear in the size of the input and a constant amount of space for execution are presented. In addition, advanced techniques for improving existing differencing algorithms are developed and applied to previous methods. These techniques allow algorithms to increase their efficiency without a loss of compression and to accept arbitrarily large inputs without sacrificing correctness or degrading the compression data rate. The differential compression methods provide a computationally efficient compression technique for applications that generate versioned data.

System and method for differential compression of data from a plurality of binary sources

A computing system provides continuous availability of data files that are maintained at a file management system and linked to a database management system (DBMS) through a Datalink data type, even while any particular file is being changed with DBMS append or update operations. When a file is linked, it is designated to be available for read-only operations. A user who wants to perform updates on a file gets a 'check-out' copy of the file for updating operations, such that the original file remains linked to the database system while the copy is being updated and remains available to other users. The file management system includes a 'check-in' function that receives the updated file, saves the updated file under a new name different from the original, updates the Datalink, generates new metadata for the updated file, and transactionally updates the file with its new metadata. In this way, data files are continuously available to all users through appending and updating actions. Since updating Datalinks requires the file management system to initiate backup, a 'delta versioning' operation, which reduces the data needed to support backup operations, permits more efficient backup of data files and enables the continuously available files to be backed up and consistent.

Continuous availability and efficient backup for externally referenced

The present invention discloses a method, apparatus, and article of manufacture for generating, transmitting, replicating, and rebuilding in-place reconstructible software updates to a file from a source computer to a target computer updates to a file from a source computer to a target computer. The present invention stores the first version of the file and the updates to the first version of the file in the memory of the source computer. The first version is also stored in the memory of the target computer. The updates are then transmitted from the memory of the source computer to the memory of the target computer. The present invention uses these updates at the target computer to build the second version of the file in-place.

Method for generating and reconstructing in-place delta files

A network storage device is disclosed for use in a secure array of such devices to support a distributed file system. Each device is an independent repository of remotely encrypted data objects to be accessed by authorized network clients. All encryption is done by the clients, rather than by the devices. In order for the system to revoke access to an object on the device, the object must be re-encrypted. Each storage device has a device owner for controlling access to the device's data. All data requests from the clients and responses to them are authenticated using keys derived from the owner key and hashed message authentication codes.

Growing community of inventors

Sunnyvale, CA, United States of America

Randal Chilton Burns