Rackspace Us, Inc.

Intel Corporation

Nvidia Corporation

Swiftstack, Inc.

John A Dickinson

Gregory Lee Holt

Charles B Thier

Michael Barton

Will Reese

Jay B Payne

Clay Gerrard

Samuel N Merritt

Paul Voccio

Alexander Leonard Walsh

Matthew Charles Dietz

Paul E Luse

Antony Joel Messerli

Vishvananda Ishaya

Trey Eugene Morris

William Randolph Mathews, Iv

Christopher James Behrens

Edward Leafe

Jason LuVern Kolker

Ziad Sawalha

Paul Luse

Yogeshwar Srikrishnan

Yogeshwar Srikrishan

Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed. In some examples, an apparatus to compile fragments of an object includes a fragment compiler to: compile an object from fragments stored in storage nodes, respective ones of the fragments corresponding to (a) a node index of storage identifiers representative of the storage nodes and (b) a fragment index of fragment identifiers associated with the respective ones of the fragments of the object, respective ones of the fragment identifiers being representative of a sequential order of the fragments of the object, the respective ones of the fragment identifiers to be associated with the respective ones of the storage identifiers to enable verification of storage locations of the respective ones of the fragments of the object relative to respective storage nodes; request a first one of the fragments from a first one of the respective storage nodes; determine if a first fragment index assigned to the first one of the fragments matches a first node index assigned to the first one of the fragments; when the first fragment index matches the first node index, compile the first one of the fragments into the object based on the first node index; and when the first fragment index does not match the first node index, compile the first one of the fragments into the object based on the first fragment index.

Methods and apparatus to assign indices and relocate object fragments in distributed storage systems

A technique is applied to eventually converge on a single data storage strategy for any set of object data which had an inconsistent data storage strategy applied during storage while there was a network partition. This state could occur in instances of a highly available distributed object storage system which can store objects according to multiple data storage strategies. Upon the healing of a network partition, the technique discovers if multiple data storage strategies were applied to the object data stored during a network partition, deterministically identifies which data storage strategy represents the correct strategy, for example based on the log of state transitions requested by the client according to the API contract, and ensures that this strategy is consistently applied to all object data in the collection.

Reconciler for a distributed storage system

Methods and apparatus to dynamically assign and relocate object fragments in distributed storage systems are disclosed. In some examples, the methods and apparatus encode an object with error correction coding to separate the object into fragments, create a first index indicative of storage nodes where the fragments of the object are to be stored, encode a second index into identifiers of the fragments of the object, the second index based on the first index, and store the fragments of the object and the corresponding second index encoded identifiers in the storage nodes based on the first index.

In one embodiment, a cloud computing system provides user extensibility by providing a plugin interface for major systems. Plugin interfaces for a compute service, object service, network service, authentication and authorization service, message service, and image service are disclosed. One or more of the plugin interfaces can be used to alter the allocation of virtual to physical resources across multiple services. Compound services and smart allocation facilities are possible with user-directed modification.

Pluggable allocation in a cloud computing system

An example method for storing data includes providing a plurality of physical storage pools, each storage pool including a plurality of storage nodes coupled to a network. The method also includes mapping a partition of a plurality of partitions to a set of physical storage pools, where each physical storage pool of the set of physical storage pools is located in a different availability zone, and the storage nodes within an availability zone are subject to a correlated loss of access to stored data. The method further includes receiving a data management request over the network, the data management request being associated with a data object. The method also includes identifying a first partition of the plurality of partitions corresponding to the received data management request and manipulating the data object in the physical storage pools mapped to the first partition in accordance with the data management request.

Massively scalable object storage for storing object replicas

Several different embodiments of a massively scalable object storage system are described. The object storage system is particularly useful for storage in a cloud computing installation whereby shared servers provide resources, software, and data to computers and other devices on demand. In several embodiments, the object storage system includes a ring implementation used to associate object storage commands with particular physical servers such that certain guarantees of consistency, availability, and performance can be met. In other embodiments, the object storage system includes a synchronization protocol used to order operations across a distributed system. In a third set of embodiments, the object storage system includes a metadata management system. In a fourth set of embodiments, the object storage system uses a structured information synchronization system. Features from each set of embodiments can be used to improve the performance and scalability of a cloud computing object storage system.

Massively scalable object storage system

The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. Manipulation of the individual object segments and/or the manifest is also used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Appending to files via server-side chunking and manifest manipulation

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Parallel upload and download of large files using bittorrent

A segmented object storage system is an object storage system that divides files into a number of object segments, each segment corresponding to a portion of an object, and stores each segment individually in the storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. Manipulation of the individual object segments and/or the manifest can be used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Growing community of inventors

Schertz, TX, United States of America

John A Dickinson