International Business Machines Corporation

Pure Storage, Inc.

Ocient Holdings LLC

Hulu, LLC

Green Market Square Limited

Ravi V Khadiwala

Jason K Resch

Greg R Dhuse

Andrew D Baptist

Wesley B Leggette

Ilya Volvovski

Manish Motwani

Ethan S Wozniak

Scott Michael Horan

Yogesh Ramesh Vedpathak

Adam Michael Gray

Daniel J Scholl

Bart Richard Cilfone

Thomas Darrel Cocagne

Gary W Grube

Michael Colin Storm

Bruno Hennig Cabral

Franco Vincent Borich

Timothy W Markison

Trevor J Vossberg

Praveen Viraraghavan

Asimuddin Kazi

George Kondiles

Anna Veselova

Joseph M Kaczmarek

Thomas Franklin Shirley, Jr

Mingyu Li

Niall John McShane

Renars W Narubin

Donald Scott Clark

Tyler K Reid

Thomas Dubucq

Baptiste Coudurier

Bohdan Lew Bodnar

Jeremy S Jarczyk

John R Carrell

Shikha Shree

Alex Marchenko

S Christopher Gladwin

Michael Edward Factor

Kumar Abhijeet

Brian Frederick Ober

Roberta Jo Cochrane

Vita Bortnikov

Teague Scott Algie

Natalia Marie Jenuwine

Andrew Michael Bass

James L Lester

Anthony J Baldocchi

Russell C Fordyce

Michele Martino Franceshini

A processing system of a storage network operates by: sending, to at least one storage unit of the storage network, at least one read request corresponding to at least a read threshold number of a set of encoded data slices to be retrieved, wherein the set of encoded data slices correspond to data, wherein the data is coded in accordance with dispersed error coding parameters that include a write threshold number and the read threshold number, wherein the write threshold number is a number of encoded data slices in the set of encoded data slices and wherein the read threshold number is a number of the set of encoded data slices that is required to decode the data; receiving, at the at least one processing circuit and from the at least one storage unit, a first subset of the set of encoded data slices, wherein at least one missing encoded data slice was not included in the first subset and wherein a number of encoded data slices in the first subset is less than the read threshold number; generating, via the at least one processing circuit, at least one rebuilt encoded data slice corresponding to the at least one missing encoded data slice utilizing locally decodable redundancy data, wherein the locally decodable redundancy data corresponds to a second subset of the set of encoded data slices that includes the at least one missing encoded data slice and wherein the locally decodable redundancy data is stored locally to the processing circuit; and recovering, via the at least one processing circuit, the data based on the at least one rebuilt encoded data slice and the first subset.

Recovering missing data in a storage network via locally decodable redundancy data

A method includes determining rebuilding performance parameter values for a storage unit of a set of storage units, where data is dispersed storage error encoded into a plurality of sets of encoded data slices in accordance with error encoding parameters, and a set of encoded data slices are stored or are to be stored in the set of storage units. The method includes updating a rate of internal rebuilding of encoded data slices by the storage unit based on the rebuilding performance parameter values. During internally rebuilding, the method includes correcting detected storage errors for one or more first encoded data slices within the storage unit. The method includes receiving a rebuilt encoded data slice from another computing device of the storage network and when the received rebuilt encoded data slice is not included in the one or more first encoded data slices, storing the received rebuilt encoded data slice.

Dispersed rebuilding of encoded data slices in a storage network

A method for execution by one or more computing devices of a storage network includes dispersed storage error encoding a data segment of data into a set of encoded data slices in accordance with an information dispersal algorithm width number and a decode threshold number, where the decode threshold number of encoded data slices of the set of encoded data slices are needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in primary storage slots of a set of storage units deployed across multiple sites of the storage network in accordance with a mapping of the primary storage slots to storage units of the set of storage units, where a number of the primary storage slots is equal to or greater than the information dispersal algorithm width number.

Storing encoded data slices in primary storage slots

A processing system is operable to encode data to produce a first set of data slices based on a value of a width parameter. The data is stored based on maintaining storage of the first set of data slices across a set of storage units of a storage pool. Storage of the first set of data slices is maintained in the set of storage units of the storage pool after addition of an additional set of storage units added to the storage pool. The value of the width parameter is increased to an increased value to produce an updated width parameter. The data is re-encoded in accordance with the updated width parameter to produce a second set of data slices. The data is re-stored based on maintaining storage of the second set of data slices across the expanded set of storage units of the storage pool.

Re-encoding data in a storage network based on addition of additional storage units

A method includes generating a first lock requests, each for transmission to a storage unit, where each first lock request indicates a first encoded data slice. First ballots are received, each of the first ballots corresponds to a storage unit, each of the first ballots indicates a first ranking of first requesting entities that includes the DST processing unit. First election result data is generated based on the first ballots. The first election result data indicates a first winner. When the first winner is the DST processing unit, slice access requests are generated in response to the first election result data indicating that the first winner is the DST processing unit, each for transmission to one of the storage units, where each of the slice access requests indicates the first encoded data slice.

Establishing distributed consensus via alternate voting strategies in a dispersed storage network

A record processing and storage system operates by: generating a set of pages from a plurality of row data via a plurality of processing core resources, wherein each processing core resource in the plurality of processing core resources generate a corresponding subset of the set of pages, independently from and in parallel with processing of other subsets of the set of pages via other ones of the plurality of processing core resources; facilitating performance of a single storage transaction to store the set of pages; identifying a page set interval based on a plurality of row number intervals of the set of pages; generating, based on completing the single storage transaction, row durability data indicating a least favorably ordered row number of a plurality of row numbers corresponding to the plurality of row data; and transmitting the row durability data to a computing device associated with the plurality of row data.

Generating row durability data in database systems

A method includes generating a plurality of pages from a plurality of records received from a plurality of data sources. Deduplication of the plurality of pages is facilitated based on a plurality of page metadata of the plurality of pages based on, for the each page of the plurality of pages. A filtered set of potentially-intersecting pages is identified for each given page as a proper subset of the plurality of pages stored in the page storage system based on first comparison parameters, and an intersecting set of pages that include a row number intersection with the given page is identified as a proper subset of the filtered set of potentially-intersecting pages based on second comparison parameters. Records with records with row numbers included in row number intersections with other pages in the intersecting set of pages are removed from the each page.

Deduplicating records received from multiple data sources

Methods and apparatus for asynchronous replication of data in a storage network. In an embodiment, a processing module(s) of a computing device identifies at least a first storage set and a second storage set for replicated storage of data. The processing module maintains a synchronization schedule for the first storage set and the second storage set. After initiating storage of a data object in the first storage set (e.g., using first error encoding parameters), the processing module determines, based at least in part on the synchronization schedule, to synchronize the first storage set and the second storage set. In response to determining to synchronize the first and second storage sets, the processing module determines that the second storage set requires the data object to maintain synchronization with the first storage set and facilitates storage of the data object in the second storage set (e.g., using second error encoding parameters).

Asynchronous data replication in a storage network

A storage system is operable to transition from operation in accordance with a first storage scheme to operation in accordance with a second storage scheme by, for each storage device of the plurality of storage devices, expanding a storage size of a single storage structure of the corresponding plurality of data storage structures identified to implement the second storage scheme from an initial size to a final size consuming full storage resources of the each storage device based on writing all new data of a plurality of write requests to the single storage structure in accordance with the second storage scheme, performing at least one expansion of the single storage structure within the each storage device in response to an expansion condition being met, and completing transition based on the single storage structure reaching the final size consuming the full storage resources of the each storage device.

Storage scheme transition via storage structure expansion in a storage system

A method for execution by one or more processing modules of a storage network begins by receiving an access request for a set of encoded data slices, where the data object is segmented into a plurality of data segments, a data segment of which is dispersed error encoded in accordance with dispersed error encoding parameters to produce the set of encoded data slices. The method continues by determining whether a revision for one or more encoded data slices in a first storage module is a most recent revision level for the one or more encoded data slices and when the one or more encoded data slices is a most recent revision level, determining whether the revision level for the one or more encoded data slices in a second storage module is the most recent revision level. When the one or more encoded data slices in the second storage module is not the most recent revision level the method continues by facilitating a read request to the first storage module for the one or more encoded data slices and synchronizing the one or more encoded data slices in the second storage module with the one or more encoded data slices in the first storage module.

Managing copy revisions in a distributed storage system

A method for execution by one or more computing devices of a storage network includes obtaining performance impact information regarding a data reconstruction operation associated with a computing device of the one or more computing devices, where the data reconstruction operation is regarding reconstructing data at a first reconstruction rate of a plurality of reconstruction rates, and where the performance impact information includes performance metrics of the storage network affected by the data reconstruction operation. The method further includes determining a second reconstruction rate of the plurality of reconstruction rates to utilize for the data reconstruction operation based on the performance impact information to achieve a first performance metric of the performance metrics. The method further includes executing the data reconstruction operation in accordance with the second reconstruction rate.

Performance impact information for adjusting rate of a data reconstruction operation

A method for execution by a distributed storage network begins by receiving a request to transfer a copy of a set of encoded data slices from at least some associated virtual storage vaults to a destination virtual storage vault and continues by determining whether the destination storage unit supports a source virtual storage vault of the at least some source virtual storage vaults. When the destination storage unit supports the source virtual storage vault the method continues by determining a sub-set of encoded data slices of the set of encoded data slices for transfer and finally, by facilitating sending the sub-set of encoded data slices to the destination storage unit.

Synchronized vault management in a distributed storage network

A processing system of a storage network operates by: sending, to at least one storage unit of the storage network, at least one read request corresponding to at least a read threshold number of a set of encoded data slices to be retrieved, wherein the set of encoded data slices correspond to data, wherein the data is coded in accordance with dispersed error coding parameters that include a write threshold number and the read threshold number, wherein the write threshold number is a number of encoded data slices in the set of encoded data slices and wherein the read threshold number is a number of the set of encoded data slices that is required to decode the data; receiving, via the at least one processing circuit and from the at least one storage unit, a first subset of the set of encoded data slices, wherein the first subset is missing at least one missing encoded data slice that was not received from the at least one storage unit in response to the at least one read request and wherein a number of encoded data slices in the first subset is less than the read threshold number; generating, via the at least one processing circuit, at least one rebuilt encoded data slice corresponding to the at least one missing encoded data slice utilizing locally decodable redundancy data, wherein the locally decodable redundancy data generated from a second subset of the set of encoded data slices that includes the at least one missing encoded data slice; and recovering, via the at least one processing circuit, the data based on the at least one rebuilt encoded data slice and the first subset.

Rebuilding missing data in a storage network via locally decodable redundancy data

A computing device includes an interface configured to interface and communicate with a storage network, a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. A computing device receives a data access request for an encoded data slice associated with a data object, determines whether the encoded data slice is stored in the first memory and in response to a determination that the encoded data slice is not stored in the first memory, issues another data access request for the encoded data slice to a second memory, where the first memory includes access characteristics that are faster than the second memory. When a data access response including the encoded data slice is received from the second memory, a response including the encoded data slice is transmitted.

Utilizing memories of different operational speeds in a vast storage network

A method includes rebuilding an appended encoded data slice of a set of appended encoded data slices stored in a set of storage units of a storage network to produce a rebuilt appended encoded data slice, where the appended encoded data slice includes an encoded data slice of a set of encoded data slices and slice integrity check value information, where the slice integrity check value information includes a slice integrity check value for the encoded data slice, and where the encoded data slice is hashed to produce the slice integrity check value. The method further includes generating current slice integrity check value information for the rebuilt appended encoded data slice. The method further includes verifying the current slice integrity check value information corresponds to the slice integrity check value information. When verified, the method includes storing the rebuilt appended encoded data slice.

Verifiable rebuilding of an appended encoded data slice

Growing community of inventors

Bartlett, IL, United States of America

Ravi V Khadiwala