Cleversafe, Inc.

International Business Machines Corporation

Pure Storage, Inc.

Other

Zachary J Mark

Ilya Volvovski

Jason K Resch

Greg R Dhuse

Andrew D Baptist

Wesley B Leggette

Vance T Thornton

S Christopher Gladwin

Sebastien Vas

Manish Motwani

John Quigley

Jamie Bellanca

Bart Richard Cilfone

Dustin M Hendrickson

Matthew Michael England

Srinivas Palthepu

Gary W Grube

Timothy W Markison

Sarah Toledano

James L Lester

Lynn Foster

Jerri Lynn Foster

Joshua J Mullin

Sejal Kumarbhai Modi

Dhanvi Gopala Krishna Kapila Lakshmana Harsha

Dhavi Gopala Krishna Kapila Lakshmana Harsha

Dusty Hendrickson

Mainsh Motwani

Vance T Thorton

James Bellanca

Ilya Volvolvski

Vance T Thomton

A method includes storing a plurality of data in a storage system. A plurality of identifiers corresponding to the plurality of data is determined and the plurality of identifiers are stored in the storage system. A first set of integrity information corresponding to a first system storage level is generated for the plurality of data by performing a first set of cyclic redundancy checks and the first set of integrity information is stored in the storage system. A second set of integrity information corresponding to a second system storage level is generated for the plurality of data and the second set of integrity information is stored in the storage system.

Generating multiple sets of integrity information in a vast storage system

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

Rebuilding data slices in a storage network based on priority

A method includes receiving, by a storage unit of a storage network, a check request message from a computing device of the storage network, where the check request message includes a group of slice names of a plurality of sets of slice names associated with a plurality of sets of encoded data slices, where a plurality of data segments are dispersed storage error encoded into the plurality of sets of encoded data slices. The method further includes determining, by the storage unit, whether an error condition exists for an encoded data slice associated with the group of slice names. When no, the method includes sending, by the storage unit, a check response message to the computing device, where the check response message includes a group of slice information regarding the group of slice names. When yes, the method includes disregarding sending the check response message.

Check requests in a storage network

A method for execution by a computing device of a storage unit of a storage network includes receiving a digest list request from a storage network entity regarding integrity data for a set of encoded data slices where the storage unit stores a plurality of encoded data slices of the set of encoded data slices and corresponding integrity data, where the digest list request is for integrity data of the plurality of encoded data slices. The method continues with the storage unit collecting integrity data, transmitting the collected integrity data to the storage network entity. The method continues with the storage unit receiving an indication that an inconsistency exists in the collected integrity data as compared to integrity data received from one or more other storage units storing a second plurality of encoded data slices of the set of encoded data slices.

End to end verification of data in a storage network

A method includes determining a plurality of identifiers based on a data retrieval request. Integrity information is generated based on determining the plurality of identifiers. Stored integrity information corresponding to the data retrieval request is compared with the integrity information. When the stored integrity information compares unfavorably with the integrity information, corruption associated with the plurality of identifiers is determined.

Utilizing integrity information to determine corruption in a vast storage system

A storage unit operates by: receiving a write slice request, wherein the write slice request includes a plurality of encoded data slices and wherein the write slice request corresponds to a range; determining whether a write lock conflict exists based on the range; issuing an unfavorable write slice response when the write lock conflict is determined to exist; and when the write lock conflict is determined to not exist: initiating local storage of the plurality of encoded data slices; and issuing a favorable write slice response.

Write lock conflicts in a storage network

Integrity processing in a dispersed storage network

A method includes receiving a data retrieval request. A plurality of identifiers are determined in accordance with the data retrieval request. Integrity information is generated based on determining the plurality of identifiers by performing a cyclic redundancy check. Stored integrity information corresponding to the data retrieval request is compared with the integrity information, where the stored integrity information was previously generated by performing the cyclic redundancy check. When the stored integrity information compares unfavorably with the integrity information, corruption associated with the plurality of identifiers is determined.

Utilizing integrity information in a vast storage system

A system includes a plurality of storage units, where one or more storage vaults is associated with the plurality of storage units and each storage vault of the one or more storage vaults represents a software-constructed grouping of storage units of the plurality of storage units. The software-constructed grouping of storage units stores encoded data slices. A data segment is encoded using an information dispersal algorithm to produce the encoded data slices. The system further includes a grid access manager that generates a data structure pertaining to the software-constructed grouping of storage units. A storage unit of the software-constructed grouping of storage units receives, from a client computer of the system, a request regarding the data segment, obtains, from the data structure, information regarding the request, determines whether the request is valid based on the information regarding the request, and when the request is valid, executes the request.

Validating requests based on stored vault information

A method includes encoding data via erasure coding to produce a plurality of data slices. The method further includes determining a plurality of identifiers corresponding to the data. The method further includes generating integrity information based on the plurality of identifiers by performing a cyclic redundancy check. The method further includes storing the plurality of data slices, the plurality of identifiers, and the integrity information in a storage system.

Generating integrity information in a vast storage system

A method for execution by a dispersed storage and task (DST) execution unit includes receiving a write slice request from a requesting entity, where the write slice request includes a plurality of encoded data slices and a corresponding plurality of slice names. The plurality of slice names are interpreted to produce a slice name range. Whether a write lock conflict exists is determined based on the slice name range. An unfavorable write slice response is issued to the requesting entity when the write lock conflict is determined to exist. When the write lock conflict is determined to not exist, local storage of the plurality of encoded data slices is initiated, a favorable write slice response is issued to the requesting entity, and the processing system indicates that the slice name range is not locked when completing the local storage of the plurality of encoded data slices.

Resolving write conflicts in a dispersed storage network

To identify slice errors, a processing module of a computing device in a dispersed storage network (DSN) sends first list digest requests to at least first and second dispersed storage (DS) units. The requests indicates a first range of slice names to include in a first list digest. The processing module receives digest responses from the DS units, and compares the digest responses to determine whether they identify the same slices. If they do not identify the same slices, the processing module sends second list digest requests indicating a sub-range of the first range of slice names to include in second list digests. The sub-range continues to be narrowed until the processing module identifies at least one sub-range of slice names where a slice error exists.

Digest listing decomposition

A method includes generating integrity data of one or more slice names of one or more error encoded data slices and evaluating the integrity data, without using any of the error encoded data slices, for an inconsistency between the plurality of received integrity data. The method further includes indicating an error based on the evaluating the integrity data indicating an inconsistency, retrieving one or more slice names for the one or more error encoded data slices, and when detecting that at least one data error has occurred at a plurality of storage units at a DSN site/pillar, determine that a DSN site/pillar failure has occurred and after a predetermined time period, if the DSN site/pillar failure has not been resolved, rebuild the error encoded data slices of the DSN site/pillar and store them at another DSN site/storage unit.

Method and apparatus for identifying data inconsistency in a dispersed storage network

A method for execution by one or more processing modules of a storage network (SN) begins with the one or more processing modules scanning a storage node of the SN at a predetermined interval to determine whether an erasure encoded data slice (EEDS) of a set of erasure encoded data slices (EEDSs) is corrupt, where the set of erasure encoded data slices (EEDSs) is generated from object data using an erasure error encoding dispersal function. The method continues with the one or more processing modules determining that an EEDS is corrupt and in response, rebuilding the EEDS in place in the storage node.

Growing community of inventors

Chicago, IL, United States of America

Zachary J Mark