International Business Machines Corporation

Pure Storage, Inc.

Ahmad Alnafoosi

Jason K Resch

Andrew D Baptist

Greg R Dhuse

Ilya Volvovski

S Christopher Gladwin

Wesley B Leggette

A method for execution by one or more computing devices of a storage network includes determining probable error locations associated within a set of storage units of the storage network based on provenance information associated with a set of encoded data slices, where the provenance information is generated as one or more tasks for error encoding a data segment of data into the set of encoded data slices and storing the set of encoded data slices are being executed. The method further includes scanning, by the computing device, the probable error locations to determine whether an error exists for the set of encoded data slices.

Smart error scanning in a storage network

A method for execution by a storage network includes receiving a request pertaining to a data object. Metadata associated with the data object is determined and used to identify data segments associated with the data object and a set of storage units associated with the data segments. Based on a set of query requests, a response is received from a storage unit from the set of storage units. When the response indicates a storage environment that is unfavorable as compared to predetermined performance metrics, the storage network facilitates migration of encoded data slices associated with the storage unit to another storage unit.

Verifying and migrating data slices in a storage network

A method includes obtaining, by a computing device of a storage network, provenance information for data associated with a set of storage units of the storage network, where the data is error encoded into a set of encoded data slices, in accordance with error encoding parameters, for storage in the set of storage units. The method further includes determining, by the computing device, probable error locations associated with the set of storage units based on the provenance information. The method further includes scanning, by the computing device, the probable error locations to determine whether an error exists for the set of encoded data slices.

Prioritizing locations for error scanning in a storage network

A method for execution in a storage network begins by generating a set of query requests for each data segment of a plurality of data segments that includes a corresponding set of slice identifiers of a plurality of sets of slice identifiers, where each slice identifier of the corresponding set of slice identifiers includes a pillar index based on a vault affiliated with the plurality of data segments. The method continues by transmitting the set of query requests to a set of storage units affiliated with the storage network, receiving a plurality of sets of query responses from the storage units, obtaining a data identifier for slice location identification and generating a storage record that includes the data identifier and an identity of the set of storage units. The method then continues by facilitating migration of at least some encoded data slices associated with the plurality of sets of slice identifiers when the storage record compares unfavorably to a storage record requirement.

Migration of data in a distributed storage network using storage records

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. For example, the computing device generates a prioritized request that includes at least one of a task for execution or a priority level based on information stored within a storage unit (SU) of a plurality of storage units (SUs) implemented within the DSN. Note that the information corresponds to a data object that is related to a set of encoded data slices (EDSs) that are distributedly stored within the DSN. The computing device then transmits the prioritized request to the SU and receives, from the SU, a response to the prioritized request.

Computing device facilitating prioritization of task execution within a distributed storage network (DSN)

Prioritization task execution within a storage unit (SU)

A method includes dispersed storage error encoding, by a computing device, a data segment of a data file to produce a set of encoded data slices. The method further includes determining, by the computing device, a storage & error encoding scheme for storing the set of encoded data slices. The method further includes sending, by the computing device, the set of encoded data slices to the set of storage units. The method further includes receiving, by a first storage unit, one or more encoded data slices. The method further includes processing, by the first storage unit, the one or more encoded data slices in accordance with a first version of the storage & erroring encoding scheme to produce a first set of encoded data sub-slices. The method further includes storing, by the first storage unit, the first set of encoded data sub-slices in a set of memory devices.

Enhanced dispersed storage error encoding using multiple encoding layers

Client provided request prioritization hints

Methods for use in a dispersed storage network (DSN) to encode data using threshold based secret sharing schemes. In one embodiment, a computing device uses a first threshold based secret sharing scheme in accordance with first secret sharing parameters to encode a data segment of a data file to produce a set of encoded data shares. The computing device then sends the set of encoded data shares to a set of storage units, including sending an encoded data share to a first storage unit of the set of storage units. Upon receiving the encoded data share, the first storage unit encodes the encoded data share using a second threshold based secret sharing scheme in accordance with second secret sharing parameters to produce a set of encoded data sub-shares. The first storage unit then stores the set of encoded data sub-shares in a set of memory devices of the storage unit.

Application of secret sharing schemes at multiple levels of a dispersed storage network

A method includes storing a first set of data slices in a standard-width vault using a first group of DS units of a distributed storage network (DSN). The first set of data slices corresponds to a first representation of a data object, and includes a first write-threshold number of data slices encoded using a first set of dispersal parameters specifying first write and read widths associated with the standard-width vault. A second set of data slices, corresponding to a second representation of the data object, is stored in a low-width vault using a second group of DS units. The second set of data slices includes at least a second write-threshold number of data slices encoded using a second set of dispersal parameters specifying second write and read widths associated with the low-width vault. The low-width vault has a lower width read and/or write width than the standard-width vault.

Low-width vault in distributed storage system

A method includes receiving a random order of encoded data slices, and de-randomizing the random order of encoded data slices to produce a plurality of sets of a transmit number of encoded data slices. The method further includes dividing the plurality of sets of transmit number of encoded data slices into collections of sets of the transmit number of encoded data slices. For a collection of sets of transmit number of encoded data slices of the collections of sets of the transmit number of encoded data slices, the method further includes concurrently decoding the sets of the transmit number of encoded data slices of the collection of sets of transmit number of encoded data slices to produce a collection of data segments and combining the collection of data segments into the one or more data partitions. The method further includes combining the plurality of data partitions into the data object.

Robust transmission of data utilizing encoded data slices

A method begins by a dispersed storage (DS) processing module obtaining a plurality of data access requests and a request to execute a distributed computing function that includes a plurality of partial tasks. The method continues with the DS processing module allocating execution of the plurality of partial tasks to a set of distributed storage and task execution (DSTE) units in accordance with a desired executing efficiency and allocating processing of the plurality of data access requests in accordance with a desired data access efficiency. The method continues with the DS processing module establishing the desired executing efficiency and the desired data access efficiency to obtain a desired utilization of the set of DSTE units.

Allocating distributed storage and task execution resources

A method begins by a processing module receiving a random order of encoded data slices and interpreting slice names to de-randomize the encoded data slices into of sets of transmit encoded data slices. The method continues with the processing module determining whether a decode threshold number of encoded data slices of a set of transmit encoded data slices have been received. When not received, the method continues with the processing module determining whether a sufficient number of encoded data slices of the set of transmit encoded data slices are still to be received and waiting until the decode threshold number of encoded data slices are received when encoded data slices are still to be received. When the decode threshold number of encoded data slices are received, the method continues with the processing module decoding the decode threshold number of encoded data slices to recapture a corresponding data segment.

Growing community of inventors

Streamwood, IL, United States of America

Ahmad Alnafoosi