International Business Machines Corporation

Pure Storage, Inc.

Cleversafe, Inc.

Ocient Holdings LLC

Other

Purage Storage, Inc.

Greg R Dhuse

Jason K Resch

Andrew D Baptist

Ilya Volvovski

Wesley B Leggette

Gary W Grube

S Christopher Gladwin

Ravi V Khadiwala

Manish Motwani

Timothy W Markison

Thomas Franklin Shirley, Jr

Bart Richard Cilfone

Ethan S Wozniak

Zachary J Mark

Adam Michael Gray

Scott Michael Horan

Yogesh Ramesh Vedpathak

Thomas Darrel Cocagne

Kumar Abhijeet

Michael Colin Storm

Daniel J Scholl

Vance T Thornton

Trevor J Vossberg

Praveen Viraraghavan

John Quigley

Franco Vincent Borich

Bruno Hennig Cabral

Jamie Bellanca

Srinivas Palthepu

Matthew Ashbeck

Ahmad Alnafoosi

Niall John McShane

Anna Veselova

Jason Arnold

Sebastien Vas

Joseph M Kaczmarek

Tyler K Reid

Sarah Kate Schieferstein

Richard George Wendel, Iii

Sarah Toledano

James L Lester

Benjamin Daniel Rabe

George Kondiles

Harsha Hegde

Robert Clark McCammon

Russell Paul Kennedy

Samuel Peter Bove

Lynn Foster

James Sherer

Gregory Palinckx

Daniel Coombs

Dustin M Hendrickson

Ram S Narayanan

Jesse Louis Young

Randy D Pfeifer

Ellis Mihalko Saupe

Susmita Saha

Kevin Michael Freese

Dusty Hendrickson

Jerri Lynn Foster

Hassan Farahani

Max Guthmann

Mainsh Motwani

Asimuddin Kazi

Brian Frederick Ober

Joseph Jablonski

Patrick Aaron Tamborski

Ilir Iljazi

Thomas Dubucq

Renars W Narubin

Bart Richard Clifone

Finley Jordan Lau

Andrew Michael Bass

Anthony J Baldocchi

Gregory A Papadopoulos

Brian S Farrell

Natalia Marie Jenuwine

Pieter Charles Jas Svenson

Russell C Fordyce

Austen Thomas Darre

Darius Russell Kish

Ilya Volvolvski

Christopher S Gladwin

Ilya Volvoski

John Guigley

Zackary Mark

Methods and apparatus for preference based selection of storage network memory for data storage. In an example, a computing device receives a data object for storage in memory of the storage network and determines a system level storage efficiency preference associated with the data object. The computing device selects a set of storage nodes of a plurality of sets of storage nodes for storage of the data object based, at least in part, on the system level storage efficiency preference. The computing device further determines dispersed storage error encoding parameters for the data object, encodes the data object in accordance with the dispersed storage error encoding parameters to produce encoded data slices, and generates system addressing information for the encoded data slices.

Preference based selection of storage network memory for data storage

A database system operates by: storing a plurality of variable-length data of a column of a plurality of rows; storing a plurality of fixed-length index values of a probabilistic indexing scheme for the column; and identifying a filtered subset of the plurality of rows having variable-length data of the column equal to a given value based on: identifying a first subset of rows as a proper subset of the plurality of rows based on the plurality of fixed-length index values; and comparing the variable-length data of only rows in the first subset of rows to the given value to identify the filtered subset as a subset of the first subset of rows.

Database system utilizing probabilistic indexing

A query processing system operates by: determining a query for execution that indicates a join expression and further indicates a threshold maximum number of output rows for the join expression; determining a query operator execution flow for the join expression that includes performance of two join operations based on the threshold maximum number of output rows for the join expression, wherein a union of output of the two join operations is semantically equivalent to an output of the join expression; and executing the query based on: performing the two join operations in parallel upon sets of input rows; and based on determining a set of output rows outputted by the two join operations has reached the threshold maximum number of output rows, finalizing execution of the query before at least one of the two join operations has finished processing its input rows.

Query processing with limit optimization in a database system

A processor in a storage network operates by: receiving an access request for a data segment, wherein the data segment is encoded utilizing an error correcting information dispersal algorithm as a set of encoded data slices that are stored in a plurality of storage units of the storage network and wherein each encoded data slice of the set of encoded data slices includes a corresponding checksum of a plurality of checksums; retrieving, from the storage network, a subset of encoded data slices that includes a threshold number of encoded data slices of the set of encoded data slices; determining, based on ones of the plurality of checksums corresponding to the subset of encoded data slices, when the subset of encoded data slices includes at least one corrupted encoded data slice; retrieving from at least one of the plurality of storage units an addition number of encoded data slices required to generate a reconstructed data segment based on the subset of encoded data slices; generating the reconstructed data segment in accordance with the error correcting information dispersal algorithm, using the additional number of encoded data slices and at least some of the subset of encoded data slices; providing the reconstructed data segment in response to the access request; forming a reconstructed set of encoded data slices utilizing the error correcting information dispersal algorithm on the reconstructed data segment; and replacing the at least one corrupted encoded data slice with at least one reconstructed encoded data slice of the reconstructed set of encoded data slices.

Data reconstruction in a storage network and methods for use therewith

A processing module of a database system is operable to determining a query for execution. A plurality of row sets are processed over a temporal period in conjunction with execution of the query based on automatically selecting a requested number of rows to be included in each row set based on row scheduling parameters, generating a scheduling request for the requested number of rows, and receiving each row set for processing in response to the scheduling request via access to one segment of a set of segments required for execution of the query. A query resultant for the query is generated based on processing of the plurality of row sets, and further based on processing of an additional plurality of rows included in the set of segments via a set of other processing modules implemented in parallel with the processing module.

Query execution via scheduling segment chunks for parallelized processing based on requested number of rows

A method for execution by at least one processor of a database system includes determining whether a first operator of a query execution plan is an input to a second operator of the query execution plan such that the second operator is trapped at a non-leaf level of the query execution plan. When the second operator is trapped at the non-leaf level of the query execution plan, the method further includes generating an updated query execution plan, wherein the updated query execution plan includes the first operator being distributed to produce a plurality of distributed first operators such that the second operator executes at a lower level than the non-leaf level of the query execution plan. The method further includes executing the updated query execution plan to produce a query resultant.

Distributed limit processing in a database system

A database system is operable to determine a query for execution. A disjunction of a plurality of conjunctive normal form (CNF) filtering predicates is determined based on filtering predicates of the query. An IO pipeline is generated based on the disjunction of the plurality of CNF filtering predicates that includes, based on applying at least one type of optimization, at least one shared IO element common to multiple ones of plurality of the CNF filtering predicates. The IO pipeline is executed in accordance with execution of the query to generate a filtered subset of a plurality of rows that each meet at least one of the plurality of CNF filtering predicates.

Query processing in a database system based on applying a disjunction of conjunctive normal form predicates

A database system is operable to generate a first segment that includes a first subset of a plurality of rows. A first secondary indexing scheme for indexing the first segment is selected from a plurality of secondary indexing options, and first index data for the first segment is generated in accordance with the first secondary indexing scheme. A second segment is generated that includes a second subset of the plurality of rows. A second secondary indexing scheme for indexing the second segment is selected from the plurality of secondary indexing options, and second index data is generated for the second segment in accordance with the second secondary indexing scheme. Execution of a query is facilitated by utilizing the first index data and the second index data.

Implementing different secondary indexing schemes for different segments stored via a database system

A method for execution by one or more computing devices of a storage network includes dispersed storage error encoding a data segment of data into a set of encoded data slices in accordance with an information dispersal algorithm width number and a decode threshold number, where the decode threshold number of encoded data slices of the set of encoded data slices are needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in primary storage slots of a set of storage units deployed across multiple sites of the storage network in accordance with a mapping of the primary storage slots to storage units of the set of storage units, where a number of the primary storage slots is equal to or greater than the information dispersal algorithm width number.

Storing encoded data slices in primary storage slots

A database system is operable to maintain lock state data that includes a mapping of a plurality of lock state structures to a plurality of lock scope identifiers. Each of the plurality of lock state structures maintains a set of lock requests based on storing a queue of pending lock requests and a set of granted lock requests. A lock management function is executed to update the lock state data for one of the plurality of lock state structures. In response to executing the lock management function, an evaluation function is executed to further update the one of the plurality of lock state structures based on moving at least one lock request from the front of the queue of pending lock requests to the set of granted lock requests based on determining to move the at least one lock request.

Executing of lock management functions to updating lock state data of a database system

A method for execution by one or more computing devices of a massive data ingestion system includes receiving data for storage in the massive data ingestion system. The method further includes storing a data segment of the data in a cache memory of the massive data ingestion system. The method further includes retrieving, based on one or more retrieval parameters, the data segment from the cache memory. The method further includes error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in a set of storage units of the massive data ingestion system.

Ingesting and storing data in a massive data ingestion system

A method for execution by a storage network begins by receiving data for storage by the storage network and continues by determining data preparation tasks for the data. The method continues by indexing the data in accordance with the data preparation tasks to generate a data index and processing the data in accordance with the data index to produce indexed data. The method then continues by determining distribution criteria for the data based on the data index and distributing the data and the data index to a set of distributed storage units in accordance with the distribution criteria, Finally, the method establishes criteria for analyzing found data of the data in the storage network.

Storing, processing and analyzing large volumes of data in a storage network

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

Rebuilding data slices in a storage network based on priority

A data processing system is operable to determining a query for execution and generate a query operator execution flow for the query that includes a first at least one operator serially before a second at least one operator. The first at least one operator of the query operator execution flow is executed based on generating a request for rows in accordance with an object storage communication protocol indicating filtering parameter data parameter, sending the request to an object storage system, and receiving a response indicating a filtered row set from the object storage system. The second at least one operator of the query operator execution flow is executed based on processing the filtered row set indicated in the response in accordance with the second at least one operator to produce the query resultant.

Query execution via communication with an object storage system via an object storage communication protocol

A device for use in a storage network including storage network memory distributed amongst a plurality of storage units. The device includes a processing module operably couple to a memory that stores operational instructions. The processing module is configured to execute the operational instructions to determine storage parameters associated with encoded data slices generated from data to be stored in the plurality of storage units. The storage parameters include information indicating a read threshold number of encoded data slices required to recover the data. The method further includes facilitating distributed storage of the encoded data slices among the plurality of storage units such that the number of the error-encoded data slices stored in any particular storage unit is chosen so that in the event of an unavailability of any individual storage unit, at least a read threshold number of encoded data slices are still accessible from the remaining storage units.

Data availability in a storage network in the event of memory device failure

Growing community of inventors

Chicago, IL, United States of America

Greg R Dhuse