International Business Machines Corporation

Pure Storage, Inc.

Cleversafe, Inc.

Ocient Holdings LLC

Other

Purage Storage, Inc.

Greg R Dhuse

Jason K Resch

Andrew D Baptist

Ilya Volvovski

Wesley B Leggette

Gary W Grube

S Christopher Gladwin

Ravi V Khadiwala

Manish Motwani

Timothy W Markison

Thomas Franklin Shirley, Jr

Bart Richard Cilfone

Ethan S Wozniak

Zachary J Mark

Adam Michael Gray

Scott Michael Horan

Yogesh Ramesh Vedpathak

Thomas Darrel Cocagne

Kumar Abhijeet

Michael Colin Storm

Daniel J Scholl

Vance T Thornton

Praveen Viraraghavan

Trevor J Vossberg

John Quigley

Franco Vincent Borich

Bruno Hennig Cabral

Jamie Bellanca

Matthew Ashbeck

Srinivas Palthepu

Ahmad Alnafoosi

Anna Veselova

Niall John McShane

Thomas Dubucq

Benjamin Daniel Rabe

Jason Arnold

Sebastien Vas

Joseph M Kaczmarek

Sarah Kate Schieferstein

Tyler K Reid

Richard George Wendel, Iii

James L Lester

Sarah Toledano

George Kondiles

Harsha Hegde

Robert Clark McCammon

Russell Paul Kennedy

Samuel Peter Bove

Gregory Palinckx

James Sherer

Lynn Foster

Daniel Coombs

Ilir Iljazi

Dustin M Hendrickson

Ram S Narayanan

Ellis Mihalko Saupe

Jesse Louis Young

Susmita Saha

Randy D Pfeifer

Kevin Michael Freese

Pieter Charles Jas Svenson

Dusty Hendrickson

Max Guthmann

Hassan Farahani

Mainsh Motwani

Jerri Lynn Foster

Asimuddin Kazi

Brian Frederick Ober

Joseph Jablonski

Patrick Aaron Tamborski

Amit H Lamba

Renars W Narubin

Jordan Harrison Williams

Akila Srinivasan

Mingyu Li

Ying Zhao Guo

Bart Richard Clifone

Finley Jordan Lau

Anthony J Baldocchi

Brian S Farrell

Gregory A Papadopoulos

Natalia Marie Jenuwine

Andrew Michael Bass

Russell C Fordyce

Austen Thomas Darre

Darius Russell Kish

Wesley H Darvin

Richard Wang

Ilya Volvolvski

Christopher S Gladwin

Ilya Volvoski

John Guigley

Zackary Mark

Jr Thomas Franklin Shirley

A database system is operable to determine a dataset for storage that includes, for each row, a plurality of column values corresponding to a plurality of columns. A plurality of column slab data are generated from the dataset, where each of the plurality of column slab data includes column values for one of the plurality of columns. Each of a plurality of compression dictionaries are trained from a corresponding one of the plurality of column slab data. Each segment of the plurality of segments is generated based on writing a set of compressed column slabs to the each segment based on compressing each of a corresponding set of the plurality of column slab data as a set of variable-length compression frames written to the each segment based on applying a corresponding one of the plurality of compression dictionaries.

Generating compressed column slabs for storage in a database system

A computing device includes an interface configured to interface and communicate with a storage network (SN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. Based on a detected storage error, the computing device is configured to determine availability status of encoded data slices (EDSs) within a set of EDSs. When at least a decode threshold number of EDSs are available, the computing device is configured to initiate a rebuilding function to abate the detected storage error. When less than a decode threshold number of EDSs are available, the computing device is configured to initiate a data recovery function for at least one storage unit (SU) to abate the detected storage error.

Data recovery in a distributed storage network

A storage network operates by: sending a slice verification request to a destination storage unit, wherein the slice verification request includes a nonce and one or more of: a slice identifier, the slice, a revision indicator, or a verification method indicator; receiving an integrity value from the destination storage unit; determining when the integrity value is verified by determining one of: when a hash of the slice and the nonce matches the integrity value; when a decrypted signature of the integrity value matches a hash of the slice and the nonce; or when the decrypted signature of the integrity value matches the slice and the nonce; and when the integrity value is verified, updating a slice assignment corresponding to the slice, wherein updating the slice assignment associates the destination storage unit with the slice and disassociates the source storage unit from the slice.

Verifying slice migration in a storage network

A database system is operable to generate an addendum part indicating deletion of a set of rows of a plurality of rows of a dataset based on generating a row list based on identifying the set of rows for deletion from the dataset and compressing the row list in a set of compressed blocks written to disk memory resources. Based on determining to process the addendum part, the addendum part is read based on decompressing the row list based on access in the set of compressed block in the disk memory resources and emitting the row list for processing.

Generating addendum parts for subsequent processing via a database system

Methods and apparatus for preference based selection of storage network memory for data storage. In an example, a computing device receives a data object for storage in memory of the storage network and determines a system level storage efficiency preference associated with the data object. The computing device selects a set of storage nodes of a plurality of sets of storage nodes for storage of the data object based, at least in part, on the system level storage efficiency preference. The computing device further determines dispersed storage error encoding parameters for the data object, encodes the data object in accordance with the dispersed storage error encoding parameters to produce encoded data slices, and generates system addressing information for the encoded data slices.

Preference based selection of storage network memory for data storage

A database system operates by: storing a plurality of variable-length data of a column of a plurality of rows; storing a plurality of fixed-length index values of a probabilistic indexing scheme for the column; and identifying a filtered subset of the plurality of rows having variable-length data of the column equal to a given value based on: identifying a first subset of rows as a proper subset of the plurality of rows based on the plurality of fixed-length index values; and comparing the variable-length data of only rows in the first subset of rows to the given value to identify the filtered subset as a subset of the first subset of rows.

Database system utilizing probabilistic indexing

A query processing system operates by: determining a query for execution that indicates a join expression and further indicates a threshold maximum number of output rows for the join expression; determining a query operator execution flow for the join expression that includes performance of two join operations based on the threshold maximum number of output rows for the join expression, wherein a union of output of the two join operations is semantically equivalent to an output of the join expression; and executing the query based on: performing the two join operations in parallel upon sets of input rows; and based on determining a set of output rows outputted by the two join operations has reached the threshold maximum number of output rows, finalizing execution of the query before at least one of the two join operations has finished processing its input rows.

Query processing with limit optimization in a database system

A processor in a storage network operates by: receiving an access request for a data segment, wherein the data segment is encoded utilizing an error correcting information dispersal algorithm as a set of encoded data slices that are stored in a plurality of storage units of the storage network and wherein each encoded data slice of the set of encoded data slices includes a corresponding checksum of a plurality of checksums; retrieving, from the storage network, a subset of encoded data slices that includes a threshold number of encoded data slices of the set of encoded data slices; determining, based on ones of the plurality of checksums corresponding to the subset of encoded data slices, when the subset of encoded data slices includes at least one corrupted encoded data slice; retrieving from at least one of the plurality of storage units an addition number of encoded data slices required to generate a reconstructed data segment based on the subset of encoded data slices; generating the reconstructed data segment in accordance with the error correcting information dispersal algorithm, using the additional number of encoded data slices and at least some of the subset of encoded data slices; providing the reconstructed data segment in response to the access request; forming a reconstructed set of encoded data slices utilizing the error correcting information dispersal algorithm on the reconstructed data segment; and replacing the at least one corrupted encoded data slice with at least one reconstructed encoded data slice of the reconstructed set of encoded data slices.

Data reconstruction in a storage network and methods for use therewith

A processing module of a database system is operable to determining a query for execution. A plurality of row sets are processed over a temporal period in conjunction with execution of the query based on automatically selecting a requested number of rows to be included in each row set based on row scheduling parameters, generating a scheduling request for the requested number of rows, and receiving each row set for processing in response to the scheduling request via access to one segment of a set of segments required for execution of the query. A query resultant for the query is generated based on processing of the plurality of row sets, and further based on processing of an additional plurality of rows included in the set of segments via a set of other processing modules implemented in parallel with the processing module.

Query execution via scheduling segment chunks for parallelized processing based on requested number of rows

A method for execution by at least one processor of a database system includes determining whether a first operator of a query execution plan is an input to a second operator of the query execution plan such that the second operator is trapped at a non-leaf level of the query execution plan. When the second operator is trapped at the non-leaf level of the query execution plan, the method further includes generating an updated query execution plan, wherein the updated query execution plan includes the first operator being distributed to produce a plurality of distributed first operators such that the second operator executes at a lower level than the non-leaf level of the query execution plan. The method further includes executing the updated query execution plan to produce a query resultant.

Distributed limit processing in a database system

A database system is operable to determine a query for execution. A disjunction of a plurality of conjunctive normal form (CNF) filtering predicates is determined based on filtering predicates of the query. An IO pipeline is generated based on the disjunction of the plurality of CNF filtering predicates that includes, based on applying at least one type of optimization, at least one shared IO element common to multiple ones of plurality of the CNF filtering predicates. The IO pipeline is executed in accordance with execution of the query to generate a filtered subset of a plurality of rows that each meet at least one of the plurality of CNF filtering predicates.

Query processing in a database system based on applying a disjunction of conjunctive normal form predicates

A database system is operable to generate a first segment that includes a first subset of a plurality of rows. A first secondary indexing scheme for indexing the first segment is selected from a plurality of secondary indexing options, and first index data for the first segment is generated in accordance with the first secondary indexing scheme. A second segment is generated that includes a second subset of the plurality of rows. A second secondary indexing scheme for indexing the second segment is selected from the plurality of secondary indexing options, and second index data is generated for the second segment in accordance with the second secondary indexing scheme. Execution of a query is facilitated by utilizing the first index data and the second index data.

Implementing different secondary indexing schemes for different segments stored via a database system

A method for execution by one or more computing devices of a storage network includes dispersed storage error encoding a data segment of data into a set of encoded data slices in accordance with an information dispersal algorithm width number and a decode threshold number, where the decode threshold number of encoded data slices of the set of encoded data slices are needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in primary storage slots of a set of storage units deployed across multiple sites of the storage network in accordance with a mapping of the primary storage slots to storage units of the set of storage units, where a number of the primary storage slots is equal to or greater than the information dispersal algorithm width number.

Storing encoded data slices in primary storage slots

A database system is operable to maintain lock state data that includes a mapping of a plurality of lock state structures to a plurality of lock scope identifiers. Each of the plurality of lock state structures maintains a set of lock requests based on storing a queue of pending lock requests and a set of granted lock requests. A lock management function is executed to update the lock state data for one of the plurality of lock state structures. In response to executing the lock management function, an evaluation function is executed to further update the one of the plurality of lock state structures based on moving at least one lock request from the front of the queue of pending lock requests to the set of granted lock requests based on determining to move the at least one lock request.

Executing of lock management functions to updating lock state data of a database system

A method for execution by one or more computing devices of a massive data ingestion system includes receiving data for storage in the massive data ingestion system. The method further includes storing a data segment of the data in a cache memory of the massive data ingestion system. The method further includes retrieving, based on one or more retrieval parameters, the data segment from the cache memory. The method further includes error encoding, in accordance with error encoding parameters, the data segment to produce a set of encoded data slices, wherein the error encoding parameters include a pillar width number and a decode threshold number, and wherein the decode threshold number is a minimum number of encoded data slices needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in a set of storage units of the massive data ingestion system.

Ingesting and storing data in a massive data ingestion system

Growing community of inventors

Chicago, IL, United States of America

Greg R Dhuse