Ocient Holdings LLC

Anna Veselova

George Kondiles

Ravi V Khadiwala

Greg R Dhuse

Matthew Ashbeck

Donald Scott Clark

Benjamin Daniel Rabe

Natalia Marie Jenuwine

Jason Arnold

Finley Jordan Lau

Richard George Wendel, Iii

Pieter Charles Jas Svenson

Alyssa Catherine Wagenmaker

Ellis Mihalko Saupe

Sarah Kate Schieferstein

Susmita Saha

David Charles Boutcher

Hassan Farahani

Austen Thomas Darre

Darius Russell Kish

Richard Wang

A database system is operable to generate an addendum part indicating deletion of a set of rows of a plurality of rows of a dataset based on generating a row list based on identifying the set of rows for deletion from the dataset and compressing the row list in a set of compressed blocks written to disk memory resources. Based on determining to process the addendum part, the addendum part is read based on decompressing the row list based on access in the set of compressed block in the disk memory resources and emitting the row list for processing.

Generating addendum parts for subsequent processing via a database system

A database system is operable to determine a query for execution. A disjunction of a plurality of conjunctive normal form (CNF) filtering predicates is determined based on filtering predicates of the query. An IO pipeline is generated based on the disjunction of the plurality of CNF filtering predicates that includes, based on applying at least one type of optimization, at least one shared IO element common to multiple ones of plurality of the CNF filtering predicates. The IO pipeline is executed in accordance with execution of the query to generate a filtered subset of a plurality of rows that each meet at least one of the plurality of CNF filtering predicates.

Query processing in a database system based on applying a disjunction of conjunctive normal form predicates

A transfer segment group task processing module of a database system is operable to send a first begin segment group transfer instruction to a first storage cluster based on a determination to transfer a set of segments stored via the first storage cluster to a second storage cluster. A target first ownership sequence number is received from the first storage cluster. A second begin segment group transfer instruction to the second storage cluster. A transfer complete notification is received from the second storage cluster. A complete storage instruction is sent to the second storage cluster. An updated second ownership sequence number is received from the second storage cluster indicating updating of second data ownership information of the second storage cluster. A commit transfer instruction is sent to the first storage cluster based on updated second ownership sequence number received from the second storage cluster.

Transfer of a set of segments between storage clusters of a database system

A record processing and storage system operates by: generating a set of pages from a plurality of row data via a plurality of processing core resources, wherein each processing core resource in the plurality of processing core resources generate a corresponding subset of the set of pages, independently from and in parallel with processing of other subsets of the set of pages via other ones of the plurality of processing core resources; facilitating performance of a single storage transaction to store the set of pages; identifying a page set interval based on a plurality of row number intervals of the set of pages; generating, based on completing the single storage transaction, row durability data indicating a least favorably ordered row number of a plurality of row numbers corresponding to the plurality of row data; and transmitting the row durability data to a computing device associated with the plurality of row data.

Generating row durability data in database systems

A method includes performing a search of an inverted index structure indexing values of a column to generate an in-range indexed value set by identifying all indexed values of the inverted index structure falling within a range corresponding to a range-based filter. A set of characteristics of the in-range indexed value set are identified based on performing the search of an inverted index structure. When the set of characteristics compare favorably to the set of index-usage requirements, output is generated based on performing a plurality of searches to the inverted index structure based on the in-range indexed value set. When the set of characteristics compare unfavorably to the set of index-usage requirements, the output is generated without performing any searches to the inverted index structure.

Applying range-based filtering during query execution based on utilizing an inverted index structure

A method includes generating a plurality of pages from a plurality of records received from a plurality of data sources. Deduplication of the plurality of pages is facilitated based on a plurality of page metadata of the plurality of pages based on, for the each page of the plurality of pages. A filtered set of potentially-intersecting pages is identified for each given page as a proper subset of the plurality of pages stored in the page storage system based on first comparison parameters, and an intersecting set of pages that include a row number intersection with the given page is identified as a proper subset of the filtered set of potentially-intersecting pages based on second comparison parameters. Records with records with row numbers included in row number intersections with other pages in the intersecting set of pages are removed from the each page.

Deduplicating records received from multiple data sources

A node of a database system is operable to determine, at a first time, to prepare for a shutdown. The shutdown of the node is performed at a second time that is a period of time after the first time based on the node delaying the shutdown until a plurality of currently running processes being run by the node that initiated prior to the first time are determined to be complete. During the period of time after the first time and prior to performing the shutdown, a set of new processing requests are rejected by the node.

Performing shutdown of a node in a database system

A database system is operable to store a plurality of column values for a first column of a plurality of rows. A query having a query predicate indicating a negation of a condition for the first column based on a first value is executed. A first index element is applied to identify a first proper subset of the plurality of rows having values for the first column meeting the condition based on accessing index data. At least one second index element is applied to identify a second proper subset of the plurality of rows having values for the first column meeting at least one missing data-based condition based on accessing the index data. A set difference between the plurality of rows and a union of the first proper subset and the second proper subset is identified. A query resultant for the query is generated based on the set difference.

Accessing index data to handle null values during execution of a query that involves negation

A database system is operable to write to a file buffer corresponding to geospatial index data for a plurality of rows based on processing each given row of the plurality of rows. A new leaf node of a set of leaf nodes in a temporary leaf node buffer when the given row includes a geospatial object. When the temporary leaf node buffer is determined to have a number of leaf nodes meeting a predetermined threshold number of leaf nodes, a new tree-based index structure of a set of tree-based index structures of the geospatial index data is built via processing the temporary leaf node buffer. The geospatial index data is stored based on writing the file buffer to disk memory resources. A query is executed against a relational database table based on accessing the geospatial index data in the disk memory resources.

Structuring geospatial index data for access during query execution via a database system

A method includes generating a predictive model mapping batch size to estimated memory utilization based on a plurality of prior batch sizes applied for performing a corresponding plurality of prior data format conversions, and further based on a plurality of prior actual memory utilization occurring in performing the corresponding plurality of prior data format conversions. A next batch size is automatically selected for performance of a subsequent data format conversion based on the predictive model and a current memory availability. A batch of data for performance of the subsequent data format conversion is identified based on the batch of data having the next batch size. The subsequent data format conversion is performed the batch of data, and an actual amount of memory utilized in performing the subsequent data format conversion is determined. The predictive model is updated based on the next batch size and the actual amount of memory utilized.

Generation of a predictive model for selection of batch sizes in performing data format conversion

A method for execution by at least one processor of a database system includes storing a plurality of column values for a first column of a plurality of rows. Each of a set of missing data-based conditions for the first column are indexed via an indexing scheme. A subset of the set of missing data-based conditions that satisfy a query predicate of a query are identified. An IO pipeline for access of the first column based on the query predicate and further based on the subset of the set of missing data-based conditions is generated. The IO pipeline is applied in conjunction with execution of the query by applying at least one index element to identify a proper subset of the plurality of rows based on index data of the indexing scheme, and by generating a query resultant based on the proper subset.

Missing data-based indexing in database systems

A record processing and storage system is operable to receive a plurality of row data that each indicate a batch number. Each of the plurality of row data is added to a pending row data pool. A plurality of pages is generated from the plurality of row data via a plurality of processing core resources. Each processing core resource in the plurality of processing core resources processing a corresponding subset of the plurality of row data by retrieving, in each time slice of a plurality of time slices, one row data from the pending row data pool with a most favorably ordered batch number of row data in the pending row data pool. Each processing core resource further processes the one row data in the in each time slice to participate in generation of at least one of the plurality of pages.

Processing row data via a plurality of processing core resources

A record processing and storage system is operable to receive a plurality of labeled row data from a data source. Each labeled row data of the plurality of labeled row data includes at least one record and a corresponding row number of a plurality of row numbers. A plurality of pages are generated from records included in the labeled row data. The plurality of pages are stored via a page storage system. A plurality of page metadata corresponding to the plurality of pages is generated, where each of the plurality of page metadata is generated based on at least corresponding one row number of at least one labeled row data with records included in a corresponding one of the plurality of pages. Deduplication of duplicated records included the plurality of pages is facilitated based on the plurality of page metadata.

Processing row data for deduplication based on corresponding row numbers

A database system operates by: receiving a plurality of row data associated with a first data source; identifying a subset of row data from the plurality of row data that includes only ones of the plurality of row data that compare favorably to maintained row durability data; generating at least one page from ones of the plurality of row data included in the subset of row data; storing the at least one page in long term storage; generating updated row durability data indicating a least favorably ordered row number of a plurality of row numbers corresponding to the subset of row data based on storing the at least one page in long term storage; and updating the maintained row durability data to indicate the least favorably ordered row number of the updated row durability data.

Maintaining row durability data in database systems

A method for execution by a record processing and storage system includes receiving a plurality of records and corresponding row numbers. Pages are generated from the received records. Page metadata is generated for each page that includes row number span data based on row numbers of the records included in each page. Pairs of pages are identified in the plurality of pages based on having row number span data in their page metadata that include a row number span overlap. For each pair of pages, row number span data is updated for a first page in the pair by removing the row number span overlap with a second page in the pair. Reads of pages are performed based on their row number span data. Only records of each first page of each pair of pages having row numbers that are within the updated row number span data are read.

Growing community of inventors

Chicago, IL, United States of America

Anna Veselova