Sap Se

Paul Willems

Till Merker

Christina Teflioudi

Oliver Horn

Manuel Mayr

Norman May

Kai-Uwe Sattler

Andreas Amler

Florian Wolf

Stefan Noll

Timm Steinbeck

Zhen Tian

Michael Brendle

Joerg Fellmann

Alexander Ostertag

Alexander Gellner

Cheng-Lun Li

Thomas Peh

Xun Cheng

Nico Bohnsack

Jin Su Lee

Nitesh Maheshwari

Dan Bi Park

Heesik Shin

Marius Eich

Canberk Morelli

Florian Drescher

Yuncong Qiao

Faming Qu

Hongyong Lu

Yanxin Luo

Myunggu Kang

A method for processing non-standardized data in a relational database may include identifying, within a sequence of operations forming a query pipeline for executing a query, a first operation ingesting a non-standardized data. In response to identifying the first operation ingesting the non-standardized data, a second operation may be inserted before the first operation. The non-standardized data may be output by a third operation preceding the first operation or a source external to the query pipeline. The second operation may serialize the non-standardized data for ingestion by the first operation, for example, by generating a relational table populated by the non-standardized data. The query may be executed by performing the sequence of operations included in the query pipeline. Related systems and computer program products are also provided.

Transport of non-standardized data between relational database operations

In an example embodiment, an early filter is applied with a query plan using intra-pipeline predicate back-propagation. Specifically, the query plan may be thought of as a pipeline of operations. A runtime variable var may be introduced, and a specialized filter using var may be pushed down below the join operation. Var is a dynamic variable that is updated to track a value from the sort or similar operation (such as max (heap), reflecting the maximum value of a max-heap used by the sort or similar operation). The runtime variable gets initialized once the heap reaches a minimum number of elements (such as K in the case of a top K sort). Thus, before the heap reaches that minimum number of elements, the filter does not apply. Once the heap does reach that minimum number of elements, the filter does apply and acts to filter elements. Since the filter has been pushed down below the join operation, this saves processing cycles.

Adaptive early filtering with intra-pipeline predicate back-propagation

A database execution engine generates a first query execution plan in response to receiving a first query, where a thread limit is specified for worker threads launched by the database execution engine. A first main executor thread is launched to process the first query and a first plurality of tasks are created to be performed in response to the first query. Then, a first plurality of worker threads are launched to perform the first plurality of tasks, where the first plurality of worker threads is less than or equal to the thread limit. In response to parallelizing processing of the first query execution plan, the first main executor thread is restricted to a first period of execution time before entering a waiting phase. The first main executor thread is woken up after the first plurality of worker threads have completed the first plurality of tasks.

Reducing activity of the main SQL executor thread

In some implementations, there is provided a method including generating a query plan including in a first pipeline a first join operator and in a second pipeline a second join operator; executing at least a portion of the query plan including the first pipeline and the first join operator; detecting, based on the at least one operator usage state and the at least one operator pruning condition, an empty state object shared between the first join operator and the second join operator in the second pipeline; and processing, by the at least one operator pruning condition, an indication of the empty state object, wherein the least one operator pruning condition is associated with the second join operator and includes at least a first rule to mark the second join operator for pruning.

Dynamic operator pruning based on state dependencies and intermediate results

A method for performing a multi-fragment index scan operation included in a query pipeline of a query accessing a database table may include performing an index scan on a first chunk of a first fragment of the database table and a second chunk of a second fragment of the database table. A first output of the index scan on the first chunk of the first fragment of the database table and a second output of the index scan on the second chunk of the second fragment of the database table may be merged to generate a partial result set for ingestion by a next operation in the query pipeline. The multi-fragment index scan operation may continue with index scans on subsequent chunks of the first fragment and second fragment of the database table. Related systems and computer program products are also provided.

Multi-fragment index scan

Arrangements for a reading scheme for column-oriented databases are provided. A virtual column may be generated by applying a structured query language operation on distinct values in a column of data in a table. Based on applying the structured query language operation, the virtual column may be populated with corresponding precalculated values. The virtual column may be compressed with dictionary-based compression. The compressing may include generating a column of value identifiers, each of the value identifiers representing a distinct value in the column of data. A database query against the compressed virtual column may be received. A result of the database query may be returned by retrieving one or more of the precalculated values from the compressed virtual column.

Calculated dictionary column readers

A method, a system, and a computer program product for generating an executable query plan. A query requiring access to data stored in a database system is received. Based on the received query, a query execution plan having a plurality of query execution pipelines is generated. Each query execution pipeline in the plurality of query execution pipelines is configured to execute a plurality of operations in a predetermined order associated with each query execution pipeline. The generated query execution plan is fragmented into a plurality of fragments. Each fragment has one or more query execution pipelines in the plurality of query execution pipelines. Each fragment of the generated query execution plan is transmitted to one or more execution locations in a plurality of execution locations for execution. Each fragment of the generated query execution plan is executed at one or more execution locations. One or more results of executions are combined to generate a response to the received query.

Distributed query plan generation

In some implementations, there is provided a method that includes detecting in a query plan a pipeline that includes a last restart query operator that can cause a retry of the query plan during execution of the query plan; configuring the pipeline to execute the pipeline using an open call configuration, configuring at least one subsequent pipeline to execute in a fetch call configuration; executing the query plan including the pipeline in the open call configuration; sending, by the send operator, a message indicating the last restart query operator cannot cause a retry of the execution of the query plan; and causing execution of at least one operator in the subsequent pipeline to execute in the fetch call configuration, in which result streaming of partial results is allowed for the at least one operator.

Result streaming for distributed query processing

Methods, systems, and computer-readable storage media for a set of pipelines that enable selective skipping of dictionary scans and/or index lookups when processing queries in database management systems, such that dictionary scans and/or index lookups are only executed when needed, eliminating unnecessary execution of dictionary scans and/or index lookups to improve efficiency of the database system in terms of reducing load on technical resources and improve query performance by reducing latency in returning query results.

Selective skipping of dictionary scans and/or index lookups during query processing in database systems

A database management system (DBMS) traverses an operator topology of a query plan to identify pipelines within the query plan. The query plan may include at least a first shared subplan reference, where the first shared subplan reference refers to a first shared subplan, and where the first shared subplan is associated with a shared node in the operator topology. Also, the DBMS assigns, based on an order of execution, pipeline indices to the pipelines in the query plan. Additionally, the DBMS tracks pipeline dependencies of the query plan. For each shared subplan reference, the database management system specifies the shared subplan reference as a pipeline breaker if its pipeline has a dependency to a pipeline with an index greater than a corresponding shared subplan. Otherwise, the shared subplan reference is specified as a non-pipeline breaker. Then, the query plan is updated based on these shared subplan reference specifications.

Shared subplan pipelining inference

In some implementations, there is provided pipeline bypassing of certain operators. In some implementations, a method includes generating a query plan including at least one pipeline of operators; determining whether the at least one pipeline of operators includes a first operator that requires a complete result set as an input and further includes a second operator that supports providing the complete result set using a state identified by a state reference; and in response to determining the at least one pipeline of operators includes the first operator that requires the complete result set as the input and further includes the second operator that supports providing the complete result set using the state identified by the state reference, bypassing in the query plan pipelining between the first operator and the second operator.

Pipeline bypass for certain query plan operators

A system and/or method for spill-to-disk in projection operations includes receiving a query including a projection, receiving a plurality of rows in response to the query processed by a processing thread of a plurality of processing threads, determining whether the query specifies an order for the plurality of rows, determining whether a disk buffer associated with the processing thread contains a stored row in response to the query specifying the order, storing the plurality of rows in the disk buffer in response to determining the disk buffer contains the stored row, storing the plurality of rows in a memory buffer associated with the processing thread in response to determining the disk buffer does not contain the stored row and the memory buffer contains at least a threshold amount of memory to store the plurality of rows, and providing the stored plurality of rows in response to the query.

Spill-to-disk in projection operations

Systems, methods, and articles of manufacture, including computer program products, provide a system including at least one data processor and at least one memory storing instructions which, when executed by the at least one data processor, cause operations comprising: generating, by a database execution engine, a query plan including a plurality of operators; inserting, by the database execution engine, an enforce compilation operator into the query plan that includes the plurality of operators, the plurality of operators comprising a first operator, the enforce compilation operator, and a second operator; executing at least the first operator of the query plan; in response to executing the first operator, evaluating, by the database execution engine, an output of the first operator to determine whether a condition is satisfied; and in response to the condition being satisfied, triggering, by the database execution engine, a just-in-time compilation of the second operator.

JIT code compilation during runtime based on runtime statistics

In some implementations, there is provided a method that includes receiving a query request including a top k query operator for query plan generation, optimization, and execution, wherein k defines a threshold limit of query results for the top k query operator; inserting into a query plan a check operator associated with the top k query operator; in response to executing the query plan, checking, by the check operator, whether an early exit occurs due to the top k query operator reaching the threshold limit; in response to the early exit occurring due to the top k query operator reaching the threshold limit, stopping processing, by the check operator, including opening of another fragment of a database table; and in response to the early exit not occurring, allowing, by the check operator, the opening of the other fragment of the database table.

Custom table scan for top k queries

A method for performing a parallelized heapsort operation may include updating, by a first worker thread, a first buffer while a second worker thread updates a second buffer in parallel. The first worker thread may update the first buffer by adding, to the first buffer, elements from a first partition of a dataset. The second worker thread may update the second buffer by adding, to the second buffer, elements from a second partition of the dataset. Upon the first buffer reaching a threshold size, the first worker thread may acquire a lock for the first worker thread to update a heap based on the first buffer while the second worker thread is prevented from updating the heap based on the second buffer. A result of a top k query comprising a k quantity of smallest elements from the dataset may be generated based on the heap.

Growing community of inventors

Heidelberg, Germany

Paul Willems