Sap Se

Sap Ag

Boris Gruschko

Christian Mohr

Martin Strenge

Florian Foebel

Bjoern Friedmann

Franz Faerber

Meinolf Block

Sascha Zorn

Joachim Fitzer

Simon Seif

Gerrit Simon Kazmaier

Lars Volker

Nico Licht

Christian Lahmer

Torsten Abraham

Marcus Lieberenz

Christoph Weyerhaeuser

Steffen Geissinger

Helmut Prestel

Reiner Singer

Sebastian Seifert

Rolf Hammer

Kai Baumgarten

Sigrid K Wortmann

Philipp Grosse

David Kernert

Yu Su

Veit Spaegele

Thomas Elvers

Junhui Liu

Juan Ge

Daniel Johannsen

Jianfeng Yan

Christian Krause

Hendrik Radke

Anton Niadzelka

In one embodiment, the present invention includes a computer-implemented method comprising storing data in an application using an application custom data type and application custom data structure. The data is stored in a database using the application custom data type and the application custom data structure. In one embodiment, a request is sent to access the data from the application to the database. The data is retrieved from the database in response to the request in the application custom data type and the application custom data structure. In one embodiment, the data is sent from the database to a shared memory in the application custom data type and the application custom data structure and the data is retrieved by the application from the shared memory in the application custom data type and the application custom data structure.

Integrated application server and data server processes with matching data formats

A method for generating a data processing pipeline is provided. The method may include generating a user interface for displaying, at a client, a first operator node and a second operator node. The first operator node and the second operator node may each correspond to a data processing operation. In response to one or more inputs received from the client via the user interface, the first operator node and/or the second operator node may be added to a graph displayed in the user interface. The graph may be representative of a data processing pipeline. The first operator node and the second operator node may further be interconnected with an directed edge. The data processing pipeline may be generated based on the graph. Related systems and articles of manufacture, including computer program products, are also provided.

Data processing pipeline engine

A method for executing a data processing pipeline may be provided. The method may include identifying a file providing a runtime environment required for executing a series of data processing operations comprising the data processing pipeline. The file may be identified based on one or more tags associated with the data processing pipeline. The one or more tags may specify at least one runtime requirement for the series of data processing operations. The file may be executed to generate an executable package that includes a plurality of components required for executing the series of data processing operations. The series of data processing operations included in the data processing pipeline may be executed by at least executing the executable package to provide the runtime environment required for executing the series of data processing operations. Related systems and articles of manufacture, including computer program products, are also provided.

Containerized runtime environments

Distributed execution of data processing pipelines

In one respect, there is provided a distributed database system that includes a global scheduler node and a first local scheduler node. The global scheduler node can be configured to generate an execution plan for a query requiring data stored at and/or managed by a first and a second worker node assigned to the first local scheduler node. The execution plan can include a first fragment having operations to be performed by the first and second worker node. The first local scheduler node can be configured to coordinate the execution of a portion of the first fragment of the execution plan by distributing, to the first and second worker node, operations included in the first fragment of the execution plan, and routing, between the first and second worker node, messages associated with executing the first fragment of the execution plan. Related methods and computer program products are also provided.

Static hierarchy based query execution

A method may include receiving, from a client, a request to customize an operator node corresponding to a data processing operation. The request may include a first key. The operator node may be selected for inclusion in a graph representative of a data processing pipeline. The operator node may be associated with a first file that includes at least one configuration parameter associated with the operator node. The at least one configuration parameter may be associated with a second key. In response to the first key being determined to match the second key, the operator node may be customized by modifying the at least one configuration parameter. Furthermore, a second file associated with a customized operator node may be generated to store the customizations made to the operator node including the modification of the at least one configuration parameter. Related systems and articles of manufacture are also provided.

Customizing operator nodes for graphical representations of data processing pipelines

A method for routing trace messages during an execution of a data processing pipeline is provided. The method may include receiving, at a master node, a request from a client to receive trace messages output by a worker node executing at least a portion of the data processing pipeline. The master node and the worker node may be part of a distributed cluster of computing nodes. The trace messages may correspond to events occurring during the execution of at least the portion of the data processing pipeline. The master node may respond to the request by subscribing to a trace stream published by the worker node. The trace stream may include the trace messages output by the worker node. The master node may generate a user interface for displaying, at the client, the trace messages. Related systems and articles of manufacture, including computer program products, are also provided.

Trace messaging for distributed execution of data processing pipelines

The present disclosure involves systems, software, and computer implemented methods for preventing reader starvation in a distributed system. One example method includes receiving, at a node that manages a data stream, a write-intent request from a writer for a first partition, the data stream associated with a graph that includes levels of partitions including a current level; identifying the first partition for the writer; providing partition information for the first partition to enable the writer to write to the first partition; determining that the writer has previously written to a second partition associated with the current level; in response to determining that the writer has previously written to the second partition: creating a new current level; associating the first partition with the new current level; identifying partitions associated with the current level; and providing a close-level instruction to each local node that includes a partition associated with the current level.

Preventing reader starvation during order preserving data stream consumption

In one respect, there is provided a distributed database system that includes a first local scheduler node and a second local scheduler node. The first local scheduler node can be configured to coordinate a performance of a first task and a second task by a first worker node associated with the first local scheduler node. The second local scheduler node can be configured to respond to a defect at the first local scheduler node by at least: collecting, from the first worker node, a log indicating one or more tasks completed by the first worker node prior to the defect at the first local scheduler node; determining, based at least on the log, that the first worker node has completed the first task but not the second task; and coordinating a performance of the second task by the first worker node. Related methods and articles of manufacture are also provided.

Handling failovers at one or more nodes in a distributed database system

In one respect, there is provided a distributed database system. The distributed database system can include a plurality of nodes and a plurality of optimizers. A first optimizer of the plurality of optimizers can be configured to: receive, from a client, a command; generate, based at least in part on global metadata cached by the first optimizer, an execution plan for fulfilling the command that includes one or more tasks to be performed by at least a first node of the plurality of nodes; determine that at least one task of the one or more tasks are not successfully performed by the first node; and in response to determining that at least one task of the one or more tasks are not successfully performed by the first node, updating the cached global metadata at the first optimizer.

Metadata versioning in a distributed database

Data is received that is derived from a plurality of geo-spatial sensors that respectively generate data characterizing a plurality of sources within a zone of interest. The data includes series time-stamped frames for each of the sensors and at least one of the sources has two or more associated sensors. The received data can be sorted and processed, for each sensor on a sensor-by-sensor basis, using a sliding window. The sorted and processed data can then be correlated and written into a data storage application. Related apparatus, systems, techniques and articles are also described.

Pre-processing of geo-spatial sensor data

A system includes reception of a request to modify the data of a database, the request including first data, execution of processing to fulfill the request, determination, during execution of the processing, that a validation exit is associated with a current state of the processing, storage of the first data in a local temporary table in response to the determination, passage of the local temporary table to the validation exit, and execution of the validation exit to validate the first data based on the local temporary table and on the data of the database.

Validation of web-based database updates

In one aspect, a computer-implemented method includes calculating cardinality differences and combined sizes for one or more data block pairs in a relational database. The method also includes selecting a data block pair from the one or more data block pairs based on a calculated cardinality difference and a calculated combined size associated with the selected data block pair. The method further includes merging the selected data block pair to form a merged data block, and replacing the selected data block pair in the relational database by the merged data block.

Mechanism for constant-space indexing optimized data block merging

Growing community of inventors

Heidelberg, Germany

Boris Gruschko