Sap Se

Isil Pekel

Gerrit Simon Kazmaier

Hinnerk Gildhoff

Lars Volker

Edward-Robert Tyercha

Tim Grouisborn

Manuel Zeise

Steven Jaeger

Christoph Weyerhaeuser

Tobias Mindnich

Johannes Merx

Julian Schwing

Daniel Patejdl

Alena Bakulina

Inputs may be received for constructing a data processing pipeline configured to implement an process to generate a machine learning model for performing a task associated with an input dataset. The process may include a plurality of machine learning trials, each of which applying, to a training dataset and/or a validation dataset generated based on the input dataset, a different type of machine learning model and/or a different set of trial parameters. The machine learning model being generated based on a result of the plurality of machine learning trials. A runtime estimate for the process to generate the machine learning model may be determined. The runtime estimate may enable the allocation of a sufficient time budget for the process. Moreover, the process may be executed if the runtime of the process does not exceed the available time budget.

Runtime estimation for machine learning data processing pipeline

Data for processing by a machine learning model may be prepared by encoding a first portion of the data including a spatial data. The spatial data may include a spatial coordinate including one or more values identifying a geographical location. The encoding of the first portion of the data may include mapping, to a cell in a grid system, the spatial coordinate such that the spatial coordinate is represented by an identifier of the cell instead of the one or more values. The data may be further prepared by embedding a second portion of the data including textual data, preparing a third portion of the data including hierarchical data, and/or preparing a fourth portion of the data including numerical data. The machine learning model may be applied to the prepared data in order to train, validate, test, and/or deploy the machine learning model to perform a cognitive task.

Preparing data for machine learning processing

A data processing pipeline may be generated to include an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset. The executor node may execute machine learning trials by applying, to the training dataset, a machine learning model and/or a different set of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, a machine learning model for performing a task. The execution of the data processing pipeline may be optimized. Examples of optimizations include pooling multiple machine learning trials for execution at a single executor node, executing at least some machine learning trials using a sub-sample of the training dataset, and adjusting a proportion of trial parameters sampled from a uniform distribution to avoid a premature convergence to a local minima within the hyper-parameter space for generating the machine learning model.

Optimizations for machine learning data processing pipeline

A data processing pipeline may be generated to include an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset. The executor node may execute machine learning trials by applying, to the training dataset, a machine learning model and/or a different set of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, a machine learning model for performing a task. Data associated with the execution of the data processing pipeline may be collected for storage in a tracking database. A report including de-normalized and enriched data from the tracking database may be generated. The hyper-parameter space of the machine learning model may be analyzed based on the report. A root cause of at least one fault associated with the execution of the data processing pipeline may be identified based on the analysis.

Hyper-parameter space optimization for machine learning data processing pipeline

A user interface may be generated to receive inputs for constructing a data processing pipeline that includes an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset and a validation dataset for a machine learning model. The executor node may execute machine learning trials by applying, to the training dataset and the validation dataset, machine learning models having different sets of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, an optimal machine learning model for performing a task. The data processing pipeline may be adapted dynamically based on the input dataset and/or computational resource budget. The optimal machine learning model for performing the task may be generated by executing, based on the graph, the data processing pipeline the orchestrator node, the preparator node, and the executor node.

Machine learning data processing pipeline

A query of spatial data is received by a database comprising a columnar data store storing data in a column-oriented structure. Thereafter, a minimal bounding rectangle associated with the query is identified using a grid order scanning technique. The spatial data set corresponding to the received query is then mapped to physical storage in the database using the identified minimal bounding rectangle so that the spatial data set can be retrieved. Related apparatus, systems, techniques and articles are also described.

Querying spatial data in column stores using grid-order scans

Methods and apparatus, including computer program products, are provided for union node pruning. In one aspect, there is provided a method, which may include receiving, by a calculation engine, a query; processing a calculation scenario including a union node; accessing a pruning table associated with the union node, wherein the pruning table includes semantic information describing the first input from the first data source node and the second input from the second data source node; determining whether the first data source node and the second data source node can be pruned by at least comparing the semantic information to at least one filter of the query; and pruning, based on a result of the determining, at least one the first data source node or the second data source node. Related apparatus, systems, methods, and articles are also described.

Data-driven union pruning in a database semantic layer

A query is received by a database server from a remote application server that is associated with a calculation scenario that defines a data flow model including one or more calculation nodes including stacked multiproviders. Subsequently, the database server instantiates the calculation scenario and afterwards optimizes the calculation scenario. As part of the optimization, the calculation scenario is optimized by merging the two multiproviders. Thereafter, the operations defined by the calculation nodes of the optimized calculation scenario can be executed to result in a responsive data set. Next, the data set is provided to the application server by the database server.

Database calculation engine with nested multiprovider merging

DBSCAN clustering analyses can be improved by pre-processing of a data set using a Hilbert curve to intelligently identify the centers for initial partitional analysis by a partitional clustering algorithm such as CLARANS. Partitions output by the partitional clustering algorithm can be process by DBSCAN running in parallel before intermediate cluster results are merged.

Hilbert curve partitioning for parallelization of DBSCAN

A database server receives a query from a remote application server that is associated with a calculation scenario. The calculation scenario defines a data flow model that includes one or more calculation nodes that each define one or more operations to execute on the database server. A top operator node of the calculation nodes specifies a plurality of attributes and the query requests a subset of the attributes specified by the top operator node; Thereafter, the database server instantiates the calculation scenario so that it is optimized by requesting only the subset of attributes. The database server then executes the operations defined by the calculation nodes of the optimized calculation scenario to result in a responsive data set. The database server then provides the data set to the application server.

Database calculation engine with dynamic top operator

A query of spatial data is received by a database comprising a columnar data store storing data in a column-oriented structure. Thereafter, a minimal bounding rectangle associated with the query is identified using a tree-order scanning technique. A spatial data set that corresponds to the received query is then mapped to the physical storage in the database using the identified minimal bounding rectangle. Next, the spatial data set is then retrieved. Related apparatus, systems, techniques and articles are also described.

Querying spatial data in column stores using tree-order scans

Cluster analysis of data points in a data set can be optimized by identification of a preferred cluster analysis method. This identification can be based on indexing the data using a Hilbert curve and determining whether the data points are predominantly in spherical or non-spherical clusters. Methods, systems, and articles of manufacture are described.

Growing community of inventors

Heidelberg, Germany

Isil Pekel