Sap Se

Manuel Zeise

Isil Pekel

Steven Jaeger

Inputs may be received for constructing a data processing pipeline configured to implement an process to generate a machine learning model for performing a task associated with an input dataset. The process may include a plurality of machine learning trials, each of which applying, to a training dataset and/or a validation dataset generated based on the input dataset, a different type of machine learning model and/or a different set of trial parameters. The machine learning model being generated based on a result of the plurality of machine learning trials. A runtime estimate for the process to generate the machine learning model may be determined. The runtime estimate may enable the allocation of a sufficient time budget for the process. Moreover, the process may be executed if the runtime of the process does not exceed the available time budget.

Runtime estimation for machine learning data processing pipeline

Data for processing by a machine learning model may be prepared by encoding a first portion of the data including a spatial data. The spatial data may include a spatial coordinate including one or more values identifying a geographical location. The encoding of the first portion of the data may include mapping, to a cell in a grid system, the spatial coordinate such that the spatial coordinate is represented by an identifier of the cell instead of the one or more values. The data may be further prepared by embedding a second portion of the data including textual data, preparing a third portion of the data including hierarchical data, and/or preparing a fourth portion of the data including numerical data. The machine learning model may be applied to the prepared data in order to train, validate, test, and/or deploy the machine learning model to perform a cognitive task.

Preparing data for machine learning processing

A data processing pipeline may be generated to include an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset. The executor node may execute machine learning trials by applying, to the training dataset, a machine learning model and/or a different set of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, a machine learning model for performing a task. The execution of the data processing pipeline may be optimized. Examples of optimizations include pooling multiple machine learning trials for execution at a single executor node, executing at least some machine learning trials using a sub-sample of the training dataset, and adjusting a proportion of trial parameters sampled from a uniform distribution to avoid a premature convergence to a local minima within the hyper-parameter space for generating the machine learning model.

Optimizations for machine learning data processing pipeline

A data processing pipeline may be generated to include an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset. The executor node may execute machine learning trials by applying, to the training dataset, a machine learning model and/or a different set of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, a machine learning model for performing a task. Data associated with the execution of the data processing pipeline may be collected for storage in a tracking database. A report including de-normalized and enriched data from the tracking database may be generated. The hyper-parameter space of the machine learning model may be analyzed based on the report. A root cause of at least one fault associated with the execution of the data processing pipeline may be identified based on the analysis.

Hyper-parameter space optimization for machine learning data processing pipeline

A user interface may be generated to receive inputs for constructing a data processing pipeline that includes an orchestrator node, a preparator node, and an executor node. The preparator node may generate a training dataset and a validation dataset for a machine learning model. The executor node may execute machine learning trials by applying, to the training dataset and the validation dataset, machine learning models having different sets of trial parameters. The orchestrator node may identify, based on a result of the machine learning trials, an optimal machine learning model for performing a task. The data processing pipeline may be adapted dynamically based on the input dataset and/or computational resource budget. The optimal machine learning model for performing the task may be generated by executing, based on the graph, the data processing pipeline the orchestrator node, the preparator node, and the executor node.

Growing community of inventors

Karlsruhe, Germany

Manuel Zeise