International Business Machines Corporation

Other

Thomas P Parnell

Charalampos Pozidis

Nikolaos Papandreou

Thomas Mittelholzer

Charles John Camp

Nikolas Ioannou

Timothy J Fisher

Roman Alexander Pletka

Sasa Tomic

Tobias Blaettler

Aaron Daniel Fry

Ioannis Koltsidas

Celestine Duenner

Milos Stanisavljevic

Andrew Dale Walls

Andreea Anghel

Gary A Tressler

Dimitrios Sarigiannis

Jan Van Lunteren

Michail Vlachos

Kubilay Atasu

Antonios Kornilios Kourtis

Vasileios Vasileiadis

Michael Kaufmann

Celestine Mendler-Duenner

Evangelos Stavros Eleftheriou

Robert Haas

Thomas D Weigold

Thomas J Griffin

Martin Petermann

Cesar Berrospi Ramis

Bryan Bordeaux Grandy

Malgorzata Lazuka

The invention is notably directed to a computer-implemented method, which aims at jointly identifying an optimal source of computerized resources and optimizing a configuration of the computerized resources. The method comprises configuring a Best-Arm Identification algorithm, in order to (i) associate arms of the algorithm with respective sources of computerized resources and (ii) connect the arms to one or more optimizers. Each of the optimizers is designed to optimize a configuration of such computerized resources. Next, the method iteratively executes the Best-Arm Identification algorithm to progressively eliminate the sources, with a view to eventually identifying one of the sources as an optimal source with an optimized configuration. Several iterations are accordingly performed. During each iteration, each of the arms is pulled and the rewards earned by pulling the arms are computed. Pulling each arm causes to optimize a configuration of computerized resources of a respectively associated source.

Configuration and optimization of a source of computerized resources

A method of dynamically optimizing decision tree inference is provided. The method, which is performed at the computerized system, repeatedly executes one or more decision trees for inference purposes and repeatedly performs an optimization procedure according to two-phase cycles. Each cycle includes two alternating phases, i.e., a first phase followed by a second phase. The decision trees are executed based on a reference data structure, whereby attributes of nodes of the decision trees are repeatedly accessed from the reference data structure during the first phase of each of the cycles. First, the accessed attributes are monitored during the first phase of each cycle, which leads to update statistical characteristics of the nodes. Second, a substitute data structure is configured during the second phase of each cycle based on the updated statistical characteristics. Third, the reference data structure is updated in accordance with the substitute data structure.

Dynamically optimizing decision tree inferences

According to one embodiment, a method, computer system, and computer program product for training a cognitive model that involves one or more decision trees as base learners is provided. The present invention may include constructing, by a tree building algorithm, the one or more decision trees, wherein the constructing further comprises associating one or more training examples with one or more leaf nodes of the one or more decision trees and iteratively running a breadth-first search tree builder on one or more of the decision trees to perform one or more tree building operations; and training the cognitive model based on the one or more decision trees.

Training decision tree-based predictive models

An approach of accelerating inferences based on decision trees based on accessing one or more decision trees, wherein each decision tree of the decision trees accessed comprises decision tree nodes, including nodes grouped into one or more supersets of nodes designed for joint execution. For each decision tree of the decision trees accessed, the nodes are executed to obtain an outcome for the one or more decision trees, respectively. For each superset of the one or more supersets of said each decision tree, the nodes of each superset are jointly executed by: loading attributes of the nodes of each superset in a respective cache line of the cache memory processing said attributes from the respective cache line until an inference result is returned based on the one or more outcomes.

Joint execution of decision tree nodes for accelerating inferences

Parallel training of a machine learning model on a computerized system may be provided. Computing tasks can be assigned to multiple workers of a system. A method may include accessing training data. A parallel training of the machine learning model can be started based on the accessed training data, so as for the training to be distributed through a first number K of workers, K>1. Responsive to detecting a change in a temporal evolution of a quantity indicative of a convergence rate of the parallel training (e.g., where said change reflects a deterioration of the convergence rate), the parallel training of the machine learning model is scaled-in, so as for the parallel training to be subsequently distributed through a second number K′ of workers, where K>K′≥1. Related computerized systems and computer program products may be provided.

Elastic training of machine learning models via re-partitioning based on feedback from the training algorithm

In an approach for constructing an ensemble model from a set of base learners, a processor performs a plurality of boosting iterations, where: at each boosting iteration of the plurality of boosting iterations, a base learner is selected at random from a set of base learners, according to a sampling probability distribution of the set of base learners, and trained according to a training dataset; and the sampling probability distribution is altered: (i) after selecting a first base learner at a first boosting iteration of the plurality of boosting iterations and (ii) prior to selecting a second base learner at a final boosting iteration of the plurality of boosting iterations. A processor constructs an ensemble model based on base learners selected and trained during the plurality of boosting iterations.

Constructing an ensemble model from randomly selected base learners

Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.

Parallel training of machine learning models

Embodiments of the present invention provide computer-implemented methods, computer program products and systems. Embodiments of the present invention can run preemptable tasks distributed according to a distributed environment, wherein each task of a plurality of preemptable tasks has been assigned two or more of the training data samples to process during each iteration. Embodiments of the present invention can, upon verifying that a preemption condition for each iteration is satisfied: preempt any task of the preemptable tasks that have started processing training data samples assigned to it, and update the cognitive model based on outputs obtained from completed tasks, including outputs obtained from both the preempted tasks and completed tasks that have finished processing all training data samples as assigned to it.

Straggler mitigation for iterative machine learning via task preemption

Methods are provided for implementing training of a machine learning model in a processing system, together with systems for performing such methods. A method includes providing a core module for effecting a generic optimization process in the processing system, and in response to a selective input, defining a set of derivative modules, for effecting computation of first and second derivatives of selected functions &#x192; and g in the processing system, to be used with the core module in the training operation. The method further comprises performing, in the processing system, the generic optimization process effected by the core module using derivative computations effected by the derivative modules.

Machine learning implementation in processing systems

Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system comprising a host computer operatively interconnected with an accelerator unit. The training includes a stochastic optimization process for optimizing a function of a training data matrix X, having data elements Xwith row coordinates i=1 to n and column coordinates j=1 to m, and a model vector w having elements w. For successive batches of the training data, defined by respective subsets of one of the row coordinates and column coordinates, random numbers associated with respective coordinates in a current batch b are generated in the host computer and sent to the accelerator unit. In parallel with generating the random numbers for batch b, batch b is copied from the host computer to the accelerator unit.

Machine learning in heterogeneous processing systems

A method for a machine learning model training is provided which operates in a mixed CPU/GPU environment. The amount of general processing unit memory is larger than the amount of special processing unit memory. The method includes loading a complete training data set into the memory of the general processing unit, determining importance values relating to training data vectors in the provided training data set of the training data vectors, dynamically transferring training data vectors of the training data set from the general processing unit memory to a special processing unit memory using as decision criteria the importance value of the training data vector, wherein the importance value used is taken from an earlier training round of the machine learning model, and executing a training algorithm on the special processing unit with the training data vectors having the highest available importance values of one of the earlier training rounds.

Memory-based data selection scheme for machine learning training on limited memory resources

Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system that includes a host computer operatively interconnected to an accelerator unit. The training operation involves an iterative optimization process for optimizing a model vector defining the model. Such a method includes, in the host computer, storing a matrix of training data and partitioning the matrix into a plurality of batches of data vectors. For each of successive iterations of the optimization process, a selected subset of the batches is provided to the accelerator unit. In the accelerator unit, each iteration of the optimization process is performed to update the model vector in dependence on vectors in the selected subset for that iteration. In the host computer, batch importance values are calculated for respective batches. The batch importance value is dependent on contributions of vectors in that batch to sub-optimality of the model vector.

Two sets Xand Xof histograms of words, and a vocabulary V are accessed. Each of the two sets is representable as a sparse matrix, each row of which corresponds to a histogram. Each histogram is representable as a sparse vector, whose dimension is determined by a dimension of the vocabulary. Two phases compute distances between pairs of histograms. The first phase includes computations performed for each histogram and for each word in the vocabulary to obtain a dense, floating-point vector y. The second phase includes computing, for each histogram, a sparse-matrix, dense-vector multiplication between a matrix-representation of the set Xof histograms and the vector y. The multiplication is performed to obtain distances between all histograms of the set Xand each histogram X[j]. Distances between all pairs of histograms are obtained, based on which distances between documents can subsequently be assessed.

Low-complexity methods for assessing distances between pairs of documents

A method for parallelizing a training of a model using a matrix-factorization-based collaborative filtering algorithm may be provided. The model can be used in a recommender system for a plurality of users and a plurality of items. The method includes providing a sparse training data matrix, selecting a number of user-item co-clusters, and building a user model data matrix by matrix factorization such that a computational load for executing the determining updated elements of the factorized sparse training data matrix is evenly distributed across the heterogeneous computing resources.

Load-balancing training of recommender system for heterogeneous systems

A technique for selecting context models (CMs) for a CM ensemble (CME) in a context mixing compressor includes measuring compression ratios (CRs) of the compressor on a dataset for each CM included in a base set of CMs. A first CM that has a maximum CR for the dataset is added to the CME. In response to a desired number of the CMs not being in the CME, subsequent CRs for the compressor are measured on the dataset for each of the CMs in the base set of CMs that are not in the CME in conjunction with one or more CMs in the CME. In response to a desired number of the CMs not being in the CME, subsequent CMs that in conjunction with the one or more CMs in the CME result in a maximum subsequent CR for the dataset are added to the CME.

Growing community of inventors

Zurich, Switzerland

Thomas P Parnell