International Business Machines Corporation

Celestine Duenner

Thomas P Parnell

Charalampos Pozidis

Michail Vlachos

Dimitrios Sarigiannis

Thomas Mittelholzer

Nikolas Ioannou

Kubilay Atasu

Vasileios Vasileiadis

Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.

Parallel training of machine learning models

Methods are provided for implementing training of a machine learning model in a processing system, together with systems for performing such methods. A method includes providing a core module for effecting a generic optimization process in the processing system, and in response to a selective input, defining a set of derivative modules, for effecting computation of first and second derivatives of selected functions &#x192; and g in the processing system, to be used with the core module in the training operation. The method further comprises performing, in the processing system, the generic optimization process effected by the core module using derivative computations effected by the derivative modules.

Machine learning implementation in processing systems

Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system comprising a host computer operatively interconnected with an accelerator unit. The training includes a stochastic optimization process for optimizing a function of a training data matrix X, having data elements Xwith row coordinates i=1 to n and column coordinates j=1 to m, and a model vector w having elements w. For successive batches of the training data, defined by respective subsets of one of the row coordinates and column coordinates, random numbers associated with respective coordinates in a current batch b are generated in the host computer and sent to the accelerator unit. In parallel with generating the random numbers for batch b, batch b is copied from the host computer to the accelerator unit.

Machine learning in heterogeneous processing systems

A method for a machine learning model training is provided which operates in a mixed CPU/GPU environment. The amount of general processing unit memory is larger than the amount of special processing unit memory. The method includes loading a complete training data set into the memory of the general processing unit, determining importance values relating to training data vectors in the provided training data set of the training data vectors, dynamically transferring training data vectors of the training data set from the general processing unit memory to a special processing unit memory using as decision criteria the importance value of the training data vector, wherein the importance value used is taken from an earlier training round of the machine learning model, and executing a training algorithm on the special processing unit with the training data vectors having the highest available importance values of one of the earlier training rounds.

Memory-based data selection scheme for machine learning training on limited memory resources

Computer-implemented methods are provided for implementing training of a machine learning model in a heterogeneous processing system that includes a host computer operatively interconnected to an accelerator unit. The training operation involves an iterative optimization process for optimizing a model vector defining the model. Such a method includes, in the host computer, storing a matrix of training data and partitioning the matrix into a plurality of batches of data vectors. For each of successive iterations of the optimization process, a selected subset of the batches is provided to the accelerator unit. In the accelerator unit, each iteration of the optimization process is performed to update the model vector in dependence on vectors in the selected subset for that iteration. In the host computer, batch importance values are calculated for respective batches. The batch importance value is dependent on contributions of vectors in that batch to sub-optimality of the model vector.

A method for parallelizing a training of a model using a matrix-factorization-based collaborative filtering algorithm may be provided. The model can be used in a recommender system for a plurality of users and a plurality of items. The method includes providing a sparse training data matrix, selecting a number of user-item co-clusters, and building a user model data matrix by matrix factorization such that a computational load for executing the determining updated elements of the factorized sparse training data matrix is evenly distributed across the heterogeneous computing resources.

Load-balancing training of recommender system for heterogeneous systems

Methods and apparatus are provided to determine entities and attributes dependencies for creating recommendations of items or entities using a highly scalable architecture. For example, a user may be recommended an item if a probability model of the method determines that the user relates to the item although the user has no contact to the item before the method is performed. The methods and apparatus provide a data structure representing a matrix having rows representing entities and columns representing attributes of the entities. Each entity of the entities of the data structure may include a user and each attribute of the attributes of the data structure may include an item. A cell of the matrix may be formed by a component pair including an entity and an attribute. In this manner, the methods and apparatus provide an efficient way for processing the probability model.

Growing community of inventors

Wettswil, Switzerland

Celestine Duenner