Sas Institute Inc.

Yan Xu

Joshua David Griffin

Lawrence Edmund Lewis

Scott Russell Pope

Xindian Long

Liping Cai

Steven E Krueger

Xingqi Du

Seyedalireza Yektamaram

Patrick Nathan Koch

Steven Joseph Gardner

Jun Liu

Brett Alan Wujek

Yan Gao

Oleg Borisovich Golovidov

Riadh Omheni

Rui Shi

Jorge Manuel Gomes Da Silva

Xiangqian Hu

Ruiwen Zhang

Manoj Keshavmurthi Chari

Xuejun Liao

Wenwen Zhou

Philipp M Christophel

Yu-Min Lin

Shunping Huang

Guixian Lin

Ben-hao Wang

Amar K Narisetty

Kushawah Yogender Singh

Kiran Devidas Bhole

Girish Vinayak Kolapkar

Aishwarya Sharma

Amod Anil Ankulkar

A system, method, and computer-program product includes selecting, by a controller node, a plurality of candidate hyperparameter search points from a hyperparameter search space; instructing, by the controller node, one or more worker nodes to concurrently train a plurality of machine learning models for a target number of epochs using the plurality of candidate hyperparameter search points; identifying, by the controller node, a collection of intermediate candidate hyperparameter search points that outperform a remainder of the plurality of candidate hyperparameter search points by evaluating a performance of the plurality of machine learning models after training for the target number of epochs; and performing, by the controller node, a crossover operation with the collection of intermediate candidate hyperparameter search points to identify one or more new candidate hyperparameter search points to test in the hyperparameter search space.

Systems and methods for parallel exploration of a hyperparameter search space

A system, method, and computer-program product includes computing, by a controller node, a hyperparameter search space for a plurality of hyperparameters; selecting, by the controller node, a plurality of hyperparameter search points from the hyperparameter search space; instructing, by the controller node, one or more worker nodes to concurrently train a plurality of machine learning models based on the plurality of hyperparameter search points; receiving, from the one or more worker nodes, a plurality of performance metrics that measure a performance of the plurality of machine learning models; determining, by the controller node, one or more sets of optimal hyperparameter values based on the plurality of performance metrics; and outputting, by the controller node, the one or more sets of optimal hyperparameter values.

A system, method, and computer-program product includes receiving, by a worker process, a plurality of chunks of data from a client process; deriving, by the worker process, an input pattern for feeding the plurality of chunks of data to a machine learning model; caching, by the worker process, a subset of data elements of the plurality of chunks of data specified by the input pattern based on a data caching policy; and training the machine learning model by feeding the subset of data elements cached by the worker process and a remainder of data elements in the plurality of chunks of data when requested by the input pattern.

Systems and methods for multi-language training of machine learning models

A system, method, and computer-program product includes receiving, by a controller node, a request to execute a client process associated with a first programming language and a plurality of threads; launching, by the controller node, a plurality of multi-language worker processes based on a number of threads associated with the client process; and instructing, by the controller node, the plurality of multi-language worker processes to execute the plurality of threads associated with the client process.

A system, method, and computer-program product includes selecting, by a controller node, a plurality of hyperparameter search points from a hyperparameter search space; instructing, by the controller node, one or more worker nodes to concurrently train a plurality of machine learning models for a target number of epochs using the plurality of hyperparameter search points; receiving, from the one or more worker nodes, a plurality of performance metrics that measure a performance of the plurality of machine learning models during the target number of epochs; and removing, by the controller node, one or more underperforming hyperparameter search points from the plurality of hyperparameter search points according to a pre-defined performance metric ranking criterion associated with the plurality of performance metrics.

A computer determines a solution to a nonlinear optimization problem. A conjugate gradient (CG) iteration is performed with a first order derivative vector and a second order derivative matrix to update a CG residual vector, an H-conjugate vector, and a residual weight vector. A CG solution vector is updated using a previous CG solution vector, the H-conjugate vector, and the residual weight vector. An eigenvector of the second order derivative matrix having a smallest eigenvalue is computed. A basis matrix is defined that includes a cubic regularization (CR) solution vector, a CR residual vector, the CG solution vector, the CG residual vector, and the eigenvector. A CR iteration is performed to update the CR solution vector. The CR residual vector is updated using the first order derivative vector, the second order derivative matrix, and the updated CR solution vector. The process is repeated until a stop criterion is satisfied.

Cubic regularization optimizer

A computing device determines an optimal number of threads for a computer task. Execution of a computing task is controlled in a computing environment based on each task configuration included in a plurality of task configurations to determine an execution runtime value for each task configuration. An optimal number of threads value is determined for each set of task configurations having common values for a task parameter value, a dataset indicator, and a hardware indicator. The optimal number of threads value is an extremum value of an execution parameter value as a function of a number of threads value. A dataset parameter value is determined for a dataset. A hardware parameter value is determined as a characteristic of each distinct executing computing device in the computing environment. The optimal number of threads value for each set of task configurations is stored in a performance dataset in association with the common values.

Optimal number of threads determination system

A computing device determines a recommendation. (A) A first parameter matrix is updated using a first direction matrix and a first step-size parameter value that is greater than one. The first parameter matrix includes a row dimension equal to a number of users of a plurality of users included in a ratings matrix and the ratings matrix includes a missing matrix value. (B) A second parameter matrix is updated using a second direction matrix and a second step-size parameter value that is greater than one. The second parameter matrix includes a column dimension equal to a number of items of a plurality of items included in the ratings matrix. (C) An objective function value is updated based on the first parameter matrix and the second parameter matrix. (D) (A) through (C) are repeated until the first parameter matrix and the second parameter matrix satisfy a convergence test.

Recommendation system

Tuned hyperparameter values are determined for training a machine learning model. When a selected hyperparameter configuration does not satisfy a linear constraint, if a projection of the selected hyperparameter configuration is included in a first cache that stores previously computed projections is determined. When the projection is included in the first cache, the projection is extracted from the first cache using the selected hyperparameter configuration, and the selected hyperparameter configuration is replaced with the extracted projection in the plurality of hyperparameter configurations. When the projection is not included in the first cache, a projection computation for the selected hyperparameter configuration is assigned to a session. A computed projection is received from the session for the selected hyperparameter configuration. The computed projection and the selected hyperparameter configuration are stored to the first cache, and the selected hyperparameter configuration is replaced with the computed projection.

Multi-objective distributed hyperparameter tuning system

A computer solves a nonlinear optimization problem. An optimality check is performed for a current solution to an objective function that is a nonlinear equation with constraint functions on decision variables. When the performed optimality check indicates that the current solution is not an optimal solution, a barrier parameter value is updated, and a Lagrange multiplier value is updated for each constraint function based on a result of a complementarity slackness test. The current solution to the objective function is updated using a search direction vector determined by solving a primal-dual linear system that includes a dual variable for each constraint function and a step length value determined for each decision variable and for each dual variable. The operations are repeated until the optimality check indicates that the current solution is the optimal solution or a predefined number of iterations has been performed.

Nonlinear optimization system

A computing device selects decision variable values. A lower boundary value and an upper boundary value is defined for a decision variable. (A) A plurality of decision variable configurations is determined using a search method. The value for the decision variable is between the lower boundary value and the upper boundary value. (B) A decision variable configuration is selected. (C) A model of the model type is trained using the decision variable configuration. (D) The model is scored to compute an objective function value. (E) The computed objective function value and the selected decision variable configuration are stored. (F) (B) through (E) is repeated for a plurality of decision variable configurations. (G) The lower boundary value and the upper boundary value are updated using the objective function value and the decision variable configuration stored. Repeat (A)-(F) with the lower boundary value and the upper boundary value updated in (G).

Distributed decision variable tuning system for machine learning

A computing device compresses a gradient boosting tree predictive model. A gradient boosting tree predictive model is trained using a plurality of observation vectors. Each observation vector includes an explanatory variable value of an explanatory variable and a response variable value for a response variable. The gradient boosting tree predictive type model is trained to predict the response variable value of each observation vector based on a respective explanatory variable value of each observation vector. The trained gradient boosting tree predictive model is compressed using a compression model with a predefined penalty constant value and with a predefined array of coefficients to reduce a number of trees of the trained gradient boosting tree predictive model. The compression model minimizes a sparsity norm loss function. The compressed, trained gradient boosting tree predictive model is output for predicting a new response variable value from a new observation vector.

Analytic system for gradient boosting tree compression

Machine-learning models (MLM) can be configured more rapidly using some examples described herein. For example, a MLM can be configured by executing an iterative process, where each iteration includes a series of operations. The series of operations can include determining a current weight value for the current iteration, determining a current gradient direction for the current iteration based on the current weight value, and determining a current learning rate for the current iteration based on the current gradient direction. The operations can also include determining a current multistage momentum value for the current iteration. A next weight value for a next iteration can then be determined based on (i) the current weight value, (ii) the current gradient direction, (iii) the current learning rate, and (iv) the current multistage momentum value. The next weight value may also be determined based on a predefined learning rate that was preset, in some examples.

Accelerating configuration of machine-learning models

A computer trains a neural network model. (B) A neural network is executed to compute a post-iteration gradient vector and a current iteration weight vector. (C) A search direction vector is computed using a Hessian approximation matrix and the post-iteration gradient vector. (D) A step size value is initialized. (E) An objective function value is computed that indicates an error measure of the executed neural network. (F) When the computed objective function value is greater than an upper bound value, the step size value is updated using a predefined backtracking factor value. The upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values. (G) (E) and (F) are repeated until the computed objective function value is not greater than the upper bound value. (H) An updated weight vector is computed to describe a trained neural network model.

Deep learning model training system

A computing device automatically selects hyperparameter values based on objective criteria to train a predictive model. Each session of a plurality of sessions executes training and scoring of a model type using an input dataset in parallel with other sessions of the plurality of sessions. Unique hyperparameter configurations are determined using a search method and assigned to each session. For each session of the plurality of sessions, training of a model of the model type is requested using a training dataset and the assigned hyperparameter configuration, scoring of the trained model using a validation dataset and the assigned hyperparameter configuration is requested to compute an objective function value, and the received objective function value and the assigned hyperparameter configuration are stored. A best hyperparameter configuration is identified based on an extreme value of the stored objective function values.

Growing community of inventors

Cary, NC, United States of America

Yan Xu