Sas Institute Inc.

Seyedalireza Yektamaram

Joshua David Griffin

Yan Xu

Wenwen Zhou

Jorge Manuel Gomes Da Silva

Yan Gao

Riadh Omheni

Rui Shi

Yu-Min Lin

Jarad Forristal

Ben-hao Wang

Kushawah Yogender Singh

Kiran Devidas Bhole

Girish Vinayak Kolapkar

Aishwarya Sharma

Majid Jahani

Amod Anil Ankulkar

A computer determines a solution to a nonlinear optimization problem. A conjugate gradient (CG) iteration is performed with a first order derivative vector and a second order derivative matrix to update a CG residual vector, an H-conjugate vector, and a residual weight vector. A CG solution vector is updated using a previous CG solution vector, the H-conjugate vector, and the residual weight vector. An eigenvector of the second order derivative matrix having a smallest eigenvalue is computed. A basis matrix is defined that includes a cubic regularization (CR) solution vector, a CR residual vector, the CG solution vector, the CG residual vector, and the eigenvector. A CR iteration is performed to update the CR solution vector. The CR residual vector is updated using the first order derivative vector, the second order derivative matrix, and the updated CR solution vector. The process is repeated until a stop criterion is satisfied.

Cubic regularization optimizer

A computer trains a neural network. A neural network is executed with a weight vector to compute a gradient vector using a batch of observation vectors. Eigenvalues are computed from a Hessian approximation matrix, a regularization parameter value is computed using the gradient vector, the eigenvalues, and a step-size value, a search direction vector is computed using the eigenvalues, the gradient vector, the Hessian approximation matrix, and the regularization parameter value, a reduction ratio value is computed, an updated weight vector is computed from the weight vector, a learning rate value, and the search direction vector or the gradient vector based on the computed reduction ratio value, and an updated Hessian approximation matrix is computed from the Hessian approximation matrix, the predefined learning rate value, and the search direction vector or the gradient vector based on the reduction ratio value. The step-size value is updated using the search direction vector.

Deep learning model training system

A computing device determines an optimal number of threads for a computer task. Execution of a computing task is controlled in a computing environment based on each task configuration included in a plurality of task configurations to determine an execution runtime value for each task configuration. An optimal number of threads value is determined for each set of task configurations having common values for a task parameter value, a dataset indicator, and a hardware indicator. The optimal number of threads value is an extremum value of an execution parameter value as a function of a number of threads value. A dataset parameter value is determined for a dataset. A hardware parameter value is determined as a characteristic of each distinct executing computing device in the computing environment. The optimal number of threads value for each set of task configurations is stored in a performance dataset in association with the common values.

Optimal number of threads determination system

A computer trains a neural network model. (A) Observation vectors are randomly selected from a plurality of observation vectors. (B) A forward and backward propagation of a neural network is executed to compute a gradient vector and a weight vector. (C) A search direction vector is computed. (D) A step size value is computed. (E) An updated weight vector is computed. (F) Based on a predefined progress check frequency value, second observation vectors are randomly selected, a progress check objective function value is computed given the weight vector, the step size value, the search direction vector, and the second observation vectors, and based on an accuracy test, the mini-batch size value is updated. (G) (A) to (F) are repeated until a convergence parameter value indicates training of the neural network is complete. The weight vector for a next iteration is the computed updated weight vector.

Machine-learning models (MLM) can be configured more rapidly using some examples described herein. For example, a MLM can be configured by executing an iterative process, where each iteration includes a series of operations. The series of operations can include determining a current weight value for the current iteration, determining a current gradient direction for the current iteration based on the current weight value, and determining a current learning rate for the current iteration based on the current gradient direction. The operations can also include determining a current multistage momentum value for the current iteration. A next weight value for a next iteration can then be determined based on (i) the current weight value, (ii) the current gradient direction, (iii) the current learning rate, and (iv) the current multistage momentum value. The next weight value may also be determined based on a predefined learning rate that was preset, in some examples.

Accelerating configuration of machine-learning models

A computer trains a neural network model. (B) A neural network is executed to compute a post-iteration gradient vector and a current iteration weight vector. (C) A search direction vector is computed using a Hessian approximation matrix and the post-iteration gradient vector. (D) A step size value is initialized. (E) An objective function value is computed that indicates an error measure of the executed neural network. (F) When the computed objective function value is greater than an upper bound value, the step size value is updated using a predefined backtracking factor value. The upper bound value is computed as a sliding average of a predefined upper bound updating interval value number of previous upper bound values. (G) (E) and (F) are repeated until the computed objective function value is not greater than the upper bound value. (H) An updated weight vector is computed to describe a trained neural network model.

Growing community of inventors

Amsterdam, Netherlands

Seyedalireza Yektamaram