Capital One Services, LLC

Christopher Larson

Oluwatobi Olabiyi

Erik Thomas Mueller

Tarek Aziz Lahlou

Zachary Kulis

Diana Mingels

Nathan Wolfe

Aspects discussed herein may relate to methods and techniques for embedding constrained and unconstrained optimization programs as layers in a neural network architecture. Systems are provided that implement a method of solving a particular optimization problem by a neural network architecture. Prior systems required use of external software to pre-solve optimization programs so that previously determined parameters could be used as fixed input in the neural network architecture. Aspects described herein may transform the structure of common optimization problems/programs into forms suitable for use in a neural network. This transformation may be invertible, allowing the system to learn the solution to the optimization program using gradient descent techniques via backpropagation of errors through the neural network architecture. Thus these optimization layers may be solved via operation of the neural network itself.

Embedding constrained and unconstrained optimization programs as neural network layers

Systems, apparatuses, and methods are described for inverting neural embeddings. One or more forward neural embeddings associated with meanings, features, and/or characteristics of data samples may be generated for one or more data samples. One or more inverse neural embeddings associated with the one or more forward neural embeddings may be determined. One or more inverse feature sets for the one or more inverse neural embeddings may be generated.

Methods for exact and approximate inverses of neural embedding models

Aspects described herein may allow for the application of stochastic gradient boosting techniques to the training of deep neural networks by disallowing gradient back propagation from examples that are correctly classified by the neural network model while still keeping correctly classified examples in the gradient averaging. Removing the gradient contribution from correctly classified examples may regularize the deep neural network and prevent the model from overfitting. Further aspects described herein may provide for scheduled boosting during the training of the deep neural network model conditioned on a mini-batch accuracy and/or a number of training iterations. The model training process may start un-boosted, using maximum likelihood objectives or another first loss function. Once a threshold mini-batch accuracy and/or number of iterations are reached, the model training process may begin using boosting by disallowing gradient back propagation from correctly classified examples while continue to average over all mini-batch examples.

Stochastic gradient boosting for deep neural networks

Aspects described herein may relate to the determination of data that is indicative of a greater range of speech properties than input text data. The determined data may be used as input to one or more speech processing tasks, such as model training, model validation, model testing, or classification. For example, after a model is trained based on the determined data, the model's performance may exhibit more resilience to a wider range of speech properties. The determined data may include one or more modified versions of the input text data. The one or more modified versions may be associated with the one or more speakers or accents and/or may be associated with one or more levels of semantic similarity in relation to the input text data. The one or more modified versions may be determined based on one or more machine learning algorithms.

Growing community of inventors

Washington, DC, United States of America

Christopher Larson