Google Inc.

Tal Shaked

Tushar Deepak Chandra

Yoram Singer

Tze Way Eugene Ie

Joshua A Redstone

Kenneth Jerome Goldman

Jeremiah Harmsen

Yonggang Zhao

James McFadden

Heng-Tze Cheng

Andrew Stephen Tomkins

Gregory Sean Corrado

Hrishikesh Balkrishna Aradhye

Wei Chai

Glen Anderson

Saurabh Mathur

Mustafa Ispir

Alexandre Tachard Passos

Tomas Lloret Llinares

Rohan Anil

Xiaobing Liu

Mehmet Levent Koc

David Edgar Lluncor

Jerry Zhao

Shahar Jamshy

Kristen Riedt LeFevre

Jim McFadden

Danny Wyatt

Tze Way Eugene Le

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training machine learning systems. One of the methods includes receiving a plurality of training examples; and training a machine learning system on each of the plurality of training examples to determine trained values for weights of a machine learning model, wherein training the machine learning system comprises: assigning an initial value for a regularization penalty for a particular weight for a particular feature; and adjusting the initial value for the regularization penalty for the particular weight for the particular feature during the training of the machine learning system.

Regularization relaxation scheme

A system includes one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the computers to implement a combined machine learning model for processing an input including multiple features to generate a predicted output for the machine learning input. The combined model includes: a deep machine learning model configured to process the features to generate a deep model output; a wide machine learning model configured to process the features to generate a wide model output; and a combining layer configured to process the deep model output generated by the deep machine learning model and the wide model output generated by the wide machine learning model to generate the predicted output, in which the deep model and the wide model have been trained jointly on training data to generate the deep model output and the wide model output.

Wide and deep machine learning models

Systems and techniques are provided for template exploration in a large-scale machine learning system. A method may include obtaining multiple base templates, each base template comprising multiple features. A template performance score may be obtained for each base template and a first base template may be selected from among the multiple base templates based on the template performance score of the first base template. Multiple cross-templates may be constructed by generating a cross-template of the selected first base template and each of the multiple base templates. Performance of a machine learning model may be tested based on each cross-template to generate a cross-template performance score for each of the cross-templates. A first cross-template may be selected from among the multiple cross-templates based on the cross-template performance score of the cross-template. Accordingly, the first cross-template may be added to the machine learning model.

Using template exploration for large-scale machine learning

The present disclosure provides systems and techniques for efficient locking of datasets in a database when updates to a dataset may be delayed. A method may include accumulating a plurality of updates to a first set of one or more values associated with one or more features. The first set of one or more values may be stored within a first database column. Next, it may be determined that a first database column update aggregation rule is satisfied. A lock assigned to at least a portion of at least a first database column may be acquired. Accordingly, one or more values in the first set within the first database column may be updated based on the plurality of updates. In an implementation, the first set of one or more values may be associated with the first lock.

Efficient locking of large data collections

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a factorization model to learning features of model inputs of a trained model such that the factorization model is predictive of outcome for which the machine learned model is trained.

Factorized models

The present disclosure provides methods and systems for using variable length representations of machine learning statistics. A method may include storing an n-bit representation of a first statistic at a first n-bit storage cell. A first update to the first statistic may be received, and it may be determined that the first update causes a first loss of precision of the first statistic as stored in the first n-bit storage cell. Accordingly, an m-bit representation of the first statistic may be stored at a first m-bit storage cell based on the determination. The first m-bit storage cell may be associated with the first n-bit storage cell. As a result, upon receiving an instruction to use the first statistic in a calculation, a combination of the n-bit representation and the m-bit representation may be used to perform the calculation.

Using variable length representations for machine learning statistics

Methods and systems for replacing feature values of features in training data with integer values selected based on a ranking of the feature values. The methods and systems are suitable for preprocessing large-scale machine learning training data.

Using an integerized representation for large-scale machine learning data

Parallel processing of data may include a set of map processes and a set of reduce processes. Each map process may include at least one map thread. Map threads may access distinct input data blocks assigned to the map process, and may apply an application specific map operation to the input data blocks to produce key-value pairs. Each map process may include a multiblock combiner configured to apply a combining operation to values associated with common keys in the key-value pairs to produce combined values, and to output intermediate data including pairs of keys and combined values. Each reduce process may be configured to access the intermediate data output by the multiblock combiners. For each key, an application specific reduce operation may be applied to the combined values associated with the key to produce output data.

Parallel processing of data

Implementations of the disclosed subject matter provide methods and systems for using a multistage learner for efficiently boosting large datasets in a machine learning system. A method may include obtaining a first plurality of examples for a machine learning system and selecting a first point in time. Next, a second point in time occurring subsequent to the first point in time may be selected. The machine learning system may be trained using m of the first plurality of examples. Each of the m examples may include a feature initially occurring after the second point in time. In addition, the machine learning system may be trained using n of the first plurality of examples, and each of the n examples may include a feature initially occurring after the first point in time.

Multistage learner for efficiently boosting large datasets

Systems and techniques are disclosed for training a machine learning model based on one or more regularization penalties associated with one or more features. A template having a lower regularization penalty may be given preference over a template having a higher regularization penalty. A regularization penalty may be determined based on domain knowledge. A restrictive regularization penalty may be assigned to a template based on determining that a template occurrence is below a stability threshold and may be modified if the template occurrence meets or exceeds the stability threshold.

Template regularization for generalization of learning systems

Systems and techniques are disclosed for generating weighted machine learned models using multi-shard combiners. A learner in a machine learning system may receive labeled positive and negative examples and workers within the learner may be configured to receive either positive or negative examples. A positive and negative statistic may be calculated for a given feature and may either be applied separately in a model or may be combined to generate an overall statistic.

Growing community of inventors

Los Altos, CA, United States of America

Tal Shaked