Google Inc.

Dragan Zivkovic

Francoise Beaufays

Peter Jin Hong

Françoise Beaufays

Rajiv Mathews

Zhongli Ding

Diego Melendo Casado

Yu Ouyang

Kurt Edward Partridge

Meltem Oktem

Andrew Hard

Mohammadinamul Hasan Sheik

Ouais Alsharif

Peter Ciccotto

Hidetoshi Tajima

Yoshitake Matsumoto

Khe Chai Sim

Akash Agrawal

Tamar Lucassen

Implementations disclosed herein are directed to systems and methods for evaluating on-device machine learning (ML) model(s) based on performance measure(s) of client device(s) and/or the on-device ML model(s). The client device(s) can include on-device memory that stores the on-device ML model(s) and a plurality of testing instances for the on-device ML model(s). When certain condition(s) are satisfied, the client device(s) can process, using the on-device ML model(s), the plurality of testing instances to generate the performance measure(s). The performance measure(s) can include, for example, latency measure(s), memory consumption measure(s), CPU usage measure(s), ML model measure(s) (e.g., precision and/or recall), and/or other measures. In some implementations, the on-device ML model(s) can be activated (or kept active) for use locally at the client device(s) based on the performance measure(s). In other implementations, the on-device ML model(s) can be sparsified based on the performance measure(s).

Evaluating on-device machine learning model(s) based on performance measures of client device(s) and/or the on-device machine learning model(s)

Processor(s) of a client device can: receive sensor data that captures environmental attributes of an environment of the client device; process the sensor data using a machine learning model to generate a predicted output that dictates whether one or more currently dormant automated assistant functions are activated; making a decision as to whether to trigger the one or more currently dormant automated assistant functions; subsequent to making the decision, determining that the decision was incorrect; and in response to determining that the determination was incorrect, generating a gradient based on comparing the predicted output to ground truth output. In some implementations, the generated gradient is used, by processor(s) of the client device, to update weights of the on-device speech recognition model. In some implementations, the generated gradient is additionally or alternatively transmitted to a remote system for use in remote updating of global weights of a global speech recognition model.

Using corrections, of automated assistant functions, for training of on-device machine learning models

On-device processor(s) of a client device may store, in on-device storage and in association with a time to live (TTL) in the on-device storage, a correction directed to ASR processing of audio data. The correction may include a portion of a given speech hypothesis that was modified to an alternate speech hypothesis. Further, the on-device processor(s) may cause an on-device ASR model to be personalized based on the correction. Moreover, and based on additional ASR processing of additional audio data, the on-device processor(s) may store, in the on-device storage and in association with an additional TTL in the on-device storage, a pseudo-correction directed to the additional ASR processing. Accordingly, the on-device processor(s) may cause the on-device ASR model to be personalized based on the pseudo-correction to prevent forgetting by the on-device ASR model.

Generation and utilization of pseudo-correction(s) to prevent forgetting of personalized on-device automatic speech recognition (ASR) model(s)

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for cross input modality learning in a mobile device are disclosed. In one aspect, a method includes activating a first modality user input mode in which user inputs by way of a first modality are recognized using a first modality recognizer; and receiving a user input by way of the first modality. The method includes, obtaining, as a result of the first modality recognizer recognizing the user input, a transcription that includes a particular term; and generating an input context data structure that references at least the particular term. The method further includes, transmitting, by the first modality recognizer, the input context data structure to a second modality recognizer for use in updating a second modality recognition model associated with the second modality recognizer.

Modality learning on mobile devices

A keyboard is described that determines, using a first decoder and based on a selection of keys of a graphical keyboard, text. Responsive to determining that a characteristic of the text satisfies a threshold, a model of the keyboard identifies the target language of the text, and determines whether the target language is different than a language associated with the first decoder. If the target language of the text is not different than the language associated with the first decoder, the keyboard outputs, for display, an indication of first candidate words determined by the first decoder from the text. If the target language of the text is different: the keyboard enables a second decoder, where a language associated with the second decoder matches the target language of the text, and outputs, for display, an indication of second candidate words determined by the second decoder from the text.

Keyboard automatic language identification and reconfiguration

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting forked auto-completions. In one aspect, a method includes receiving characters from a user device, obtaining an auto-completion that corresponds to the received characters, obtaining corpora and respective corpus scores associated with the auto-completion, selecting corpora based on the corpus scores, and providing the user device with data identifying the auto-completion and the selected corpora.

Presenting forked auto-completions

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing asynchronous and synchronous links to resources. According to one example implementation, a method includes receiving a request for a resource, identifying resources to be referenced by the requested resource, and identifying one or more of the referenced resources that are associated with client-side click tracking, and one or more of the referenced resources that are associated with server-side click tracking. The method also includes providing the requested resource. The provided resource includes one or more client-side click tracking links to the referenced resources that are associated with client-side click tracking, and one or more server-side click tracking links to the referenced resources that are associated with server-side click tracking.

Asynchronous and synchronous resource links

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing client-side click tracking. In one aspect, a method includes receiving, by a client, a resource from one or more servers. The resource includes (i) a link, (ii) click tracking code, and (iii) local storage monitoring code. The method includes invoking, by the client, the click tracking code and the local storage monitoring code, detecting, by the invoked click tracking code, a selection of the link, and generating, by the invoked click tracking code, a representation of the selection. The method includes storing, by the invoked click tracking code, the representation of the selection in a local storage on the client, determining, by the invoked local storage monitoring code, to communicate the representation of the selection; and communicating, by the invoked local storage monitoring code, the representation of the selection to the one or more servers.

Client-side click tracking

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting search results. In one aspect, a method includes displaying a search engine results page in a first presentation mode or a second presentation mode, wherein, in the first presentation mode and the second presentation mode, the search engine results page includes (i) a text region that displays textual information relating to one or more search results that are responsive to a search query and (ii) an image region, wherein, in the first presentation mode, the image region displays contextual information relating to two or more of the search results that are responsive to the search query, and wherein, in the second presentation mode, the image region displays contextual information relating to a single search result that is responsive to the search query; and receiving a user input, then switching the search engine results page between the first presentation mode and the second presentation mode.

Growing community of inventors

Sunnyvale, CA, United States of America

Dragan Zivkovic