Google Inc.

X Development LLC

Meltem Oktem

Diego Melendo Casado

Yu Ouyang

Francoise Beaufays

Dragan Zivkovic

Mohammadinamul Hasan Sheik

Ignacio Lopez Moreno

Alexander Houston Gruenstein

Pu-sen Chao

Taral Pradeep Joglekar

Salil Rajadhyaksha

Fnu Heryandi

Xiyang Luo

Roza Chojnacka

Max Benjamin Braun

Jamieson Robert Kerns

Rajan Patel

Archana Kannan

Nada Elassal

Uday Idnani

Pu-Sen Chao

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for cross input modality learning in a mobile device are disclosed. In one aspect, a method includes activating a first modality user input mode in which user inputs by way of a first modality are recognized using a first modality recognizer; and receiving a user input by way of the first modality. The method includes, obtaining, as a result of the first modality recognizer recognizing the user input, a transcription that includes a particular term; and generating an input context data structure that references at least the particular term. The method further includes, transmitting, by the first modality recognizer, the input context data structure to a second modality recognizer for use in updating a second modality recognition model associated with the second modality recognizer.

Modality learning on mobile devices

In some implementations, processor(s) can receive an utterance from a speaker, and determine whether the speaker is a known user of a user device or not a known user of the user device. The user device can be shared by a plurality of known users. Further, the processor(s) can determine whether the utterance corresponds to a personal request or non-personal request. Moreover, and in response to determining that the speaker is not a known user of the user device and in response to determining that the utterance corresponds to a non-personal request, the processor(s) can cause a response to the utterance to be provided for presentation to the speaker at the user device response to the utterance, or can cause an action to be performed by the user device responsive to the utterance.

Multi-user authentication on a device

A method can perform a process with a method including capturing an image, determining an environment that a user is operating a computing device, detecting a hand gesture based on an object in the image, determining, using a machine learned model, an intent of a user based on the hand gesture and the environment, and executing a task based at least on the determined intent.

Intent detection with a computing device

In some implementations, authentication tokens corresponding to known users of a device are stored on the device. An utterance from a speaker is received. The speaker of the utterance is classified as not a known user of the device. A query that includes the authentication tokens that correspond to known users of the device, a representation of the utterance, and an indication that the speaker was classified as not a known user of the device is provided to the server. A response to the query is received at the device and from the server based on the query.

A system and method may provide for spatial and semantic auto-completion of an augmented or mixed reality environment. The system may detect physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device. The system may detect spaces in the physical environment that are occupied by the detected physical objects, and may detect spaces that are unoccupied in the physical environment. Based on the identification of the detected physical objects, the system may gain a semantic understanding of the physical environment, and may determine suggested objects for placement in the physical environment based on the semantic understanding. The system may place virtual representations of the suggested objects in a mixed reality scene of the physical environment for user consideration.

Spatial and semantic augmented reality autocompletion in an augmented reality environment

Methods, systems, and apparatus, including computer-readable storage devices, for managing voice interface devices. In some implementations, messages are received from a plurality of devices, each of the messages indicating a respective voice input detected by the device that sent the message. Audio signatures are obtained for the voice inputs detected by the plurality of devices. The audio signatures for the voice inputs and times that the voice inputs were detected are evaluated. Based on the evaluation, at least some of the plurality of devices are grouped to form a group of multiple devices that detected a same user utterance. A device from the group is selected to respond to the user utterance, and the multiple devices in the group are managed so that only the selected device outputs a synthesized speech response to the user utterance.

Managing voice interface devices

In some implementations, authentication tokens corresponding to known users of a device are stored on the device. An utterance from a speaker is received. The utterance is classified as spoken by a particular known user of the known users. A query that includes a representation of the utterance and an indication of the particular known user as the speaker is provided using the authentication token of the particular known user.

Growing community of inventors

Toronto, Canada

Meltem Oktem