Google Inc.

Waymo LLC

Deepmind Technologies Limited

Other

Matthew Nirvan Sharifi

Victor Carbune

Dominik Roblek

David Petrou

Gheorghe Postelnicu

Jakob Nicolaus Foerster

Jakob Foerster

Annie Chen

Alexander Houston Gruenstein

Yaniv Bernstein

Abhanshu Sharma

Aleksandar Kracun

Diego Melendo Casado

Blaise Hilary Aguera-Arcas

George Tzanetakis

Julian James Odell

Matthew Wiseman

Kenneth Edward Mixter

Christopher Thaddeus Hughes

Terry Tai

Yossi Matias

Thomas Bugnon

Jan Heinrich Althaus

Jorge Pereira

Beat Gfeller

Tom Hume

Cong Li

Kevin I Kilgour

Mihajlo Velimirović

Alfred Zalmon Spector

Kai Wang

Sergey Ioffe

Ignacio Lopez Moreno

Johan Schalkwyk

Gabriel Taubman

Brandon Charles Barbello

Daniel Ramage

Balint Miklos

Luv Kothari

Thomas Weedon Hume

Bogdan Prisacari

Marco Tagliasacchi

Christopher Thornton

Alexander Mordvintsev

Jacopo Sannazzaro Natta

Domenico Carbotta

Jae Seo

Felix Weissenberger

James David Lyon

Mucun Lu

Katsiaryna Naliuka

Thomas Qian

Neha Garg

Fo Lee

Matthew Miao

Barbara Poblocka

Kevin Fu

Ray Chen

Christian Frank

Kevin Kilgour

Ant Oztaskent

Yaroslav Volovich

Jay Yagnik

Richard Francis Lyon

Jay Michael Ponte

Wei Huang

Ioannis Agiomyrgiannakis

Michael Golikov

Daniel Jules Switkin

Sebastian Millius

Nicolo D'Ercole

Neil Zeghidour

Alexa Greenberg

Jesse Engel

Adam Roberts

Zalán Borsos

Muhammad Mohsin

Aleks Kracun

Justin Lu

James Lin

Jacek Slawosz Jurewicz

Jing Cao

Yanchao Su

Thomas Deselaers

Daniel M Keysers

David A Ross

Owen Daniel Otto

Raziel Alvarez Guevara

Haroon Baig

Graeme John Rimmer

Sammy El Ghazzal

Lars Fabian Krüger

Pranav Khaitan

Ben M Shahshahani

Thomas Binder

Sanjeev Dhanda

Damien Vincent

Arash Sadr

Olivier Claude Pietquin

Systems and methods are provided for suggesting actions for selected text based on content displayed on a mobile device. An example method can include converting a selection made via a display device into a query, providing the query to an action suggestion model that is trained to predict an action given a query, each action being associated with a mobile application, receiving one or more predicted actions, and initiating display of the one or more predicted actions on the display device. Another example method can include identifying, from search records, queries where a website is highly ranked, the website being one of a plurality of websites in a mapping of websites to mobile applications. The method can also include generating positive training examples for an action suggestion model from the identified queries, and training the action suggestion model using the positive training examples.

Action suggestions for user-selected content

The present disclosure is directed to interactive voice navigation. In particular, a computing system can provide audio information including one or more navigation instructions to a user via a computing system associated with the user. The computing system can activate an audio sensor associated with the computing system. The computing system can collect, using the audio sensor, audio data associated with the user. The computing system can determine, based on the audio data, whether the audio data is associated with one or more navigation instructions. The computing system can, in accordance with a determination that the audio data is associated with one or more navigation instructions, determine a context-appropriate audio response. The computing system can provide the context-appropriate audio response to the user.

Interactive voice navigation

Implementations set forth herein relate to an automated assistant that allows a user to create, edit, and/or share documents without directly interfacing with a document editing application. The user can provide an input to the automated assistant in order to cause the automated assistant to interface with the document editing application and create a document. In order to identify a particular action to perform with respect to a document, and/or identify a particular subsection within the document to direct the action, the automated assistant can rely on semantic annotations. As a user continues to interact with the automated assistant to edit a document, the semantic annotations can be updated according to how the document is changing and/or how the user refers to the document. This can allow the automated assistant to more readily fulfill document-related requests that may lack express details.

Document creation and editing via automated assistant interactions

Implementations can determine an ambient state that reflects a state of a user and/or an environment of the user based on an instance of sensor data. The ambient state can be processed, using an ambient sensing machine learning (ML) model, to generate suggested action(s) that are suggested to be performed, on behalf of the user, by an automated assistant. In some implementations, a corresponding representation of the suggested action(s) can be provided for presentation to the user, and the suggested action(s) can be performed by the automated assistant in response to a user selection of the suggested action(s). In additional or alternative implementations, the suggested action(s) can be automatically performed by the automated assistant. Implementations can additionally or alternatively generate training instances for training the ambient sensing ML model based on interactions with the automated assistant.

Inferring assistant action(s) based on ambient sensing by assistant device(s)

Implementations relate to automatic generation of speaker features for each of one or more particular text-dependent speaker verifications (TD-SVs) for a user. Implementations can generate speaker features for a particular TD-SV using instances of audio data that each capture a corresponding spoken utterance of the user during normal non-enrollment interactions with an automated assistant via one or more respective assistant devices. For example, a portion of an instance of audio data can be used in response to: (a) determining that recognized term(s) for the spoken utterance captured by that the portion correspond to the particular TD-SV; and (b) determining that an authentication measure, for the user and for the spoken utterance, satisfies a threshold. Implementations additionally or alternatively relate to utilization of speaker features, for each of one or more particular TD-SVs for a user, in determining whether to authenticate a spoken utterance for the user.

Automatic generation and/or use of text-dependent speaker verification features

Implementations relate to transferring actions from a shared device to a personal device that is associated with an account of a user. Some implementations relate to determining that a request is associated with sensitive information, determining that one or more other users are co-present with the shared device, and transferring the request that is related to sensitive information to a personal device of the user. Some implementations relate determining that a user is no longer co-present with a shared device that is currently performing one or more actions and transferring one or more of the actions to a personal device that is associated with an account of the user.

Transferring actions from a shared device to a personal device associated with an account of a user

A method includes a first assistant-enabled device (AED) receiving an assignment instruction assigning a group hotword to a selected group of AEDs that includes the first AED and one or more other AEDs. Each AED is configured to wake-up from a low-power state when the group hotword is detected in streaming audio by at least one of the AEDs. The method also includes receiving audio data that corresponds to an utterance spoken by the user and includes a query that specifies an operation to perform. In response to detecting the group hotword in the audio data, the method also includes triggering the first AED to wake-up from the low-power state and executing a collaboration routine to cause the first AED and each other AED in the selected group of AEDs to collaborate with one another to fulfill performance of the operation specified by the query.

Group hotwords

Automated content switching rules may be generated and/or utilized for automatically switching away from certain interactive content during presentation of that interactive content when one or more switch conditions are met. In some instances, automated content switching rules may define one or more non-temporal switch conditions, e.g., based upon reaching certain points or milestones in certain interactive content, that may be used to initiate actions that switch away from the interactive content. In addition, in some instances, automated content switching rules may be used to not only switch away from particular interactive content, but additionally switch to other interactive content, thereby enabling a user to effectively schedule a workflow across different interactive content, applications and/or other computer-related tasks.

Automated content switching rule for controlling access to interactive content on a computing device

A method for sharing assistant profiles includes receiving, at a profile service, from an assistant service interacting with a user device of a user, a request requesting the profile service to release personal information associated with the user to the assistant service. The operations also include performing, through the assistant service, a verification process to verify that the user consents to releasing the requested personal information by: instructing the assistant service to prompt the user to recite a unique token prescribed to the user; receiving audio data characterizing a spoken utterance captured by the user device of the user; processing the audio data to determine whether a transcription of the spoken utterance recites the unique token; and when the transcription of the spoken utterance recites the unique token, releasing, to the assistant service, the requested personal information stored on a centralized data store managed by the profile service.

Shared assistant profiles verified via speaker identification

Implementations set forth herein relate to an automated assistant that can be accessible via a virtual environment for controlling features of the virtual environment and/or devices in a physical environment of the user. When the automated assistant is invoked, the automated assistant can materialize in the virtual environment according to any request that the automated assistant has been invoked to fulfill. For example, depending on the request from the user, the automated assistant can cause rendering of a virtual object for fulfilling the request and/or controlling an ongoing operation of the automated assistant. When the virtual object is rendered to control an operation of the automated assistant, or another application, the virtual object can include a virtual feature that the user can interact with to control the virtual environment and/or devices in a physical environment of the user.

Facilitating virtual or physical assistant interactions with virtual objects in a virtual environment

Methods, systems, and apparatus for receiving, from a user, a request that includes an entity identifier associated with an entity that is referenced by one or more query terms of a search query, determining that the entity is identified in a media consumption database as a media item that has been indicated as consumed by the user or that the entity is associated with a media item that is identified in the media consumption database as a media item that has been indicated as consumed by the user, and based on the determination, providing a response to the request, the response including data indicating that the entity is a media item that has been indicated as consumed by the user or that the entity is associated with a media item that has been indicated as consumed by the user.

Media consumption history

A method includes detecting a presence of multiple users within an environment of an assistant-enabled device (AED) and obtaining, for each user, a respective active set of warm words that each specify a respective action for a digital assistant to perform. Based on each respective active set of warm words, the method also includes executing a warm word arbitration routine to enable a final set of warm words for detection by the AED. Here, the final set of warm words includes warm words selected from the respective active set of warm words. While the final set of warm words are enabled, the method also includes receiving audio data corresponding to an utterance captured by the AED, detecting a warm word from the final set of warm words in the audio data, and instructing the digital assistant to perform the respective action specified by the detected warm word.

Multi-user warm words

A method for adapting hotword recognition includes receiving audio data characterizing a hotword event detected by a first stage hotword detector in streaming audio captured by a user device. The method also includes processing, using a second stage hotword detector, the audio data to determine whether a hotword is detected by the second stage hotword detector in a first segment of the audio data. When the hotword is not detected by the second stage hotword detector, the method includes, classifying the first segment of the audio data as containing a negative hotword that caused a false detection of the hotword event in the streaming audio by the first stage hotword detector. Based on the first segment of the audio data classified as containing the negative hotword, the method includes updating the first stage hotword detector to prevent triggering the hotword event in subsequent audio data that contains the negative hotword.

Adapting hotword recognition based on personalized negatives

Implementations set forth herein relate to an automated assistant that can employ various utilities according to whether certain conditions are satisfied for a given utility and/or whether a particular input is determined to have a threshold degree of relevance to the given utility. The automated assistant can be customized by a third party entity, which can make the automated assistant available via a device and/or application. The automated assistant can operate according to certain utilities that have been permitted by the third party entity. However, the third party entity can allow the automated assistant to employ other scalable utilities when a user input is determined to have a threshold degree of relevance and/or when usage of certain scalable utilities has not exceeded a threshold. The utility can refer to a machine learning model and/or other data that can be employed to resolve an input.

Condition dependent scalable utilities for an automated assistant

Methods, systems, devices, and tangible non-transitory computer readable media for using incoming communications to generate suggestions for navigation. The disclosed technology can include accessing route data that includes information associated with navigation from a starting location to a destination. Based on the route data, one or more routes from the starting location to the destination can be determined. Message data including one or more messages to a user can be accessed. Based on the message data and one or more machine-learned models, at least one entity and objectives that are associated with the one or more messages can be determined. Based on the one or more routes, the at least one entity, and the objectives, suggestions associated with the one or more messages can be determined. Furthermore, output including indications associated with the suggestions directed to the user can be generated via a user interface.

Message based navigational assistance

Growing community of inventors

Kilchberg, Switzerland

Matthew Nirvan Sharifi