Google Inc.

Waymo LLC

Deepmind Technologies Limited

Other

Matthew Nirvan Sharifi

Victor Carbune

Dominik Roblek

David Petrou

Gheorghe Postelnicu

Jakob Nicolaus Foerster

Jakob Foerster

Annie Chen

Alexander Houston Gruenstein

Yaniv Bernstein

Abhanshu Sharma

Aleksandar Kracun

Blaise Hilary Aguera-Arcas

Diego Melendo Casado

George Tzanetakis

Julian James Odell

Matthew Wiseman

Kenneth Edward Mixter

Christopher Thaddeus Hughes

Terry Tai

Yossi Matias

Thomas Bugnon

Jan Heinrich Althaus

Jorge Pereira

Beat Gfeller

Tom Hume

Cong Li

Kevin I Kilgour

Mihajlo Velimirović

Alfred Zalmon Spector

Kai Wang

Sergey Ioffe

Ignacio Lopez Moreno

Johan Schalkwyk

Gabriel Taubman

Brandon Charles Barbello

Daniel Ramage

Balint Miklos

Marco Tagliasacchi

Luv Kothari

Thomas Weedon Hume

Bogdan Prisacari

Christopher Thornton

Alexander Mordvintsev

Jacopo Sannazzaro Natta

Domenico Carbotta

Jae Seo

Felix Weissenberger

James David Lyon

Mucun Lu

Katsiaryna Naliuka

Thomas Qian

Neha Garg

Fo Lee

Matthew Miao

Barbara Poblocka

Kevin Fu

Ray Chen

Christian Frank

Kevin Kilgour

Ant Oztaskent

Yaroslav Volovich

Jay Yagnik

Richard Francis Lyon

Jay Michael Ponte

Wei Huang

Ioannis Agiomyrgiannakis

Michael Golikov

Daniel Jules Switkin

Sebastian Millius

Neil Zeghidour

Nicolo D'Ercole

Alexa Greenberg

Jesse Engel

Adam Roberts

Zalán Borsos

Muhammad Mohsin

Aleks Kracun

Justin Lu

James Lin

Jacek Slawosz Jurewicz

Jing Cao

Yanchao Su

Thomas Deselaers

Daniel M Keysers

David A Ross

Owen Daniel Otto

Raziel Alvarez Guevara

Haroon Baig

Graeme John Rimmer

Sammy El Ghazzal

Lars Fabian Krüger

Pranav Khaitan

Ben M Shahshahani

Thomas Binder

Sanjeev Dhanda

Arash Sadr

Damien Vincent

Olivier Claude Pietquin

Implementations are directed to enabling a representative associated with an entity to quickly and efficiently modify a voice bot associated with the entity. The voice bot can be previously trained to communicate with user(s) on behalf of the entity through various communication channels (e.g., a telephone communication channel, a software application communication channel, a messaging communication channel, etc.). Processor(s) of a computing device can receive, from the representative, representative input to modify behavior(s) and/or parameter(s) that the voice bot utilizes in communicating with the plurality of users via the communication channels, determine whether the representative is authorized to cause the behavior(s) and/or parameter(s) to be modified, and cause the behavior(s) and/or parameter(s) to be modified in response to determining that the representative is authorized. Notably, the representative input can be received through the same communication channels that the user(s) utilize to communicate with the voice bot.

System(s) and method(s) for enabling a representative associated with an entity to modify a trained voice bot associated with the entity

Implementations set forth herein relate to an automated assistant that can selectively communicate audio data to a recipient when a user solicits the automated assistant to send a text message to the recipient. The audio data can include a snippet of audio that characterizes content of the text message, and the automated assistant can communicate the audio data to the recipient when score data for a speech recognition hypothesis does not satisfy a confidence threshold. The score data can correspond to an entirety of content of a text message and/or speech recognition hypothesis, and/or less than an entirety of the content. A recipient device can optionally re-process the audio data using a model that is associated with the recipient device. This can provide more accurate transcripts in some instances, thereby improving accuracy of communications and decreasing a number of corrective messages sent between users.

Adaptive sending or rendering of audio with text messages sent via automated assistant

Systems and methods for obfuscating and/or omitting potentially sensitive information in a spoken query before providing the query to a secondary automated assistant. A general automated assistant may be invoked by a user, followed by a query. The audio data can be processed to omit and/or obfuscate potentially sensitive information before providing one or more processed queries to secondary automated assistants based on a trust metric associated with each of the secondary automated assistants. The trust metric for a secondary automated assistant is indicative of trust in being provided with sensitive information. In response, the automated assistants can generate responses, which can be filtered to provide a response to the user.

Selectively masking query content to provide to a secondary digital assistant

Systems and methods are provided for suggesting actions for selected text based on content displayed on a mobile device. An example method can include converting a selection made via a display device into a query, providing the query to an action suggestion model that is trained to predict an action given a query, each action being associated with a mobile application, receiving one or more predicted actions, and initiating display of the one or more predicted actions on the display device. Another example method can include identifying, from search records, queries where a website is highly ranked, the website being one of a plurality of websites in a mapping of websites to mobile applications. The method can also include generating positive training examples for an action suggestion model from the identified queries, and training the action suggestion model using the positive training examples.

Action suggestions for user-selected content

The present disclosure is directed to interactive voice navigation. In particular, a computing system can provide audio information including one or more navigation instructions to a user via a computing system associated with the user. The computing system can activate an audio sensor associated with the computing system. The computing system can collect, using the audio sensor, audio data associated with the user. The computing system can determine, based on the audio data, whether the audio data is associated with one or more navigation instructions. The computing system can, in accordance with a determination that the audio data is associated with one or more navigation instructions, determine a context-appropriate audio response. The computing system can provide the context-appropriate audio response to the user.

Interactive voice navigation

Implementations set forth herein relate to an automated assistant that allows a user to create, edit, and/or share documents without directly interfacing with a document editing application. The user can provide an input to the automated assistant in order to cause the automated assistant to interface with the document editing application and create a document. In order to identify a particular action to perform with respect to a document, and/or identify a particular subsection within the document to direct the action, the automated assistant can rely on semantic annotations. As a user continues to interact with the automated assistant to edit a document, the semantic annotations can be updated according to how the document is changing and/or how the user refers to the document. This can allow the automated assistant to more readily fulfill document-related requests that may lack express details.

Document creation and editing via automated assistant interactions

Implementations can determine an ambient state that reflects a state of a user and/or an environment of the user based on an instance of sensor data. The ambient state can be processed, using an ambient sensing machine learning (ML) model, to generate suggested action(s) that are suggested to be performed, on behalf of the user, by an automated assistant. In some implementations, a corresponding representation of the suggested action(s) can be provided for presentation to the user, and the suggested action(s) can be performed by the automated assistant in response to a user selection of the suggested action(s). In additional or alternative implementations, the suggested action(s) can be automatically performed by the automated assistant. Implementations can additionally or alternatively generate training instances for training the ambient sensing ML model based on interactions with the automated assistant.

Inferring assistant action(s) based on ambient sensing by assistant device(s)

Implementations relate to automatic generation of speaker features for each of one or more particular text-dependent speaker verifications (TD-SVs) for a user. Implementations can generate speaker features for a particular TD-SV using instances of audio data that each capture a corresponding spoken utterance of the user during normal non-enrollment interactions with an automated assistant via one or more respective assistant devices. For example, a portion of an instance of audio data can be used in response to: (a) determining that recognized term(s) for the spoken utterance captured by that the portion correspond to the particular TD-SV; and (b) determining that an authentication measure, for the user and for the spoken utterance, satisfies a threshold. Implementations additionally or alternatively relate to utilization of speaker features, for each of one or more particular TD-SVs for a user, in determining whether to authenticate a spoken utterance for the user.

Automatic generation and/or use of text-dependent speaker verification features

Implementations relate to transferring actions from a shared device to a personal device that is associated with an account of a user. Some implementations relate to determining that a request is associated with sensitive information, determining that one or more other users are co-present with the shared device, and transferring the request that is related to sensitive information to a personal device of the user. Some implementations relate determining that a user is no longer co-present with a shared device that is currently performing one or more actions and transferring one or more of the actions to a personal device that is associated with an account of the user.

Transferring actions from a shared device to a personal device associated with an account of a user

A method includes a first assistant-enabled device (AED) receiving an assignment instruction assigning a group hotword to a selected group of AEDs that includes the first AED and one or more other AEDs. Each AED is configured to wake-up from a low-power state when the group hotword is detected in streaming audio by at least one of the AEDs. The method also includes receiving audio data that corresponds to an utterance spoken by the user and includes a query that specifies an operation to perform. In response to detecting the group hotword in the audio data, the method also includes triggering the first AED to wake-up from the low-power state and executing a collaboration routine to cause the first AED and each other AED in the selected group of AEDs to collaborate with one another to fulfill performance of the operation specified by the query.

Group hotwords

Automated content switching rules may be generated and/or utilized for automatically switching away from certain interactive content during presentation of that interactive content when one or more switch conditions are met. In some instances, automated content switching rules may define one or more non-temporal switch conditions, e.g., based upon reaching certain points or milestones in certain interactive content, that may be used to initiate actions that switch away from the interactive content. In addition, in some instances, automated content switching rules may be used to not only switch away from particular interactive content, but additionally switch to other interactive content, thereby enabling a user to effectively schedule a workflow across different interactive content, applications and/or other computer-related tasks.

Automated content switching rule for controlling access to interactive content on a computing device

A method for sharing assistant profiles includes receiving, at a profile service, from an assistant service interacting with a user device of a user, a request requesting the profile service to release personal information associated with the user to the assistant service. The operations also include performing, through the assistant service, a verification process to verify that the user consents to releasing the requested personal information by: instructing the assistant service to prompt the user to recite a unique token prescribed to the user; receiving audio data characterizing a spoken utterance captured by the user device of the user; processing the audio data to determine whether a transcription of the spoken utterance recites the unique token; and when the transcription of the spoken utterance recites the unique token, releasing, to the assistant service, the requested personal information stored on a centralized data store managed by the profile service.

Shared assistant profiles verified via speaker identification

Implementations set forth herein relate to an automated assistant that can be accessible via a virtual environment for controlling features of the virtual environment and/or devices in a physical environment of the user. When the automated assistant is invoked, the automated assistant can materialize in the virtual environment according to any request that the automated assistant has been invoked to fulfill. For example, depending on the request from the user, the automated assistant can cause rendering of a virtual object for fulfilling the request and/or controlling an ongoing operation of the automated assistant. When the virtual object is rendered to control an operation of the automated assistant, or another application, the virtual object can include a virtual feature that the user can interact with to control the virtual environment and/or devices in a physical environment of the user.

Facilitating virtual or physical assistant interactions with virtual objects in a virtual environment

Methods, systems, and apparatus for receiving, from a user, a request that includes an entity identifier associated with an entity that is referenced by one or more query terms of a search query, determining that the entity is identified in a media consumption database as a media item that has been indicated as consumed by the user or that the entity is associated with a media item that is identified in the media consumption database as a media item that has been indicated as consumed by the user, and based on the determination, providing a response to the request, the response including data indicating that the entity is a media item that has been indicated as consumed by the user or that the entity is associated with a media item that has been indicated as consumed by the user.

Media consumption history

A method includes detecting a presence of multiple users within an environment of an assistant-enabled device (AED) and obtaining, for each user, a respective active set of warm words that each specify a respective action for a digital assistant to perform. Based on each respective active set of warm words, the method also includes executing a warm word arbitration routine to enable a final set of warm words for detection by the AED. Here, the final set of warm words includes warm words selected from the respective active set of warm words. While the final set of warm words are enabled, the method also includes receiving audio data corresponding to an utterance captured by the AED, detecting a warm word from the final set of warm words in the audio data, and instructing the digital assistant to perform the respective action specified by the detected warm word.

Multi-user warm words

Growing community of inventors

Kilchberg, Switzerland

Matthew Nirvan Sharifi