Google Inc.

Waymo LLC

Deepmind Technologies Limited

Other

Matthew Nirvan Sharifi

Victor Carbune

Dominik Roblek

David Petrou

Gheorghe Postelnicu

Jakob Nicolaus Foerster

Jakob Foerster

Annie Chen

Alexander Houston Gruenstein

Yaniv Bernstein

Abhanshu Sharma

Aleksandar Kracun

Diego Melendo Casado

Blaise Hilary Aguera-Arcas

George Tzanetakis

Julian James Odell

Matthew Wiseman

Kenneth Edward Mixter

Christopher Thaddeus Hughes

Terry Tai

Yossi Matias

Thomas Bugnon

Jan Heinrich Althaus

Jorge Pereira

Beat Gfeller

Tom Hume

Cong Li

Kevin I Kilgour

Alfred Zalmon Spector

Kai Wang

Sergey Ioffe

Ignacio Lopez Moreno

Johan Schalkwyk

Gabriel Taubman

Brandon Charles Barbello

Balint Miklos

Thomas Weedon Hume

Bogdan Prisacari

Luv Kothari

Marco Tagliasacchi

Christopher Thornton

Alexander Mordvintsev

Domenico Carbotta

Jacopo Sannazzaro Natta

Felix Weissenberger

Jae Seo

James David Lyon

Christian Frank

Neha Garg

Katsiaryna Naliuka

Fo Lee

Mucun Lu

Barbara Poblocka

Matthew Miao

Kevin Fu

Ray Chen

Kevin Kilgour

Marvin Karl Ritter

Thomas Qian

Ludwig Schmidt

Ant Oztaskent

Yaroslav Volovich

Jay Yagnik

Richard Francis Lyon

Jay Michael Ponte

Daniel Ramage

Wei Huang

Ioannis Agiomyrgiannakis

Michael Golikov

Daniel Jules Switkin

Sebastian Millius

Nicolo D'Ercole

Neil Zeghidour

Alexa Greenberg

Jesse Engel

Adam Roberts

Muhammad Mohsin

Zalán Borsos

Aleks Kracun

James Lin

Jing Cao

Yanchao Su

Justin Lu

Jacek Slawosz Jurewicz

Nicholas Kong

Thomas Deselaers

Daniel M Keysers

David A Ross

Owen Daniel Otto

Raziel Alvarez Guevara

Haroon Baig

Graeme John Rimmer

Sammy El Ghazzal

Ben M Shahshahani

Pranav Khaitan

Lars Fabian Krüger

Sanjeev Dhanda

Thomas Binder

Arash Sadr

A method for sharing assistant profiles includes receiving, at a profile service, from an assistant service interacting with a user device of a user, a request requesting the profile service to release personal information associated with the user to the assistant service. The operations also include performing, through the assistant service, a verification process to verify that the user consents to releasing the requested personal information by: instructing the assistant service to prompt the user to recite a unique token prescribed to the user; receiving audio data characterizing a spoken utterance captured by the user device of the user; processing the audio data to determine whether a transcription of the spoken utterance recites the unique token; and when the transcription of the spoken utterance recites the unique token, releasing, to the assistant service, the requested personal information stored on a centralized data store managed by the profile service.

Shared assistant profiles verified via speaker identification

Implementations set forth herein relate to an automated assistant that can be accessible via a virtual environment for controlling features of the virtual environment and/or devices in a physical environment of the user. When the automated assistant is invoked, the automated assistant can materialize in the virtual environment according to any request that the automated assistant has been invoked to fulfill. For example, depending on the request from the user, the automated assistant can cause rendering of a virtual object for fulfilling the request and/or controlling an ongoing operation of the automated assistant. When the virtual object is rendered to control an operation of the automated assistant, or another application, the virtual object can include a virtual feature that the user can interact with to control the virtual environment and/or devices in a physical environment of the user.

Facilitating virtual or physical assistant interactions with virtual objects in a virtual environment

Methods, systems, and apparatus for receiving, from a user, a request that includes an entity identifier associated with an entity that is referenced by one or more query terms of a search query, determining that the entity is identified in a media consumption database as a media item that has been indicated as consumed by the user or that the entity is associated with a media item that is identified in the media consumption database as a media item that has been indicated as consumed by the user, and based on the determination, providing a response to the request, the response including data indicating that the entity is a media item that has been indicated as consumed by the user or that the entity is associated with a media item that has been indicated as consumed by the user.

Media consumption history

A method includes detecting a presence of multiple users within an environment of an assistant-enabled device (AED) and obtaining, for each user, a respective active set of warm words that each specify a respective action for a digital assistant to perform. Based on each respective active set of warm words, the method also includes executing a warm word arbitration routine to enable a final set of warm words for detection by the AED. Here, the final set of warm words includes warm words selected from the respective active set of warm words. While the final set of warm words are enabled, the method also includes receiving audio data corresponding to an utterance captured by the AED, detecting a warm word from the final set of warm words in the audio data, and instructing the digital assistant to perform the respective action specified by the detected warm word.

Multi-user warm words

A method for adapting hotword recognition includes receiving audio data characterizing a hotword event detected by a first stage hotword detector in streaming audio captured by a user device. The method also includes processing, using a second stage hotword detector, the audio data to determine whether a hotword is detected by the second stage hotword detector in a first segment of the audio data. When the hotword is not detected by the second stage hotword detector, the method includes, classifying the first segment of the audio data as containing a negative hotword that caused a false detection of the hotword event in the streaming audio by the first stage hotword detector. Based on the first segment of the audio data classified as containing the negative hotword, the method includes updating the first stage hotword detector to prevent triggering the hotword event in subsequent audio data that contains the negative hotword.

Adapting hotword recognition based on personalized negatives

Implementations set forth herein relate to an automated assistant that can employ various utilities according to whether certain conditions are satisfied for a given utility and/or whether a particular input is determined to have a threshold degree of relevance to the given utility. The automated assistant can be customized by a third party entity, which can make the automated assistant available via a device and/or application. The automated assistant can operate according to certain utilities that have been permitted by the third party entity. However, the third party entity can allow the automated assistant to employ other scalable utilities when a user input is determined to have a threshold degree of relevance and/or when usage of certain scalable utilities has not exceeded a threshold. The utility can refer to a machine learning model and/or other data that can be employed to resolve an input.

Condition dependent scalable utilities for an automated assistant

Methods, systems, devices, and tangible non-transitory computer readable media for using incoming communications to generate suggestions for navigation. The disclosed technology can include accessing route data that includes information associated with navigation from a starting location to a destination. Based on the route data, one or more routes from the starting location to the destination can be determined. Message data including one or more messages to a user can be accessed. Based on the message data and one or more machine-learned models, at least one entity and objectives that are associated with the one or more messages can be determined. Based on the one or more routes, the at least one entity, and the objectives, suggestions associated with the one or more messages can be determined. Furthermore, output including indications associated with the suggestions directed to the user can be generated via a user interface.

Message based navigational assistance

Techniques are described herein for identifying a failed hotword attempt. A method includes: receiving first audio data; processing the first audio data to generate a first predicted output; determining that the first predicted output satisfies a secondary threshold but does not satisfy a primary threshold; receiving second audio data; processing the second audio data to generate a second predicted output; determining that the second predicted output satisfies the secondary threshold but does not satisfy the primary threshold; in response to the first predicted output and the second predicted output satisfying the secondary threshold but not satisfying the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more temporal criteria relative to one another, identifying a failed hotword attempt; and in response to identifying the failed hotword attempt, providing a hint that is responsive to the failed hotword attempt.

Detecting near matches to a hotword or phrase

Implementations set forth herein relate to an automated assistant that can perform operations to revert various applications to prior states that the applications may have arrived at via certain user inputs. The user can provide a spoken utterance such as, 'undo,' in order to cause the automated assistant to identify a particular application that the user may want to affect with the “undo” command. When the particular application is identified, the automated assistant can identify one or more operations recently performed using the particular application. In some implementations, the automated assistant can provide the user with a variety of undo options in response to an “undo” command. For instance, the automated assistant can prompt the user to select one of a first cluster of operations and/or a second cluster of operations to be undone, and each cluster can refer to different operations.

Undoing application operation(s) via user interaction(s) with an automated assistant

A method includes receiving user queries directed toward a cloud-based assistant service. For each received user query directed toward the cloud-based assistant service, the method also includes extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The method also includes determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. The distilled model of the cloud-based assistant service is configured to execute on one or more target client devices.

Distilling to a target device based on observed query patterns

Implementations are directed to receiving a search query from a user, obtaining environmental signal(s) associated with an environment in which the user is located when the search query is received, processing the environmental signal(s) to generate a privacy measure associated with submission of the search query, obtaining additional environmental signal(s) associated with the environment in which the user is located when user input directed to a search interface is received, processing the additional environmental signal(s) to generate an additional privacy measure associated with the user input, selecting, from a superset of historical search queries of the user, a subset of the historical search queries based on at least the privacy measure and the additional privacy measure, and causing the subset of the historical search queries to be presented to the user in response to receiving the user input directed to the search interface.

Methods and systems for presenting privacy friendly query activity based on environmental signal(s)

A method for combining hotwords in a single utterance receives, at a first assistant-enabled device (AED), audio data corresponding to an utterance directed toward the first AED and a second AED among two or more AEDs where the audio data includes a query specifying an operation to perform. The method also detects, using a hotword detector, a first hotword assigned to the first AED that is different than a second hotword assigned to the second AED. In response to detecting the first hotword, the method initiates processing on the audio data to determine that the audio data includes a term preceding the query that at least partially matches the second hotword assigned. Based on the at least partial match, the method executes a collaboration routine to cause the first AED and the second AED to collaborate with one another to fulfill the query.

Combining device or assistant-specific hotwords in a single utterance

Implementations set forth herein relate to an automated assistant that can operate in a transient personalization mode, and/or assist a separate automated assistant with providing output according to a transient personalization mode. The transient personalization mode can allow a guest user of an assistant enabled-device to receive personalized responses from the assistant-enabled device—despite not being signed into the assistant-enabled device. A host automated assistant of the assistant-enabled device can securely communicate with a guest user's automated assistant through a backend process. In this way, input queries from the guest user to the host automated assistant can be personalized according to the guest automated assistant—without the guest user directly engaging with their own personal device.

Transient personalization mode for guest users of an automated assistant

Systems and methods for determining, based on invocation input that is common to multiple automated assistants, which automated assistant to invoke in lieu of invoking other automated assistants. The invocation input is processed to determine one or more invocation features that may be utilized to determine which, of a plurality of candidate automated assistants, to invoke. Further, additional features are processed that can indicate which, of the plurality of invocable automated assistants, to invoke. Once an automated assistant has been invoked, additional audio data and/or features of additional audio data are provided to the invoked automated assistant for further processing.

Selecting between multiple automated assistants based on invocation properties

While an assistant-enabled device is playing back media content, a method includes receiving a contextual signal from an environment of the assistant-enabled device and executing an event recognition routine to determine whether the received contextual signal is indicative of an event that conflicts with the playback of the media content from the assistant-enabled device. When the event recognition routine determines that the received contextual signal is indicative of the event that conflicts with the playback of the media content, the method also includes adjusting content playback settings of the assistant-enabled device.

Integrating short-term context for content playback adaption

Growing community of inventors

Kilchberg, Switzerland

Matthew Nirvan Sharifi