Google Inc.

Victor Carbune

Matthew Nirvan Sharifi

Thomas Deselaers

Daniel M Keysers

Sandro Feuz

Pedro Gonnet Anders

Luv Kothari

Jacopo Sannazzaro Natta

Nicolo D'Ercole

Justin Lu

Behshad Behzadi

Zaheed Sabur

Balint Miklos

Bogdan Prisacari

Julia Proskurnia

Domenico Carbotta

Felix Weissenberger

Jae Seo

Barbara Poblocka

Alexandru-Marian Damian

Thomas Qian

Matthew Miao

Mucun Lu

Fo Lee

Kevin Fu

Ray Chen

Neha Garg

Michael Golikov

Mohamad Hassan Rom

Daniel Valcarce

Ondrej Skopek

Krishna Sapkota

Magali Boizot-Roche

Márius Sajgalík

Henry Allan Rowley

Blaise Hilary Aguera-Arcas

Haroon Baig

Alvin Abdagic

Kevin I Kilgour

Arash Sadr

Matthew Sharifi

Srikanth Pandiri

Andrii Maksai

Kevin Kilgour

Viesturs Zarins

Kevin Allekotte

Paula Marques Fernandes

Krzysztof Andrzej Goj

Marcos Calvo

Daniel Keysers

Daniel Martin Keysers

Matt Sharifi

Implementations set forth herein relate to an automated assistant that allows a user to create, edit, and/or share documents without directly interfacing with a document editing application. The user can provide an input to the automated assistant in order to cause the automated assistant to interface with the document editing application and create a document. In order to identify a particular action to perform with respect to a document, and/or identify a particular subsection within the document to direct the action, the automated assistant can rely on semantic annotations. As a user continues to interact with the automated assistant to edit a document, the semantic annotations can be updated according to how the document is changing and/or how the user refers to the document. This can allow the automated assistant to more readily fulfill document-related requests that may lack express details.

Document creation and editing via automated assistant interactions

Implementations can determine an ambient state that reflects a state of a user and/or an environment of the user based on an instance of sensor data. The ambient state can be processed, using an ambient sensing machine learning (ML) model, to generate suggested action(s) that are suggested to be performed, on behalf of the user, by an automated assistant. In some implementations, a corresponding representation of the suggested action(s) can be provided for presentation to the user, and the suggested action(s) can be performed by the automated assistant in response to a user selection of the suggested action(s). In additional or alternative implementations, the suggested action(s) can be automatically performed by the automated assistant. Implementations can additionally or alternatively generate training instances for training the ambient sensing ML model based on interactions with the automated assistant.

Inferring assistant action(s) based on ambient sensing by assistant device(s)

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing actionable suggestions are disclosed. In one aspect, a method includes receiving (i) an indication that an event detection module has determined that a shared event of a particular type is presently occurring or has occurred, and (ii) data referencing an attribute associated with the shared event. The method includes selecting, from among multiple output templates that are each associated with a different type of shared event, a particular output template associated with the particular type of shared event detected by the module. The method generates a notification for output using at least (i) the selected particular output template, and (ii) the data referencing the attribute associated with the shared event. The method then provides, for output to a user device, the notification that is generated.

Actionable suggestions for activities

Implementations relate to automatic generation of speaker features for each of one or more particular text-dependent speaker verifications (TD-SVs) for a user. Implementations can generate speaker features for a particular TD-SV using instances of audio data that each capture a corresponding spoken utterance of the user during normal non-enrollment interactions with an automated assistant via one or more respective assistant devices. For example, a portion of an instance of audio data can be used in response to: (a) determining that recognized term(s) for the spoken utterance captured by that the portion correspond to the particular TD-SV; and (b) determining that an authentication measure, for the user and for the spoken utterance, satisfies a threshold. Implementations additionally or alternatively relate to utilization of speaker features, for each of one or more particular TD-SVs for a user, in determining whether to authenticate a spoken utterance for the user.

Automatic generation and/or use of text-dependent speaker verification features

Implementations relate to transferring actions from a shared device to a personal device that is associated with an account of a user. Some implementations relate to determining that a request is associated with sensitive information, determining that one or more other users are co-present with the shared device, and transferring the request that is related to sensitive information to a personal device of the user. Some implementations relate determining that a user is no longer co-present with a shared device that is currently performing one or more actions and transferring one or more of the actions to a personal device that is associated with an account of the user.

Transferring actions from a shared device to a personal device associated with an account of a user

A method includes a first assistant-enabled device (AED) receiving an assignment instruction assigning a group hotword to a selected group of AEDs that includes the first AED and one or more other AEDs. Each AED is configured to wake-up from a low-power state when the group hotword is detected in streaming audio by at least one of the AEDs. The method also includes receiving audio data that corresponds to an utterance spoken by the user and includes a query that specifies an operation to perform. In response to detecting the group hotword in the audio data, the method also includes triggering the first AED to wake-up from the low-power state and executing a collaboration routine to cause the first AED and each other AED in the selected group of AEDs to collaborate with one another to fulfill performance of the operation specified by the query.

Group hotwords

Implementations described herein relate to an application and/or automated assistant that can identify arrangement operations to perform for arranging text during speech-to-text operations—without a user having to expressly identify the arrangement operations. In some instances, a user that is dictating a document (e.g., an email, a text message, etc.) can provide a spoken utterance to an application in order to incorporate textual content. However, in some of these instances, certain corresponding arrangements are needed for the textual content in the document. The textual content that is derived from the spoken utterance can be arranged by the application based on an intent, vocalization features, and/or contextual features associated with the spoken utterance and/or a type of the application associated with the document, without the user expressly identifying the corresponding arrangements. In this way, the application can infer content arrangement operations from a spoken utterance that only specifies the textual content.

Arranging and/or clearing speech-to-text content without a user providing express instructions

Automated content switching rules may be generated and/or utilized for automatically switching away from certain interactive content during presentation of that interactive content when one or more switch conditions are met. In some instances, automated content switching rules may define one or more non-temporal switch conditions, e.g., based upon reaching certain points or milestones in certain interactive content, that may be used to initiate actions that switch away from the interactive content. In addition, in some instances, automated content switching rules may be used to not only switch away from particular interactive content, but additionally switch to other interactive content, thereby enabling a user to effectively schedule a workflow across different interactive content, applications and/or other computer-related tasks.

Automated content switching rule for controlling access to interactive content on a computing device

A method for sharing assistant profiles includes receiving, at a profile service, from an assistant service interacting with a user device of a user, a request requesting the profile service to release personal information associated with the user to the assistant service. The operations also include performing, through the assistant service, a verification process to verify that the user consents to releasing the requested personal information by: instructing the assistant service to prompt the user to recite a unique token prescribed to the user; receiving audio data characterizing a spoken utterance captured by the user device of the user; processing the audio data to determine whether a transcription of the spoken utterance recites the unique token; and when the transcription of the spoken utterance recites the unique token, releasing, to the assistant service, the requested personal information stored on a centralized data store managed by the profile service.

Shared assistant profiles verified via speaker identification

Implementations set forth herein relate to an automated assistant that can be accessible via a virtual environment for controlling features of the virtual environment and/or devices in a physical environment of the user. When the automated assistant is invoked, the automated assistant can materialize in the virtual environment according to any request that the automated assistant has been invoked to fulfill. For example, depending on the request from the user, the automated assistant can cause rendering of a virtual object for fulfilling the request and/or controlling an ongoing operation of the automated assistant. When the virtual object is rendered to control an operation of the automated assistant, or another application, the virtual object can include a virtual feature that the user can interact with to control the virtual environment and/or devices in a physical environment of the user.

Facilitating virtual or physical assistant interactions with virtual objects in a virtual environment

A method includes detecting a presence of multiple users within an environment of an assistant-enabled device (AED) and obtaining, for each user, a respective active set of warm words that each specify a respective action for a digital assistant to perform. Based on each respective active set of warm words, the method also includes executing a warm word arbitration routine to enable a final set of warm words for detection by the AED. Here, the final set of warm words includes warm words selected from the respective active set of warm words. While the final set of warm words are enabled, the method also includes receiving audio data corresponding to an utterance captured by the AED, detecting a warm word from the final set of warm words in the audio data, and instructing the digital assistant to perform the respective action specified by the detected warm word.

Multi-user warm words

Implementations set forth herein relate to an automated assistant that can employ various utilities according to whether certain conditions are satisfied for a given utility and/or whether a particular input is determined to have a threshold degree of relevance to the given utility. The automated assistant can be customized by a third party entity, which can make the automated assistant available via a device and/or application. The automated assistant can operate according to certain utilities that have been permitted by the third party entity. However, the third party entity can allow the automated assistant to employ other scalable utilities when a user input is determined to have a threshold degree of relevance and/or when usage of certain scalable utilities has not exceeded a threshold. The utility can refer to a machine learning model and/or other data that can be employed to resolve an input.

Condition dependent scalable utilities for an automated assistant

Techniques are described herein for identifying a failed hotword attempt. A method includes: receiving first audio data; processing the first audio data to generate a first predicted output; determining that the first predicted output satisfies a secondary threshold but does not satisfy a primary threshold; receiving second audio data; processing the second audio data to generate a second predicted output; determining that the second predicted output satisfies the secondary threshold but does not satisfy the primary threshold; in response to the first predicted output and the second predicted output satisfying the secondary threshold but not satisfying the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more temporal criteria relative to one another, identifying a failed hotword attempt; and in response to identifying the failed hotword attempt, providing a hint that is responsive to the failed hotword attempt.

Detecting near matches to a hotword or phrase

Implementations set forth herein relate to an automated assistant that can perform operations to revert various applications to prior states that the applications may have arrived at via certain user inputs. The user can provide a spoken utterance such as, 'undo,' in order to cause the automated assistant to identify a particular application that the user may want to affect with the “undo” command. When the particular application is identified, the automated assistant can identify one or more operations recently performed using the particular application. In some implementations, the automated assistant can provide the user with a variety of undo options in response to an “undo” command. For instance, the automated assistant can prompt the user to select one of a first cluster of operations and/or a second cluster of operations to be undone, and each cluster can refer to different operations.

Undoing application operation(s) via user interaction(s) with an automated assistant

A method includes receiving user queries directed toward a cloud-based assistant service. For each received user query directed toward the cloud-based assistant service, the method also includes extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The method also includes determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. The distilled model of the cloud-based assistant service is configured to execute on one or more target client devices.

Distilling to a target device based on observed query patterns

Growing community of inventors

Winterthur, Switzerland

Victor Carbune