Google Inc.

Victor Carbune

Matthew Nirvan Sharifi

Thomas Deselaers

Daniel M Keysers

Sandro Feuz

Pedro Gonnet Anders

Luv Kothari

Jacopo Sannazzaro Natta

Nicolo D'Ercole

Justin Lu

Behshad Behzadi

Balint Miklos

Bogdan Prisacari

Domenico Carbotta

Jae Seo

Felix Weissenberger

Ray Chen

Kevin Fu

Fo Lee

Mucun Lu

Neha Garg

Barbara Poblocka

Matthew Miao

Thomas Qian

Alexandru-Marian Damian

Zaheed Sabur

Michael Golikov

Julia Proskurnia

Daniel Valcarce

Magali Boizot-Roche

Krishna Sapkota

Ondrej Skopek

Mohamad Hassan Rom

Henry Allan Rowley

Blaise Hilary Aguera-Arcas

Haroon Baig

Alvin Abdagic

Kevin I Kilgour

Arash Sadr

Kevin Kilgour

Andrii Maksai

Srikanth Pandiri

Viesturs Zarins

Marcos Calvo

Márius Sajgalík

Krzysztof Andrzej Goj

Kevin Allekotte

Marius Sajgalik

Paula Marques Fernandes

Implementations described herein relate to an application and/or automated assistant that can identify arrangement operations to perform for arranging text during speech-to-text operations—without a user having to expressly identify the arrangement operations. In some instances, a user that is dictating a document (e.g., an email, a text message, etc.) can provide a spoken utterance to an application in order to incorporate textual content. However, in some of these instances, certain corresponding arrangements are needed for the textual content in the document. The textual content that is derived from the spoken utterance can be arranged by the application based on an intent, vocalization features, and/or contextual features associated with the spoken utterance and/or a type of the application associated with the document, without the user expressly identifying the corresponding arrangements. In this way, the application can infer content arrangement operations from a spoken utterance that only specifies the textual content.

Arranging and/or clearing speech-to-text content without a user providing express instructions

Automated content switching rules may be generated and/or utilized for automatically switching away from certain interactive content during presentation of that interactive content when one or more switch conditions are met. In some instances, automated content switching rules may define one or more non-temporal switch conditions, e.g., based upon reaching certain points or milestones in certain interactive content, that may be used to initiate actions that switch away from the interactive content. In addition, in some instances, automated content switching rules may be used to not only switch away from particular interactive content, but additionally switch to other interactive content, thereby enabling a user to effectively schedule a workflow across different interactive content, applications and/or other computer-related tasks.

Automated content switching rule for controlling access to interactive content on a computing device

A method for sharing assistant profiles includes receiving, at a profile service, from an assistant service interacting with a user device of a user, a request requesting the profile service to release personal information associated with the user to the assistant service. The operations also include performing, through the assistant service, a verification process to verify that the user consents to releasing the requested personal information by: instructing the assistant service to prompt the user to recite a unique token prescribed to the user; receiving audio data characterizing a spoken utterance captured by the user device of the user; processing the audio data to determine whether a transcription of the spoken utterance recites the unique token; and when the transcription of the spoken utterance recites the unique token, releasing, to the assistant service, the requested personal information stored on a centralized data store managed by the profile service.

Shared assistant profiles verified via speaker identification

Implementations set forth herein relate to an automated assistant that can be accessible via a virtual environment for controlling features of the virtual environment and/or devices in a physical environment of the user. When the automated assistant is invoked, the automated assistant can materialize in the virtual environment according to any request that the automated assistant has been invoked to fulfill. For example, depending on the request from the user, the automated assistant can cause rendering of a virtual object for fulfilling the request and/or controlling an ongoing operation of the automated assistant. When the virtual object is rendered to control an operation of the automated assistant, or another application, the virtual object can include a virtual feature that the user can interact with to control the virtual environment and/or devices in a physical environment of the user.

Facilitating virtual or physical assistant interactions with virtual objects in a virtual environment

A method includes detecting a presence of multiple users within an environment of an assistant-enabled device (AED) and obtaining, for each user, a respective active set of warm words that each specify a respective action for a digital assistant to perform. Based on each respective active set of warm words, the method also includes executing a warm word arbitration routine to enable a final set of warm words for detection by the AED. Here, the final set of warm words includes warm words selected from the respective active set of warm words. While the final set of warm words are enabled, the method also includes receiving audio data corresponding to an utterance captured by the AED, detecting a warm word from the final set of warm words in the audio data, and instructing the digital assistant to perform the respective action specified by the detected warm word.

Multi-user warm words

Implementations set forth herein relate to an automated assistant that can employ various utilities according to whether certain conditions are satisfied for a given utility and/or whether a particular input is determined to have a threshold degree of relevance to the given utility. The automated assistant can be customized by a third party entity, which can make the automated assistant available via a device and/or application. The automated assistant can operate according to certain utilities that have been permitted by the third party entity. However, the third party entity can allow the automated assistant to employ other scalable utilities when a user input is determined to have a threshold degree of relevance and/or when usage of certain scalable utilities has not exceeded a threshold. The utility can refer to a machine learning model and/or other data that can be employed to resolve an input.

Condition dependent scalable utilities for an automated assistant

Techniques are described herein for identifying a failed hotword attempt. A method includes: receiving first audio data; processing the first audio data to generate a first predicted output; determining that the first predicted output satisfies a secondary threshold but does not satisfy a primary threshold; receiving second audio data; processing the second audio data to generate a second predicted output; determining that the second predicted output satisfies the secondary threshold but does not satisfy the primary threshold; in response to the first predicted output and the second predicted output satisfying the secondary threshold but not satisfying the primary threshold, and in response to the first spoken utterance and the second spoken utterance satisfying one or more temporal criteria relative to one another, identifying a failed hotword attempt; and in response to identifying the failed hotword attempt, providing a hint that is responsive to the failed hotword attempt.

Detecting near matches to a hotword or phrase

Implementations set forth herein relate to an automated assistant that can perform operations to revert various applications to prior states that the applications may have arrived at via certain user inputs. The user can provide a spoken utterance such as, 'undo,' in order to cause the automated assistant to identify a particular application that the user may want to affect with the “undo” command. When the particular application is identified, the automated assistant can identify one or more operations recently performed using the particular application. In some implementations, the automated assistant can provide the user with a variety of undo options in response to an “undo” command. For instance, the automated assistant can prompt the user to select one of a first cluster of operations and/or a second cluster of operations to be undone, and each cluster can refer to different operations.

Undoing application operation(s) via user interaction(s) with an automated assistant

A method includes receiving user queries directed toward a cloud-based assistant service. For each received user query directed toward the cloud-based assistant service, the method also includes extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The method also includes determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. The distilled model of the cloud-based assistant service is configured to execute on one or more target client devices.

Distilling to a target device based on observed query patterns

Implementations are directed to receiving a search query from a user, obtaining environmental signal(s) associated with an environment in which the user is located when the search query is received, processing the environmental signal(s) to generate a privacy measure associated with submission of the search query, obtaining additional environmental signal(s) associated with the environment in which the user is located when user input directed to a search interface is received, processing the additional environmental signal(s) to generate an additional privacy measure associated with the user input, selecting, from a superset of historical search queries of the user, a subset of the historical search queries based on at least the privacy measure and the additional privacy measure, and causing the subset of the historical search queries to be presented to the user in response to receiving the user input directed to the search interface.

Methods and systems for presenting privacy friendly query activity based on environmental signal(s)

Systems and methods enable a computing system to recognize a sequence of repeated actions and offer to automatically repeat any such recognized actions. An example method includes determining a current sequence of user actions is similar to previous sequence of user actions, determining whether the previous sequence is reproducible and, when reproducible, initiating display of a prompt that requests approval for completing the current sequence based on the previous sequence and, responsive to receiving an indication of approval, completing the previous sequence. Another example method included determining that a first current sequence of user interactions is complete and is not similar to any saved sequence of user interactions, saving the first current sequence as a previous sequence, identifying a second sequence as satisfying a similarity threshold with the previous sequence, and initiating display of a prompt that requests approval for saving the previous sequence as a shortcut.

Smart assist for repeated actions

A method for combining hotwords in a single utterance receives, at a first assistant-enabled device (AED), audio data corresponding to an utterance directed toward the first AED and a second AED among two or more AEDs where the audio data includes a query specifying an operation to perform. The method also detects, using a hotword detector, a first hotword assigned to the first AED that is different than a second hotword assigned to the second AED. In response to detecting the first hotword, the method initiates processing on the audio data to determine that the audio data includes a term preceding the query that at least partially matches the second hotword assigned. Based on the at least partial match, the method executes a collaboration routine to cause the first AED and the second AED to collaborate with one another to fulfill the query.

Combining device or assistant-specific hotwords in a single utterance

Implementations set forth herein relate to an automated assistant that can operate in a transient personalization mode, and/or assist a separate automated assistant with providing output according to a transient personalization mode. The transient personalization mode can allow a guest user of an assistant enabled-device to receive personalized responses from the assistant-enabled device—despite not being signed into the assistant-enabled device. A host automated assistant of the assistant-enabled device can securely communicate with a guest user's automated assistant through a backend process. In this way, input queries from the guest user to the host automated assistant can be personalized according to the guest automated assistant—without the guest user directly engaging with their own personal device.

Transient personalization mode for guest users of an automated assistant

Systems and methods for determining, based on invocation input that is common to multiple automated assistants, which automated assistant to invoke in lieu of invoking other automated assistants. The invocation input is processed to determine one or more invocation features that may be utilized to determine which, of a plurality of candidate automated assistants, to invoke. Further, additional features are processed that can indicate which, of the plurality of invocable automated assistants, to invoke. Once an automated assistant has been invoked, additional audio data and/or features of additional audio data are provided to the invoked automated assistant for further processing.

Selecting between multiple automated assistants based on invocation properties

A method is provided that can include determining, based on a plurality of media streams of a plurality of client devices participating in a video conference, a subject being discussed during the video conference at a point in time, and providing one or more inputs to a machine learning model, the one or more inputs comprising a plurality of transcriptions each corresponding to one of the plurality of media streams, and an indication of a request to identify one or more media streams that are relevant the subject. The method can further include determining, based on one or more outputs of the machine learning model, a media stream that is most relevant to the subject, and providing the determined media stream for presentation on one or more of the plurality of client devices while the subject is being discussed during the video conference.

Video conference content auto-retrieval and focus based on learned relevance

Growing community of inventors

Winterthur, Switzerland

Victor Carbune