Google Inc.

Luv Kothari

Behshad Behzadi

Zaheed Sabur

Victor Carbune

Steve Cheng

Jacopo Sannazzaro Natta

Domenico Carbotta

Bryan Christopher Horling

Lucas Mirelmann

Daniel Cotting

Denis Burakov

Sergey Nazarov

Michael Golikov

Bohdan Vlasyuk

Mario Bertschler

Lan Huo

Matthew Nirvan Sharifi

Ulas Kirazci

Jason Brant Douglas

Ilya Firman

Balint Miklos

Bogdan Prisacari

Nicolo D'Ercole

Carey Radebaugh

Julia Proskurnia

Jae Seo

Felix Weissenberger

Neha Garg

Barbara Poblocka

Matthew Miao

Yang Lu

Xu Liu

Gokay Baris Gultekin

Angana Ghosh

Akshay Kannan

Alvin Abdagic

Sergey Igorevich Nazarov

Matthew Sharifi

Thomas Qian

Srikanth Pandiri

Kevin Fu

Fo Lee

Mucun Lu

Ray Chen

Justin Lu

Qi Wang

Krishna Sapkota

Magali Boizot-Roche

Márius Sajgalík

Viesturs Zarins

Krzysztof Andrzej Goj

Implementations described herein relate to an application and/or automated assistant that can identify arrangement operations to perform for arranging text during speech-to-text operations—without a user having to expressly identify the arrangement operations. In some instances, a user that is dictating a document (e.g., an email, a text message, etc.) can provide a spoken utterance to an application in order to incorporate textual content. However, in some of these instances, certain corresponding arrangements are needed for the textual content in the document. The textual content that is derived from the spoken utterance can be arranged by the application based on an intent, vocalization features, and/or contextual features associated with the spoken utterance and/or a type of the application associated with the document, without the user expressly identifying the corresponding arrangements. In this way, the application can infer content arrangement operations from a spoken utterance that only specifies the textual content.

Arranging and/or clearing speech-to-text content without a user providing express instructions

Systems and methods of invoking functions of agents via digital assistant applications are provided. Each action-inventory can have an address template for an action by an agent. The address template can include a portion having an input variable used to execute the action. A data processing system can parse an input audio signal from a client device to identify a request and a parameter to be executed by the agent. The data processing system can select an action-inventory for the action corresponding to the request. The data processing system can generate, using the address template, an address. The address can include a substring having the parameter used to control execution of the action. The data processing system can direct an action data structure including the address to the agent to cause the agent to execute the action and to provide output for presentation.

Invoking functions of agents via digital assistant applications using address templates

Implementations can reduce the time required to obtain responses from an automated assistant through proactive caching, locally at a client device, of proactive assistant cache entries—and through on-device utilization of the proactive assistant cache entries. Different proactive cache entries can be provided to different client devices, and various implementations relate to technique(s) utilized in determining which proactive cache entries to provide to which client devices. In some of those implementations, in determining which proactive cache entries to provide (proactively or in response to a request) to a given client device, a remote system selects, from a superset of candidate proactive cache entries, a subset of the cache entries for providing to the given client device.

Proactive caching of assistant action content to enable resolution of spoken or typed utterances

Systems and methods described herein relate to determining whether to incorporate recognized text, that corresponds to a spoken utterance of a user of a client device, into a transcription displayed at the client device, or to cause an assistant command, that is associated with the transcription and that is based on the recognized text, to be performed by an automated assistant implemented by the client device. The spoken utterance is received during a dictation session between the user and the automated assistant. Implementations can process, using automatic speech recognition model(s), audio data that captures the spoken utterance to generate the recognized text. Further, implementations can determine whether to incorporate the recognized text into the transcription or cause the assistant command to be performed based on touch input being directed to the transcription, a state of the transcription, and/or audio-based characteristic(s) of the spoken utterance.

Voice commands for an automated assistant utilized in smart dictation

Implementations set forth herein relate to an automated assistant that can selectively determine whether to incorporate a verbatim interpretation of portions spoken utterances into an entry field and/or incorporate synonymous content into the entry field. For instance, a user can be accessing an interface that provides an entry field (e.g., address field) for receiving user input. In order to provide input for entry field, the user can select the entry field and/or access a GUI keyboard to initialize an automated assistant for assisting with filling the entry field. Should the user provide a spoken utterance, the user can elect to provide a spoken utterance that embodies the intended input (e.g., an actual address) or a reference to the intended input (e.g., a name). In response to the spoken utterance, the automated assistant can fill the entry field with the intended input without necessitating further input from the user.

Voice-controlled entry of content into graphical user interfaces

Implementations set forth herein relate to an automated assistant that can control a camera according to one or more conditions specified by a user. A condition can be satisfied when, for example, the automated assistant detects a particular environment feature is apparent. In this way, the user can rely on the automated assistant to identify and capture certain moments without necessarily requiring the user to constantly monitor a viewing window of the camera. In some implementations, a condition for the automated assistant to capture media data can be based on application data and/or other contextual data that is associated with the automated assistant. For instance, a relationship between content in a camera viewing window and other content of an application interface can be a condition upon which the automated assistant captures certain media data using a camera.

Conditional camera control via automated assistant commands

Implementations can reduce the time required to obtain responses from an automated assistant through proactive caching, locally at a client device, of proactive assistant cache entries&#x2014;and through on-device utilization of the proactive assistant cache entries. Different proactive cache entries can be provided to different client devices, and various implementations relate to technique(s) utilized in determining which proactive cache entries to provide to which client devices. In some of those implementations, in determining which proactive cache entries to provide (proactively or in response to a request) to a given client device, a remote system selects, from a superset of candidate proactive cache entries, a subset of the cache entries for providing to the given client device.

Proactive caching of assistant action content at a client device to enable on-device resolution of spoken or typed utterances

Growing community of inventors

Sunnyvale, CA, United States of America

Luv Kothari