Google Inc.

Moises Morgenstern Gali

Vikram Aggarwal

Haris Ramic

Joseph Pirozzo

Alexandr Maltsev

Brandon Stuut

David Roy Schairer

Yao Chen

Hongjie Bai

Sumedha K Swamy

Shireesh Agrawal

Methods, systems, apparatus, including computer programs encoded on a computer storage medium, for a user device to learn offline voice actions. In one aspect, the method includes actions of detecting, by the user device, an utterance at a first time when the user device is connected to a server by a network, providing, by the user device, the utterance to the server using the network, receiving, by the user device and from the server, an update to the grammar of the user device, detecting, by the user device, a subsequent utterance of the utterance at a second time when the user device is not connected to the server by a network, and in response to detecting, by the user device, the subsequent utterance of the utterance at the second time, identifying, by the user device, an operation to perform based on (i) the subsequent utterance, and (ii) the updated grammar.

Learning offline voice commands based on usage of online voice commands

Methods, apparatus, systems, and computer-readable media are provided for resuming a partially completed action that is to be performed by an automated assistant. The action can require the automated assistant to prompt the user to provide information that the automated assistant can use to complete the action. During a dialog session in which the user is providing the information, an event can occur that interferes with the completion of the action. In response, the automated assistant can cause any information obtained during the dialog session to be stored locally, in order that the automated assistant can resume completing the action at a later time. For instance, the user can be prompted by the automated assistant to complete the action, or the user can independently invoke the automated assistant to complete the action at a time that is convenient for the user.

Systems, methods, and apparatuses for managing incomplete automated assistant actions

The present disclosure is generally related to a data processing system to selectively invoke applications for execution. A data processing system can receive an input audio signal and can parse the input audio signal to identify a command. The data processing system can identify a first functionality of a first digital assistant application hosted on the data processing system in the vehicle and a second functionality of a second digital assistant application accessible via a client device. The data processing system can determine that one of the first functionality or the second functionality supports the command. The data processing system can select one of the first digital assistant application or the second digital assistant application based on the determination. The data processing system invoke one of the first digital assistant application or the second digital assistant application based on the selection.

Multiple digital assistant coordination in vehicular environments

Dynamically adapting provision of notification output to reduce distractions and/or to mitigate usage of computational resources. In some implementations, an automated assistant application predicts a level of engagement for a user and determines, based on the predicted level of engagement (and optionally future predicted level(s) of engagement), provisioning (e.g., whether, when, and/or how) of output that is based on a received notification. For example, the automated assistant application can, based on predicted level(s) of engagement, determine whether to provide any output based on a received notification, determine whether to suppress provision of output that is based on the received notification (e.g., until a later time with a decreased predicted level of engagement), determine whether to provide output that is a condensed version of the received notification, determine whether to automatically respond to the notification, and/or select an output modality for providing output that is based on the received notification.

Dynamically adapting provision of notification output to reduce user distraction and/or mitigate usage of computational resources

The present disclosure is generally related to a data processing system to validate vehicular functions in a voice activated computer network environment. The data processing system can improve the efficiency of the network by discarding action data structures and requests that invalid prior to their transmission across the network. The system can invalidate requests by comparing attributes of a vehicular state to attributes of a request state.

Vehicle function control with sensor based validation

The present disclosure is generally related to systems and methods of interfacing among multiple applications in a networked computer environment. A data processing system can access a navigation application to retrieve point locations within a reference frame corresponding to a geographic region displayed in a viewport of the navigation application. Each point location can have an identifier. The data processing system can parse an input audio signal to identify a request and a referential word. The data processing system can identify a point location within the reference frame based on the referential word parsed from the input audio signal and the identifier for the point location. The data processing system can generate an action data structure including the point location identified. The data processing system can transmit the action data structure to the navigation application to initiate a navigation guidance process using the point location.

Interfacing between digital assistant applications and navigation applications

Methods, apparatus, systems, and computer-readable media are provided for isolating at least one device, from multiple devices in an environment, for being responsive to assistant invocations (e.g., spoken assistant invocations). A process for isolating a device can be initialized in response to a single instance of a spoken utterance, of a user, that is detected by multiple devices. One or more of the multiple devices can be caused to query the user regarding identifying a device to be isolated for receiving subsequent commands. The user can identify the device to be isolated by, for example, describing a unique identifier for the device. Unique identifiers can be generated by each device of the multiple devices and/or by a remote server device. The unique identifiers can be presented graphically and/or audibly to the user, and user interface input. Any device that is not identified can become temporarily unresponsive to certain commands, such as spoken invocation commands.

Isolating a device, from multiple devices in an environment, for being responsive to spoken assistant invocation(s)

Implementations set forth herein relate providing and/or detecting an automated assistant suppressing output for limiting responsiveness of an automated assistant that is available via another computing device. The output can be rendered as an inaudible tone in an environment, in order to pre-emptively suppress responses from automated assistants that may detect a forthcoming spoken utterance from a user. Limiting responsiveness in this way can preserve computational resources in environments, such as vehicles, in which multiple users may be carrying devices that are capable of being invoked via a spoken utterance. For instance, pre-emptively limiting responsiveness of other automated assistants can eliminate interruptions at devices that may other be inadvertently invoked as a result of detecting certain spoken utterances.

Pre-emptively limiting responsiveness of various assistant devices in an environment using an inaudible tone or other rendered output

Implementations relate to enabling of authorization of certain automated assistant functions via one or more modalities available within a vehicle. Implementations can eliminate wasting of computational and communication resources by at least allowing other users to authorize execution of certain input commands from a user, without requesting the user to re-submit the commands. The vehicle can include a computing device that provides access to restricted data, which can be accessed in order for an action to be performed by the automated assistant. However, when a restricted user requests that the automated assistant perform an action involving accessing the restricted data, the automated assistant can be authorized or unauthorized to proceed with fulfilling the request via a modality controlled by an unrestricted user. The unrestricted user can also cause contextual restrictions to be established for limiting functionality of the automated assistant during a trip, for certain types of requests, and/or for certain passengers.

Modalities for authorizing access when operating an automated assistant enabled vehicle

Implementations relate to interactions, over a wireless communication modality, between an automated assistant of an automobile computer system, and separate client device(s). In some of those implementations, the client device(s) can be paired with the automobile computer system, and the client device(s) and the automobile computer system can include separate instances of an automated assistant, which can be associated with different user accounts. In some additional or alternative implementations, particular actions to be performed by a local application of a client device can be advanced via user interaction with the automobile automated assistant, despite a counterpart to the local application not being installed directly on automobile computer system. For example, despite an automobile computer system not having a third-party messaging application installed, the automobile automated assistant can access requested actions to be performed by the third-party messaging application in order to further a requested action. The automobile computer system can transmit, via the wireless communication modality, content to cause the third-party application to further the requested action.

Growing community of inventors

San Francisco, CA, United States of America

Moises Morgenstern Gali