Google Inc.

David Roy Schairer

Yuzhao Ni

Mark Spates, Iv

Barnaby John James

Bo Wang

Petar Aleksic

Ulas Kirazci

Sunil Vemuri

Vincent Yanton Mo

Ertan Dogrultan

Di Lin

Cindy Tran

Zhenguo Guan

Vikram Aggarwal

Moises Morgenstern Gali

Haris Ramic

Chi Yeung Jonathan Ng

Steven Clark

Prem Kumar

Di Zhu

Sumer Mohammed

Yao Chen

Pravir Kumar Gupta

Jun Yang

Triona Butler

Lisa Williams

Ashwin Limaye

Thomas Clifton

Lei Zhong

Lucas Palmer

Manit Limlamai

Kyle Lund

Matthew Swartwout

Gaurav Nolkha

Stephen Lanham

Emil John Feig

Amy Lynn Baldwin

Iv Mark Spates

Alexei Sakhartchouk

Junjie Bu

Mark Spates Iv

Mark Iv Spates

Methods, apparatus, systems, and computer-readable media are provided for tailoring composite graphical assistant interfaces for interacting with multiple different connected devices. The composite graphical assistant interfaces can be generated in response to a user providing a request for an automated assistant to cause a connected device to perform a particular function. In response to the automated assistant receiving the request, the automated assistant can identify other functions that the connected device is capable of performing. The other functions can then be mapped to various graphical control elements in order to provide a composite graphical assistant interface from which the user can interact with the connected device. Each graphical control element can be arranged according to a status of the connected device, in order to reflect how the connected device is operating simultaneous to the presentation of the composite graphical assistant interface.

Methods, systems, and apparatus for providing composite graphical assistant interfaces for controlling connected devices

Efficient control and/or linking of smart network connected devices. Some implementations are directed to rendering, within an active application that enables control of multiple smart devices from multiple parties, both: (1) an interactive graphical interface element that, when interacted with, causes corresponding control of smart device(s) of a third-party (3P); and (2) a selectable element that, when selected, causes execution of a deeplink to cause a 3P application to open in a state that presents additional information for the smart device(s). Some implementations are additionally or alternatively directed to, in response to determining that a 3P application is installed at a client device, but a smart device topology lacks any entries for 3P smart devices of the 3P, efficiently adding one or more particular smart devices of the 3P to the smart device topology.

Efficient control and/or linking of smart devices

The present disclosure provides systems and methods that perform structure-based access control. In particular, rather than relying upon a user-specific credential scheme, which can require manual sharing of user-specific credentials and/or switching between the multiple accounts to access the particular devices, applications, or services associated with such accounts, the systems and methods of the present disclosure facilitate user credentials to be inherited by or otherwise assigned to a structure identifier associated with a structure (e.g., a home in which the user resides), thereby generating a set of structure credentials. This enables other users in the structure, who may be part of a collaborative user group, to access devices, applications, and/or services using the structure credentials.

Structure-based access control

Methods, apparatus, systems, and computer-readable media are provided for tailoring composite graphical assistant interfaces for interacting with multiple different connected devices. The composite graphical assistant interfaces can be generated in response to a user providing a request for an automated assistant to cause a connected device to perform a particular function. In response to the automated assistant receiving the request, the automated assistant can identify other connected devices, and other functions capable of being performed by the other connected devices. The other functions can then be mapped to various graphical control elements in order to provide a composite graphical assistant interface from which the user can interact with different connected devices. Each graphical control element can be arranged to reflect how each connected device is operating simultaneous to the presentation of the composite graphical assistant interface.

Providing composite graphical assistant interfaces for controlling various connected devices

Various implementations relate to techniques, for controlling smart devices, that are low latency and/or that provide computational efficiencies (client and/or server) and/or network efficiencies. Those implementations relate to generating and/or utilizing cache entries, of a cache that is stored locally at an assistant client device, in control of various smart devices (e.g., smart lights, smart thermostats, smart plugs, smart appliances, smart routers, etc.). Each of the cache entries includes a mapping of text to one or more corresponding semantic representations.

Efficient and low latency automated assistant control of smart devices

Implementations relate to dynamically, and in a context-sensitive manner, biasing voice to text conversion. In some implementations, the biasing of voice to text conversions is performed by a voice to text engine of a local agent, and the biasing is based at least in part on content provided to the local agent by a third-party (3P) agent that is in network communication with the local agent. In some of those implementations, the content includes contextual parameters that are provided by the 3P agent in combination with responsive content generated by the 3P agent during a dialog that: is between the 3P agent, and a user of a voice-enabled electronic device; and is facilitated by the local agent. The contextual parameters indicate potential feature(s) of further voice input that is to be provided in response to the responsive content generated by the 3P agent.

Voice to text conversion based on third-party agent content

The present disclosure is generally related to a data processing system to validate vehicular functions in a voice activated computer network environment. The data processing system can improve the efficiency of the network by discarding action data structures and requests that invalid prior to their transmission across the network. The system can invalidate requests by comparing attributes of a vehicular state to attributes of a request state.

Vehicle function control with sensor based validation

Remote automated assistant component(s) generate client device notification(s) based on a received IoT state change notification that indicates a change in at least one state associated with at least one IoT device. The generated client device notification(s) can each indicate the change in state associated with the at least one IoT device, and can optionally indicate the at least one IoT device. Further, the remote automated assistant component(s) can identify candidate assistant client devices that are associated with the at least one IoT device, and determine whether each of the one or more of the candidate assistant client device(s) should render a corresponding client device notification. The remote automated assistant component(s) can then transmit a corresponding command to each of the assistant client device(s) it determines should render a corresponding client device notification, where each transmitted command causes the corresponding assistant client device to render the corresponding client device notification.

Generating IoT-based notification(s) and provisioning of command(s) to cause automatic rendering of the IoT-based notification(s) by automated assistant client(s) of client device(s)

Implementing methods to provide a shortened textual summary that includes status information that is most pertinent to the user for a plurality of connected smart devices. The method includes determining a list of current statuses for a plurality of enabled smart devices and filtering the list to remove statuses that may not be of interest to the user. The filtering of the list is based on a current context of the requesting user and one or more previous contexts of the user. The resulting filtered statuses are then converted to textual snippets, summarized, and provided to the user via one or more output devices.

Summarily conveying smart appliance statuses

Methods, apparatus, systems, and computer-readable media are provided for tailoring composite graphical assistant interfaces for interacting with multiple different connected devices. The composite graphical assistant interfaces can be generated proactively and/or in response to a user providing a request for an automated assistant to cause a connected device to perform a particular function. In response to the automated assistant receiving the request, the automated assistant can identify other connected devices, and other functions capable of being performed by the other connected devices. The other functions can then be mapped to various graphical control elements in order to provide a composite graphical assistant interface from which the user can interact with different connected devices. Each graphical control element can be arranged to reflect how each connected device is operating simultaneous to the presentation of the composite graphical assistant interface.

Methods, apparatus, systems, and computer-readable media for engaging an automated assistant to perform multiple tasks through a multitask command. The multitask command can be a command that, when provided by a user, causes the automated assistant to invoke multiple different agent modules for performing tasks to complete the multitask command. During execution of the multitask command, a user can provide input that can be used by one or more agent modules to perform their respective tasks. Furthermore, feedback from one or more agent modules can be used by the automated assistant to dynamically alter tasks in order to more effectively use resources available during completion of the multitask command.

Invoking an automated assistant to perform multiple tasks through an individual command

In one example, a method includes receiving audio data generated by one or more microphones of a computing device, the audio data representing a spoken utterance; identifying, based on the audio data, a user that provided the spoken utterance; identifying, based on the audio data, an automation action associated with one or more automation devices, the automation action corresponding to the spoken utterance; determining whether the identified user is authorized to cause performance of the identified automation action; and responsive to determining that the identified user is authorized to cause performance of the identified automation action, causing the one or more automation devices to perform the identified automation action.

Multi-user virtual assistant for verbal device control

Growing community of inventors

San Jose, CA, United States of America

David Roy Schairer