Google Inc.

Kenneth Edward Mixter

Tuan Anh Nguyen

Yuan Yuan

Diego Melendo Casado

Matthew Nirvan Sharifi

Alexander Houston Gruenstein

Christopher Thaddeus Hughes

Terry Tai

Daniel Colish

Raunaq Shah

Tomer Shekel

Bibo Xu

Johan Schalkwyk

William Alexander Drewry

Sumit Gwalani

Marci Meingast

Byungchul Kim

Jennifer Yick

Ken MacKay

Hideaki Matsui

Caroline Rebecca Pantofaru

Jaclyn Konzelmann

Sourish Chaudhuri

Vinay Bettadapura

Glenn Wilson

Mattias Stefan Nissler

Ashton Udall

Darin S Petkov

Joao Paulo Pagaime Da Silva

Raymond Lo

Bin Zhao

Caroline Pantofaru

Byungchui Kim

Ken Mackay

During a sleep session of a user of a display assistant device, the device records sound, movement, and ambient conditions in proximity to the device. The ambient conditions include a light level. The device analyzes the recorded sound and movement to identify throughout the sleep session of the user a time-varying sleep state of the user. The sleep state is characterized by a sleep quality and a sleep phase. The device also analyzes the recorded ambient conditions and the recorded sound throughout the sleep session to identify a plurality of time-varying sleep disturbances occurring during the sleep session of the user. After the sleep session of the user has concluded, the device presents on a screen of the device a sleep summary of the sleep session. The sleep summary includes visual indications of the sleep quality and disturbances identified throughout the sleep session.

Using ambient light sensors and ambient audio sensors to determine sleep quality

An electronic device stores a voice assistant library for execution on the electronic device based on the electronic device having a first device type. The electronic device receives a verbal input from a user. It extracts request information from the verbal input by processing the verbal input using the voice assistant library executing on the device. It transmits a request to a remote system. The electronic device receives a response to the request. The response is generated by the remote system. The electronic device performs an operation in accordance with the response by one or more voice-processing modules of the configured voice assistant library.

Implementations for voice assistant on devices

Systems and techniques are provided for distributed ambient computing within an environment. A first version of an ambient computing library running on a device may check a manifest in the storage of the device to identify a second device that provides a service. The first version of the ambient computing library may send data to be processed using the service to the second device through a second version of the ambient computing library running on the second device. The first device and the second device may include different computing hardware. The first version of the ambient computing library may receive data including results from the second device through the second version of the ambient computing library. The data including the results may be generated by the second device based on processing the data sent from the device by the first version of the ambient computing library.

Distributed ambient computing within an environment

A method for proactive notifications in a voice interface device includes: receiving a first user voice request for an action with an future performance time; assigning the first user voice request to a voice assistant service for performance; subsequent to the receiving, receiving a second user voice request and in response to the second user voice request initiating a conversation with the user; and during the conversation: receiving a notification from the voice assistant service of performance of the action; triggering a first audible announcement to the user to indicate a transition from the conversation and interrupting the conversation; triggering a second audible announcement to the user to indicate performance of the action; and triggering a third audible announcement to the user to indicate a transition back to the conversation and rejoining the conversation.

Conversation-aware proactive notifications for a voice interface device

Methods, apparatus, systems, and computer-readable media are provided for generating and/or adapting automated assistant content according to a distance of a user relative to an automated assistant interface that renders the automated assistant content. For instance, the automated assistant can provide data for a client device to render. The client device can request additional data when the user relocates closer to, or further from, the client device. In some implementations, a request for additional data can identify a distance between the user and the client device. In this way, the additional data can be generated or selected according to the distance in the request. Other implementations can allow an automated assistant to determine an active user from a group of users in an environment, and determine a distance between the active user and the client device in order that any rendered content can be tailored for the active user.

Generating and/or adapting automated assistant content according to a distance between user(s) and an automated assistant interface

The various implementations described herein include methods and systems for determining device leadership among voice interface devices. In one aspect, a method is performed at a first electronic device of a plurality of electronic devices, each having microphones, a speaker, processors, and memory storing programs for execution by the processors. The first device detects a voice input. It determines a device state and a relevance of the voice input. It identifies a subset of electronic devices from the plurality to which the voice input is relevant. In accordance with a determination that the subset includes the first device, the first device determines a first score of a criterion associated with the voice input and receives second scores of the criterion from other devices in the subset. In accordance with a determination that the first score is higher than the second scores, the first device responds to the detected input.

Device leadership negotiation among voice interface devices

A first electronic device of a local group of connected electronic devices receives a first voice command including a request for a first operation assigns a first target device from among a local group of connected electronic devices as an in-focus device for performing the first operation, causes the first operation to be performed by the first target device via operation of a server-implemented common network service, receives a second voice command including a request for a second operation and based on a determination that the second voice command does not include an explicit designation of a second target device and a determination that the second operation can be performed by the first target device, assigning the first target device.

Focus session at a voice interface device

A method at an electronic device with an array of indicator lights includes: obtaining first visual output instructions stored at the electronic device, where the first visual output instructions control operation of the array of indicator lights based on operating state of the electronic device; receiving a voice input; obtaining from a remote system a response to the voice input and second visual output instructions, where the second visual output instructions are provided by the remote system along with the response in accordance with a determination that the voice input satisfies one or more criteria; executing the response; and displaying visual output on the array of indicator lights in accordance with the second visual output instructions, where otherwise in absence of the second visual output instructions the electronic device displays visual output on the array of indicator lights in accordance with the first visual output instructions.

Server-provided visual output at a voice interface device

An electronic device configures a device-agnostic voice assistant library for execution on the electronic device based on the electronic device having a first device type. The electronic device also selects an implementation for the voice assistant library. After the configuring, the electronic device receives a verbal input from a user. It extracts request information from the verbal input by processing the verbal input using the voice assistant library executing on the device. It transmits a request to a remote system, the request including the extracted request information. The electronic device receives a response to the request. The response is generated by the remote system in accordance with the extracted request information. The electronic device performs an operation in accordance with the response by one or more voice processing modules of the configured voice assistant library.

A method at an electronic device with one or more microphones and a speaker, the electronic device configured to be responsive to any of a plurality of affordances including a voice-based affordance, includes determining background noise of an environment associated with the electronic device, and before detecting the voice-based affordance: determining whether the background noise would interfere with recognition of the hotword in voice inputs detected by the electronic device, and if so, indicating to a user to use an affordance other than the voice-based affordance.

Noise mitigation for a voice interface device

The various implementations described herein include methods, devices, and systems for attending to a presenting user. In one aspect, a method is performed at an electronic device that includes an image sensor, microphones, a display, processor(s), and memory. The device (1) obtains audio signals by concurrently receiving audio data at each microphone; (2) determines based on the obtained audio signals that a person is speaking in a vicinity of the device; (3) obtains video data from the image sensor; (4) determines via the video data that the person is not within a field of view of the image sensor; (5) reorients the electronic device based on differences in the received audio data; (6) after reorienting the electronic device, obtains second video data from the image sensor and determines that the person is within the field of view; and (7) attends to the person by directing the display toward the person.

Methods and systems for attending to a presenting user

Growing community of inventors

Santa Clara, CA, United States of America

Kenneth Edward Mixter