Google Inc.

Adobe, Inc.

Quandx Inc.

Yuan Yuan

Kenneth Edward Mixter

Tuan Anh Nguyen

C Philip Gossett

Carroll Philip Gossett

Michial Allen Gunter

Kevin Yu

David James Carmichael

Emmanouil Koukoumidis

Jeremy Thorpe

Shuyi Chen

Xiaofang Jiang

Johan Schalkwyk

Zhenyu Yan

Qiong Huang

Abhishek Pani

Bibo Xu

Tianyu Wang

Chen Dong

Gabriel Leblanc

Anurag Jain

William A Truong

Yixing Cai

Tzu-Chan Chuang

Alex Gaysinsky

Alexey Galata

Bruce J Richardson

Christopher D McKillop

Timothy Kilbourn

Ronan Hayes

Kareem Aladdin Nassar

Paul Fleming

Xiaojing Dong

Andrew James Edward Mutton

Yiwen Sun

Brian Lewis

Xiaojun Lei

Qian Xu

Manuel Roman Cuesta

Caroll Philip Gossett

Xiaofan Jiang

Hot word free adaptation, of function(s) of an automated assistant, responsive to determining, based on gaze measure(s) and/or active speech measure(s), that a user is engaging with the automated assistant. Implementations relate to techniques for mitigating false positive occurrences of and/or false negative occurrences, of hot word free adaptation, through utilization of a permissive parameter set in some situation(s) and a restrictive parameter set in other situation(s). For example, utilizing the restrictive parameter set when it is determined that a user is engaged in conversation with additional user(s). The permissive parameter set includes permissive parameter(s) that are more permissive than counterpart(s) in the restrictive parameter set. A parameter set is utilized in determining whether condition(s) are satisfied, where those condition(s), if satisfied, indicate that the user is engaging in hot word free interaction with the automated assistant and result in adaptation of function(s) of the automated assistant.

Dynamic adaptation of parameter set used in hot word free adaptation of automated assistant

Techniques are described herein for detecting and/or enrolling (or commissioning) new 'hot commands' that are useable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands. Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation.

Automated assistant interaction prediction using fusion of visual and audio input

Methods, apparatus, systems, and computer-readable media are provided for generating and/or adapting automated assistant content according to a distance of a user relative to an automated assistant interface that renders the automated assistant content. For instance, the automated assistant can provide data for a client device to render. The client device can request additional data when the user relocates closer to, or further from, the client device. In some implementations, a request for additional data can identify a distance between the user and the client device. In this way, the additional data can be generated or selected according to the distance in the request. Other implementations can allow an automated assistant to determine an active user from a group of users in an environment, and determine a distance between the active user and the client device in order that any rendered content can be tailored for the active user.

Generating and/or adapting automated assistant content according to a distance between user(s) and an automated assistant interface

Detection and/or enrollment of hot commands to trigger responsive action by automated assistant

A method at an electronic device with an array of indicator lights includes: obtaining first visual output instructions stored at the electronic device, where the first visual output instructions control operation of the array of indicator lights based on operating state of the electronic device; receiving a voice input; obtaining from a remote system a response to the voice input and second visual output instructions, where the second visual output instructions are provided by the remote system along with the response in accordance with a determination that the voice input satisfies one or more criteria; executing the response; and displaying visual output on the array of indicator lights in accordance with the second visual output instructions, where otherwise in absence of the second visual output instructions the electronic device displays visual output on the array of indicator lights in accordance with the first visual output instructions.

Server-provided visual output at a voice interface device

Techniques are described herein for detecting and/or enrolling (or commissioning) new 'hot commands' that are useable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation.

Techniques are described herein for detecting and/or enrolling (or commissioning) new 'hot commands' that are usable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands. Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation.

The various implementations described herein include methods, devices, and systems for attending to a presenting user. In one aspect, a method is performed at an electronic device that includes an image sensor, microphones, a display, processor(s), and memory. The device (1) obtains audio signals by concurrently receiving audio data at each microphone; (2) determines based on the obtained audio signals that a person is speaking in a vicinity of the device; (3) obtains video data from the image sensor; (4) determines via the video data that the person is not within a field of view of the image sensor; (5) reorients the electronic device based on differences in the received audio data; (6) after reorienting the electronic device, obtains second video data from the image sensor and determines that the person is within the field of view; and (7) attends to the person by directing the display toward the person.

Methods and systems for attending to a presenting user

Adapting an automated assistant based on detecting: movement of a mouth of a user; and/or that a gaze of the user is directed at an assistant device that provides an automated assistant interface (graphical and/or audible) of the automated assistant. The detecting of the mouth movement and/or the directed gaze can be based on processing of vision data from one or more vision components associated with the assistant device, such as a camera incorporated in the assistant device. The mouth movement that is detected can be movement that is indicative of a user (to whom the mouth belongs) speaking.

Adapting automated assistant based on detected mouth movement and/or gaze

Invoking one or more previously dormant functions of an automated assistant in response to detecting, based on processing of vision data from one or more vision components: (1) a particular gesture (e.g., of one or more 'invocation gestures') of a user; and/or (2) detecting that a gaze of the user is directed at an assistant device that provides an automated assistant interface (graphical and/or audible) of the automated assistant. For example, the previously dormant function(s) can be invoked in response to detecting the particular gesture, detecting that the gaze of the user is directed at an assistant device for at least a threshold amount of time, and optionally that the particular gesture and the directed gaze of the user co-occur or occur within a threshold temporal proximity of one another.

Invoking automated assistant function(s) based on detected gesture and gaze

Techniques are described herein for reducing false positives in vision sensor-equipped assistant devices. In various implementations, initial image frame(s) may be obtained from vision sensor(s) of an assistant device and analyzed to classify a particular region of the initial image frames as being likely to contain visual noise. Subsequent image frame(s) obtained from the vision sensor(s) may then be analyzed to detect actionable user-provided visual cue(s), in a manner that reduces or eliminates false positives. In some implementations, no analysis may be performed on the particular region of the subsequent image frame(s). Additionally or alternatively, in some implementations, a first candidate visual cue detected within the particular region may be weighted less heavily than a second candidate visual cue detected elsewhere in the one or more subsequent image frames. An automated assistant may then take responsive action based on the detected actionable visual cue(s).

Selective detection of visual cues for automated assistants

The present disclosure includes systems, methods, and non-transitory computer readable media that train and utilize multi-phase learning models to predict performance during digital content campaigns and provide digital content to client devices in a real-time bidding environment. In particular, one or more embodiments leverage organizational structure of digital content campaigns to train two learning models, utilizing different data sources, to predict performance, generate bid responses, and provide digital content to client devices. For example, the disclosed systems can train a first performance learning model in an offline mode utilizing parent-level historical data. Then, in an online mode, the disclosed systems can train a second performance learning model utilizing child-level historical data and utilize the first performance learning model and the second performance learning model to generate bid responses and bid amounts in a real-time bidding environment.

Training and utilizing multi-phase learning models to provide digital content to client devices in a real-time digital bidding environment

Growing community of inventors

Sunnyvale, CA, United States of America

Yuan Yuan