Google Inc.

Jacek Szmigiel

Joseph Lange

Marcin M Nowak-Przygodzki

Gabriel Taubman

Ilya Firman

Adam Coimbra

James Stout

Thomas Weedon Hume

Abhanshu Sharma

Gökhan Bakir

Jindong Chen

Kyle Pedersen

Reed Enger

Vishwath Mohan

Yunfan Jin

Gilles Baechler

Implementations set forth herein relate to an automated assistant that can control graphical user interface (GUI) elements via voice input using natural language understanding of GUI content in order to resolve ambiguity and allow for condensed GUI voice input requests. When a user is accessing an application that is rendering various GUI elements at a display interface, the automated assistant can operate to process actionable data corresponding to the GUI elements. The actionable data can be processed in order to determine a correspondence between GUI voice input requests to the automated assistant and at least one of the GUI elements rendered at the display interface. When a particular spoken utterance from the user is determined to correspond to multiple GUI elements, an indication of ambiguity can be rendered at the display interface in order to encourage the user to provide a more specific spoken utterance.

Condensed spoken utterances for automated assistant control of an intricate application GUI

Implementations set forth herein relate to an automated assistant that can interact with applications that may not have been pre-configured for interfacing with the automated assistant. The automated assistant can identify content of an application interface of the application to determine synonymous terms that a user may speak when commanding the automated assistant to perform certain tasks. Speech processing operations employed by the automated assistant can be biased towards these synonymous terms when the user is accessing an application interface of the application. In some implementations, the synonymous terms can be identified in a responsive language of the automated assistant when the content of the application interface is being rendered in a different language. This can allow the automated assistant to operate as an interface between the user and certain applications that may not be rendering content in a native language of the user.

Automated assistant control of non-assistant applications via identification of synonymous term and/or speech processing biasing

Growing community of inventors

Zurich, Switzerland

Jacek Szmigiel