International Business Machines Corporation

Peloton Interactive, Inc.

Google Inc.

Yahoo! Inc.

Aiqudo, Inc

Samsung Electronics Co., Ltd.

Verity, Inc.

Rajat Mukherjee

Kiran Bindhu Hemaraj

Mark Robinson

Daniel Manuel Dias

Abhinandan Sujit Das

Matan Levi

Amit Kumar

Dan Moisa

Jennifer Liu Low

Jianchang Mao

Prabhakar Raghavan

Adam Westall

J Eric Baldeschwieler

Arun Kwangil Iyengar

Murthy V Devarakonda

Hidayatullah Habeebullah Shaikh

Kalyan Desineni

Manikandan Sankaranarasimhan

Richard Pervin King

German Sergio Goldszmidt

Tomer Kagan

Guerney Douglass Holloway Hunt

Joseph Phillip Nelson

Michel A Tourn

Rajesh Konda

Kalpana Ravinarayanan

Irfan Presswala

Andrew Pariser

Kevin Patrick Thill

Bjoern Heckel

Mani Abrol

Sunil Patil

Panayiotis Tsaparas

Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Talk back from actions in applications

Embodiments of the present invention are directed to action recipes for a crowdsourced digital assistant. Users can define an action recipe by recording a set of inputs across one or more applications, by providing multiple sub-commands in a single on-the-fly command, by providing one or more associated commands, or otherwise. An action recipe dataset is generated, and stored and indexed on a user device and/or on an action cloud server. As such, any user can invoke an action recipe by providing an associated command to a crowdsourced digital assistant application on a user device. The crowdsourced digital assistant searches for a matching command on the user device and/or the action cloud server, and if a match is located, the corresponding action recipe dataset is accessed, and the crowdsourced digital assistant emulates the actions in the action recipe on the user device.

Action recipes for a crowdsourced digital assistant system

Embodiments of the present invention provide systems, methods, and computer storage media directed to optimizing engagement with a display during digital assistant-performed operations in response to a received command. The digital assistant generates an overlay having user interface elements that present information determined to be relevant to a user based on the received command and contextual data. The overlay is presented over the underlying operations performed on corresponding applications to mask the visible steps of the operations being performed. In this way, the digital assistant optimizes display resources that are typically rendered useless during the processing of digital assistant-performed operations.

Optimizing display engagement in action automation

Embodiments facilitate the intuitive creation, maintenance, and distribution of action datasets that include computing events or tasks that can be reproduced when a command is received by a digital assistant. The digital assistant can generate new action datasets, on-board new action datasets, and receive new action datasets or updates to existing action datasets locally or via a digital assistant server, among other things. The digital assistant server can also receive action datasets, maintain action datasets, and distribute action datasets to one or more digital assistants, among other things.

Systems and methods for crowdsourced actions and commands

Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant and system. In particular, embodiments facilitate the intuitive creation and distribution of action datasets that include computing events or tasks that can be reproduced when an associated command, stored in an action dataset, is determined received by a digital assistant device. The digital assistant device described herein can generate new action datasets, on-board new action datasets, and receive new action datasets or updates to existing action datasets. Each digital assistant device in the described system can participate in the building of action datasets, so as to crowd-source a dialect that can be understood by a digital assistant device.

Crowdsourced on-boarding of digital assistant operations

Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant system and related methods. In particular, embodiments facilitate the intuitive creation, maintenance, and distribution of action datasets that include computing events or tasks that can be reproduced when an associated command, stored in an action dataset, is determined received by a digital assistant device. The digital assistant device described herein can generate new action datasets, on-board new action datasets to a remote server, and receive new action datasets or updates to existing action datasets from the remote server. The digital assistant server described herein can receive action datasets, maintain action datasets, and distribute action datasets to one or more digital assistant devices. Each digital assistant device in the described system can participate in the generation of action datasets, so as to crowd-source various dialects that can be understood by any other digital assistant device.

Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant system and techniques for disambiguating commands based on personalized usage of a digital assistant device, among other things. In various embodiments, the digital assistant device can use personal data, collected device usage data, and other types of collected contextual information, to disambiguate received commands for the proper selection and execution of operations on the digital assistant device. The digital assistant can process and interpret ambiguous commands and even unique user dialects without requiring extensive training to recognize and act on the received commands, even if the particular phraseology of the command has not previously been encountered by the digital assistant.

Personalized digital assistant device and related methods

Various embodiments, methods and systems for implementing a digital assistant connectivity system are provided. In operation, a request to receive a unique identifier is communicated from a digital assistant device. The unique identifier is utilized to pair the digital assistant device with a smart assistant device. The unique identifier is received from and generated by a digital assistant server to correspond with the digital assistant device and a corresponding digital assistant device application instance. An instruction to perform an action on the digital assistant device is received at the digital assistant device application. The instruction is communicated based on an established command-driven session between the digital assistant device application and the smart assistant device. The command-driven session is associated with the unique identifier that paired the digital assistant device and the smart assistant device. The action is executed on the digital assistant device based on commands received by the smart assistant device.

Connecting multiple mobile devices to a smart home assistant account

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for related search suggestions for products. In one aspect, a method includes receiving a query for searching documents wherein the query specifies a first product. A number of occurrences of the first product in a plurality of prior queries of the documents submitted by users is determined. A product category for the first product is determined based on one or more attributes associated with the first product, the category having a predetermined plurality of category attributes. A suggested product is selected wherein the suggested product is associated with attributes that satisfy the category attributes.

Method, system, and computer readable medium for rendering a graphical user interface visually indicating search results including related suggested products

According to some implementations of the present disclosure, a method is provided. The method includes presenting a state of a first application and obtaining state indication data corresponding to the state of the first application. The method further includes transmitting the state indication data to a card system and receiving an availability indicator indicating whether the card system can provide a card corresponding to the state of the first application. The method includes displaying a first selection element overlaying a graphical user interface of the first application when the availability indicator indicates that the card system can provide the card and displaying a second selection element overlaying a graphical user interface of the first application when the availability indicator indicates that the card system cannot provide the card.

Sharing cards to a target application

A method includes executing a messaging application to generate a messaging session between a user device and an application contact that represents a communication channel with a business's application. The method further includes requesting historical items from one or more remote servers for insertion into the messaging session. The historical items include links that were generated by the remote servers and selected by a user prior to creation of the messaging session. The links are associated with the business's application. The method further includes receiving historical items from one or more of the remote servers, inserting the received historical items into the messaging session, and receiving user selection of one of the historical items. Additionally, the method includes launching an application according to an access mechanism of the selected historical item and setting the launched application state into a state specified by the access mechanism of the historical item.

Enhancing messaging application functionality

Growing community of inventors

San Jose, CA, United States of America

Rajat Mukherjee