Peloton Interactive, Inc.

Aiqudo, Inc

Mark Robinson

Rajat Mukherjee

Kiran Bindhu Hemaraj

Matan Levi

Michael Lai

Sunil Patil

Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Talk back from actions in applications

Embodiments of the present invention are directed to action recipes for a crowdsourced digital assistant. Users can define an action recipe by recording a set of inputs across one or more applications, by providing multiple sub-commands in a single on-the-fly command, by providing one or more associated commands, or otherwise. An action recipe dataset is generated, and stored and indexed on a user device and/or on an action cloud server. As such, any user can invoke an action recipe by providing an associated command to a crowdsourced digital assistant application on a user device. The crowdsourced digital assistant searches for a matching command on the user device and/or the action cloud server, and if a match is located, the corresponding action recipe dataset is accessed, and the crowdsourced digital assistant emulates the actions in the action recipe on the user device.

Action recipes for a crowdsourced digital assistant system

Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant system and related methods. In particular, embodiments facilitate techniques to crowdsource the training of a language model of the crowd-sourced digital assistant system. The digital assistant device can generate new action datasets based on manual inputs detected by the digital assistant device. The manual inputs can be recorded as a set of instructions, which can be interpreted by another digital assistant device to reproduce the detected manual inputs based on a command received by the other digital assistant device. The digital assistant server can receive action datasets, maintain action datasets, and distribute action datasets to one or more digital assistant devices. In various embodiments, the digital assistant device or server can also determine whether received action datasets are related. Each digital assistant device in the described system can participate in the training of a language model that establishes relationships between action datasets and/or command templates based on user feedback received via the digital assistant device.

Sharing commands and command groups across digital assistant operations

Various embodiments, methods and systems for implementing a digital assistant connectivity system are provided. In operation, a request to receive a unique identifier is communicated from a digital assistant device. The unique identifier is utilized to pair the digital assistant device with a smart assistant device. The unique identifier is received from and generated by a digital assistant server to correspond with the digital assistant device and a corresponding digital assistant device application instance. An instruction to perform an action on the digital assistant device is received at the digital assistant device application. The instruction is communicated based on an established command-driven session between the digital assistant device application and the smart assistant device. The command-driven session is associated with the unique identifier that paired the digital assistant device and the smart assistant device. The action is executed on the digital assistant device based on commands received by the smart assistant device.

Growing community of inventors

Union City, CA, United States of America

Mark Robinson