Apple Inc.

Alkeshkumar M Patel

Saurabh Adya

Myra C Lukens

Jerome Rene Bellegarda

Aswath Manoharan

Srikanth Vishnubhotla

Lin Li

Xiaochuan Niu

Sameer Badaskar

Ahmed S Hussen Abdelaziz

Karan M Daryanani

Hong Yu

Jianpeng Cheng

Shruti Bhargava

Stephen G Pulman

Dhivya Piraviperumal

Akanksha Bindal

Systems and processes for operating a digital assistant are provided. An example process for determining a response includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, performing a semantic analysis on the spoken input, determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and in accordance with the determined likelihood exceeding a threshold, enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device.

Selectively using sensors for contextual data

Systems and processes for operating an intelligent automated assistant are provided. An example process includes receiving an utterance including a user request and determining whether at least a portion of the user request is ambiguous. If at least the portion of the user request is ambiguous then a set of context data based on the ambiguous portion of the user request is determined, metadata is extracted from the context data and a response to the user request is determined based on the extracted metadata.

Multimodal assistant understanding using on-screen and device context

Systems and processes for operating a digital assistant are provided. An example method for processing an image include receiving an image, generating, based on the image, a question corresponding to a first object in the image, generating, based on the image, a caption corresponding to a second object of the image, receiving an utterance from a user, and determining a plurality of speech recognition results from the utterance based on the question and the caption.

Using visual context to improve a virtual assistant

Systems and processes for operating a digital assistant are provided. An example process for performing a task includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, receiving an image input including a plurality of objects, selecting a reference resolution module of a plurality of reference resolution modules based on the request and the image input, determining, with the selected reference resolution module, whether the request references a first object of the plurality of objects based on at least the spoken input, and in accordance with a determination that the request references the first object of the plurality of objects, determining a response to the request including information about the first object.

Digital assistant reference resolution

Systems and processes for operating an intelligent automated assistant to provide a set of predicted responses are provided. An example method includes, at an electronic device having one or more processors, receiving one or more messages and analyzing the unstructured natural language information of the one or more messages. The method also includes determining, based on the analysis of the unstructured natural language information, whether one or more predicted responses are to be provided. The method further includes, in accordance with a determination that one or more predicted responses are to be provided, determining, from a plurality of sets of candidate predicted responses, one set of predicted responses to be provided to the user based on context information. The method further includes providing the determined set of one or more predicted responses to the user.

Context-aware ranking of intelligent response suggestions

Growing community of inventors

San Jose, CA, United States of America

Alkeshkumar M Patel