Apple Inc.

Myra C Lukens

Aswath Manoharan

Saurabh Adya

Alkeshkumar M Patel

Karan M Daryanani

Stephen Olivier Lemay

Pedro Mari

Jay Moon

Sreeneel K Maddika

Chaitanya Mannemala

Keith P Avery

Lynn I Streja

Kisun You

Lin Li

Abhishek Rawat

Shehryar Lasi

Hong Yu

Jianpeng Cheng

Shruti Bhargava

Stephen G Pulman

Dhivya Piraviperumal

Systems and processes for operating a digital assistant are provided. An example process for determining a response includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, performing a semantic analysis on the spoken input, determining, based on the semantic analysis, a likelihood that the electronic device requires additional contextual data to satisfy the request, and in accordance with the determined likelihood exceeding a threshold, enabling a camera of the electronic device and determining a response to the request based on data captured by the camera of the electronic device.

Selectively using sensors for contextual data

An example process includes: while displaying a portion of an extended reality (XR) environment representing a current field of view of a user: detecting a user gaze at a first object displayed in the XR environment, where the first object is persistent in the current field of view of the XR environment; in response to detecting the user gaze at the first object, expanding the first object into a list of objects including a second object representing a digital assistant; detecting a user gaze at the second object; in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a digital assistant session is initiated; receiving a first audio input from the user; and displaying a second animation of the second object indicating that the digital assistant is actively listening to the user.

Extended reality based digital assistant interactions

Systems and processes for operating an intelligent automated assistant are provided. An example process includes receiving an utterance including a user request and determining whether at least a portion of the user request is ambiguous. If at least the portion of the user request is ambiguous then a set of context data based on the ambiguous portion of the user request is determined, metadata is extracted from the context data and a response to the user request is determined based on the extracted metadata.

Multimodal assistant understanding using on-screen and device context

Systems and processes for operating a digital assistant are provided. An example process for performing a task includes, at an electronic device having one or more processors and memory, receiving a spoken input including a request, receiving an image input including a plurality of objects, selecting a reference resolution module of a plurality of reference resolution modules based on the request and the image input, determining, with the selected reference resolution module, whether the request references a first object of the plurality of objects based on at least the spoken input, and in accordance with a determination that the request references the first object of the plurality of objects, determining a response to the request including information about the first object.

Digital assistant reference resolution

An appliance can include a microphone transducer configured to receive sound from an environment and to convert the received sound into an audio signal. The appliance can be configured to receive an input audio signal via the microphone transducer, responsive to an activation input. The appliance can be configured to initiate a bi-directional, wireless communication connection with an external device in parallel with receiving the input audio signal, and responsive to the activation input. The appliance can be configured to transmit an output audio signal corresponding to the input audio signal over the communication connection when the wireless communication connection is established. Related aspects also are described.

Hybrid routing for hands-free voice assistant, and related systems and methods

Growing community of inventors

San Francisco, CA, United States of America

Myra C Lukens