Apple Inc.

Devin William Chalmers

Rahul Nair

Thomas G Salter

Avi Bar-Zeev

Sean Boland Kelly

Luis R Deliz Centeno

Anshu Kameswar Chimalamarri

Gregory Patrick Lane Lutter

Giancarlo Yerkes

Ioana Negoita

Gregory Lutter

Brian Warren Temple

Bart Colin Trzynadlowski

Ryan S Burgoyne

Allison W Dryer

Jae Hwang Lee

Timothy Robert Oriol

Alexis Henri Palangie

William D Lindmeier

Anna L Brewer

Yutaka Yokokawa

Fletcher R Rothkopf

Christopher T Eubank

Lejing Wang

Karlin Y Bark

Grant H Mulliken

Golnaz Abdollahian

Samuel Lee Iglesias

Thomas J Moore

Kirill Kalinichev

Holly Gerhard

Rohit Sethi

Dhruv Aditya Govil

Jonathan C Moisant-Thompson

Bryce L Schmidtchen

Lilli I Jonsson

David H Y Huang

Ganghun Kim

Banafsheh Jalali

Stephen Olivier Lemay

Clarisse Mazuir

Giovanni M Agnoli

David E Kingman

Gregory A Cohoon

Budhadipta Dan

Yiqiang Nie

Avi Bar-zeev

Nahckjoon Kim

Elena J Nattinger

Peter Burgner

Jessica Trinh

Holly E Gerhard

In some embodiments, the present disclosure includes techniques and user interfaces for interacting with graphical user interfaces using gaze. In some embodiments, the present disclosure includes techniques and user interfaces for repositioning virtual objects. In some embodiments, the present disclosure includes techniques and user interfaces for transitioning modes of a camera capture user interface.

Devices, methods, for interacting with graphical user interfaces

In one implementation, a method for improving comfortability of virtual content during head roll motions includes: determining a scene roll origin value for an XR environment; detecting ahead pose with pose roll value; when a difference between the scene roll origin value and the pose roll value is less than a first angular threshold, presenting virtual content within the XR environment in a first presentation mode, wherein the virtual content is presented with a first content roll value in the first presentation mode based the pose roll value; and when the difference between the scene roll origin value and the pose roll value is not less than the first angular threshold, presenting the virtual content within the XR environment in a second presentation mode, wherein the virtual content is presented with a second content roll value in the second presentation mode based on the scene roll origin value.

Method and device for improving comfortability of virtual content

Various implementations disclosed herein include devices, systems, and methods for using a gaze vector and head pose information to effectuate a user interaction with a virtual object. In some implementations, a device includes a sensor for sensing a head pose of a user, a display, one or more processors, and a memory. In various implementations, a method includes displaying a set of virtual objects. Based on a gaze vector, it is determined that a gaze of the user is directed to a first virtual object of the set of virtual objects. A head pose value corresponding to the head pose of the user is obtained. An action relative to the first virtual object is performed based on the head pose value satisfying a head pose criterion.

Gaze and head pose interaction

Electronic devices provide extended reality experiences. In some embodiments, a media capture user interface is displayed, including a capture guide. In some embodiments, gaze information is used for targeting. In some embodiments, a virtual object is manipulated.

User interfaces for capturing media and manipulating virtual objects

Various implementations disclosed herein include devices, systems, and methods that present virtual content based on detecting a reflective object and determining a three-dimensional (3D) position of the reflective object in a physical environment. For example, an example process may include obtaining sensor data in a physical environment that includes one or more objects. The method may further include detecting a reflection of a first object of the one or more objects upon a reflective surface of a reflective object based on the sensor data. The method may further include determining a 3D position of the reflective object in the physical environment based on determining a 3D position of the reflection of the first object. The method may further include presenting virtual content in a view of the physical environment. The virtual content may be positioned at a 3D location based on the 3D position of the reflective object.

Content transformations based on reflective object recognition

Various implementations disclosed herein include devices, systems, and methods that determine that a user is interested in audio content by determining that a movement (e.g., a user's head bob) has a time-based relationship with detected audio content (e.g., the beat of music playing in the background). Some implementations involve obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment. A time-based relationship between one or more elements of the audio and one or more aspects of the body movement is identified based on the first sensor data and the second sensor data. An interest in content of the audio is identified based on identifying the time-based relationship. Various actions may be performed proactively based on identifying the interest in the content.

Proactive actions based on audio and body movement

In one implementation, a method for dynamically changing sensory and/or input modes associated with content based on a current contextual state. The method includes: while in a first contextual state, presenting extended reality (XR) content, via the display device, according to a first presentation N mode and enabling a first set of input modes to be directed to the XR content; detecting a change from the first contextual state to a second contextual state; and in response to detecting the change from the first contextual state to the second contextual state, presenting, via the display device, the XR content according to a second presentation mode different from the first presentation mode and enabling a second set of input modes to be directed to the XR content that are different from the first set of input modes.

Method and device for dynamic sensory and input modes based on contextual state

In an exemplary process for interacting with user interface objects using an eye gaze, a placement point for a selected object is designated at a first position based on a gaze position. In response to a user input, the placement point is moved to a second position that is not based on the gaze position, and the object is placed at the second position.

Gaze-based user interactions

In one implementation, a method of localizing a device is performed at a device including one or more processors and non-transitory memory. The method includes obtaining an estimate of a pose of the device in an environment. The method includes obtaining an environmental model of the environment including a spatial feature in the environment defined by a first spatial feature location. The method includes determining a second spatial feature location of the spatial feature based on the estimate of the pose of the device. The method includes determining an updated estimate of the pose of the device based on the first spatial feature location and the second spatial feature location.

Localization based on detected spatial features

A method includes, while presenting a representation of a physical environment of the device, obtaining contextual data that indicates a characteristic of the physical environment, the device or a user of the device, and obtaining an indication of historical content associated with the device. The method includes determining, based on the characteristic indicated by the contextual data and the historical content, a user interest value that indicates a user interest level in the physical environment. The method includes storing an image captured by an image sensor in response to the user interest value being greater than a threshold interest value.

Capturing and storing an image of a physical environment

A method performed by an audio system comprising a headset. The method sends a playback signal containing user-desired audio content to drive a speaker of the headset that is being worn by a user, receives a microphone signal from a microphone that is arranged to capture sounds within an ambient environment in which the user is located, performs a speech detection algorithm upon the microphone signal to detect speech contained therein, in response to a detection of speech, determines that the user intends to engage in a conversation with a person who is located within the ambient environment, and, in response to determining that the user intends to engage in the conversation, adjusts the playback signal based on the user-desired audio content.

Method and system for adjusting sound playback to account for speech detection

An electronic device is described. In some embodiments, the electronic device includes instructions for: while presenting an extended reality environment, receiving, by the first electronic device, a request to present a virtual representation of a remote participant of a communication session, where the first electronic device is connected to the communication session; obtaining a capability of the remote participant of the communication session; and in response to receiving the request to present the virtual representation of the remote participant of the communication session, presenting the virtual representation of the remote participant of the communication session based on the obtained capability of the remote participant of the communication session.

Environment for remote communication

In some embodiments, the present disclosure includes techniques and user interfaces for interacting with virtual objects in an extended reality environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects in an extended reality environment, including repositioning virtual objects relative to the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects, in an extended reality environment, including virtual objects that aid a user in navigating within the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects, including objects displayed based on changes in a field-of-view of a user, in an extended reality environment, including repositioning virtual objects relative to the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects in an extended reality environment, including virtual objects that provide a different perspective on the environment.

User interfaces that include representations of the environment

Various implementations disclosed herein include devices, systems, and methods for displaying a visual indication of audibility of an audible signal. In various implementations, a device includes a display, an audio sensor, a processor and a non-transitory memory. In various implementations, a method includes receiving, via the audio sensor, an audible signal and converting the audible signal to electronic signal data. In various implementations, the method includes obtaining environmental data that indicates audio response characteristics of a physical environment in which the device is located. In various implementations, the method includes displaying, on the display, an indicator that indicates a distance from a source of the audible signal at which the audible signal is audible. In some implementations, the distance is based on an amplitude associated with the electronic signal data and the audio response characteristics of the physical environment.

Visual indication of audibility

A method is performed at an electronic device with a one or more processors, a non-transitory memory, and a display. The method includes presenting, on the display, a plurality of objects including a first object and a second object. The method includes obtaining a first display priority value that is associated with the first object. The method includes prioritizing the first object over the second object based on a function of the first display priority value. The method includes, in response to determining that each of the first object and the second object satisfies an offscreen criterion, displaying, on the display, a first offscreen indicator that is associated with the first object according to the prioritization.

Displaying a prioritized offscreen indicator

Growing community of inventors

Oakland, CA, United States of America

Devin William Chalmers