Apple Inc.

Thomas G Salter

Devin William Chalmers

Anshu Kameswar Chimalamarri

Rahul Nair

Ioana Negoita

Paul Ewers

Brian Warren Temple

Gregory Patrick Lane Lutter

Thomas J Moore

Scott G Wade

Bart Colin Trzynadlowski

Christopher D Fu

Dhruv Aditya Govil

David W Padgett

Dimitris Ladopoulos

Christopher T Eubank

Anselm Grundhoefer

Mohamed Selim Ben Himane

Adeeti V Ullal

Arun Srivatsan Rangaprasad

Tom Sengelaub

Kirill Kalinichev

Jeffrey S Norris

Daniel M Trietsch

Ganghun Kim

Alexander G Bruno

James J Dunne

Matthew S DeMers

Giancarlo Yerkes

Giovanni M Agnoli

Allison W Dryer

Yiqiang Nie

Gregory Lutter

Bryce L Schmidtchen

Paulo R Jansen Dos Reis

Anna L Brewer

Yutaka Yokokawa

In Young Yang

Alesha Unpingco

Peter Burgner

Richard Ignatius Pusal Lozada

Ashwin K Vijay

Guilherme Klink

Tigran Khachatryan

Various implementations disclosed herein include devices, systems, and methods for using a gaze vector and head pose information to effectuate a user interaction with a virtual object. In some implementations, a device includes a sensor for sensing a head pose of a user, a display, one or more processors, and a memory. In various implementations, a method includes displaying a set of virtual objects. Based on a gaze vector, it is determined that a gaze of the user is directed to a first virtual object of the set of virtual objects. A head pose value corresponding to the head pose of the user is obtained. An action relative to the first virtual object is performed based on the head pose value satisfying a head pose criterion.

Gaze and head pose interaction

Various implementations disclosed herein include devices, systems, and methods that present virtual content based on detecting a reflective object and determining a three-dimensional (3D) position of the reflective object in a physical environment. For example, an example process may include obtaining sensor data in a physical environment that includes one or more objects. The method may further include detecting a reflection of a first object of the one or more objects upon a reflective surface of a reflective object based on the sensor data. The method may further include determining a 3D position of the reflective object in the physical environment based on determining a 3D position of the reflection of the first object. The method may further include presenting virtual content in a view of the physical environment. The virtual content may be positioned at a 3D location based on the 3D position of the reflective object.

Content transformations based on reflective object recognition

Various implementations disclosed herein include devices, systems, and methods that determine that a user is interested in audio content by determining that a movement (e.g., a user's head bob) has a time-based relationship with detected audio content (e.g., the beat of music playing in the background). Some implementations involve obtaining first sensor data and second sensor data corresponding to a physical environment, the first sensor data corresponding to audio in the physical environment and the second sensor data corresponding to a body movement in the physical environment. A time-based relationship between one or more elements of the audio and one or more aspects of the body movement is identified based on the first sensor data and the second sensor data. An interest in content of the audio is identified based on identifying the time-based relationship. Various actions may be performed proactively based on identifying the interest in the content.

Proactive actions based on audio and body movement

Various implementations disclosed herein include devices, systems, and methods for displaying presentation notes at varying positions within a presenter's field of view. In some implementations, a device includes a display, one or more processors, and a memory. A first portion of a media content item corresponding to a presentation is displayed at a first location in a three-dimensional environment. Audience engagement data corresponding to an engagement level of a member of an audience is received. A second portion of the media content item is displayed at a second location in the three-dimensional environment. The second location is selected based on the audience engagement data.

Audience engagement for placement of content in an HMD

In one implementation, a method of localizing a device is performed at a device including one or more processors and non-transitory memory. The method includes obtaining an estimate of a pose of the device in an environment. The method includes obtaining an environmental model of the environment including a spatial feature in the environment defined by a first spatial feature location. The method includes determining a second spatial feature location of the spatial feature based on the estimate of the pose of the device. The method includes determining an updated estimate of the pose of the device based on the first spatial feature location and the second spatial feature location.

Localization based on detected spatial features

A method includes: presenting a posture summary interface including: a representation of the user, a visualization of a current accumulated strain value for the user, and a first affordance for initiating an animated posture summary associated with the accumulated strain value for the user over a respective time window; and in response to detecting a user input directed to the first affordance within the posture summary interface, presenting an animation of the representation of the user over the respective time window that corresponds to one or more instances in which head pose information changes associated with the user caused an increase or a decrease to the accumulated strain value greater than a significance threshold, wherein an appearance of the visualization of the current accumulated strain value for the user changes to represent the accumulated strain value for the user over the respective time window.

Method and device for surfacing accumulated strain information

Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Visual marker with user selectable appearance

A method performed by an audio system comprising a headset. The method sends a playback signal containing user-desired audio content to drive a speaker of the headset that is being worn by a user, receives a microphone signal from a microphone that is arranged to capture sounds within an ambient environment in which the user is located, performs a speech detection algorithm upon the microphone signal to detect speech contained therein, in response to a detection of speech, determines that the user intends to engage in a conversation with a person who is located within the ambient environment, and, in response to determining that the user intends to engage in the conversation, adjusts the playback signal based on the user-desired audio content.

Method and system for adjusting sound playback to account for speech detection

A head-mounted device may be used to perform a visual search on a physical environment around the head-mounted device. A user may wish to visually search one out of multiple physical objects in the physical environment. To clearly show the user which physical object was the target of a visual search, the head-mounted device may present a thumbnail of the candidate physical object on a display. The thumbnail may be cropped and/or zoomed using an image from a camera on the head-mounted device. By displaying a thumbnail that is taken by a camera on the head-mounted device, the thumbnail will directly match the physical object in the user's physical environment, eliminating ambiguity regarding which physical object is the subject of the visual search.

Thumbnail for disambiguation in a visual search

In some embodiments, the present disclosure includes techniques and user interfaces for interacting with virtual objects in an extended reality environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects in an extended reality environment, including repositioning virtual objects relative to the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects, in an extended reality environment, including virtual objects that aid a user in navigating within the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects, including objects displayed based on changes in a field-of-view of a user, in an extended reality environment, including repositioning virtual objects relative to the environment. In some embodiments, the techniques and user interfaces are for interacting with virtual objects in an extended reality environment, including virtual objects that provide a different perspective on the environment.

User interfaces that include representations of the environment

Various implementations disclosed herein include devices, systems, and methods for augmenting a physical writing surface. In various implementations, a device includes a display, a non-transitory memory and one or more processors coupled with the display and the non-transitory memory. In various implementations, a method includes presenting, via the display, a pass-through representation of a physical writing surface that corresponds to an application installed on the device. In some implementations, the method includes detecting a difference between the physical writing surface and an electronic record stored in association with the application. In some implementations, the method includes overlaying an element on the pass-through representation of the physical writing surface based on the difference between the physical writing surface and the electronic record.

Augmenting a physical writing surface

A method includes obtaining an image of a machine-readable data representation that is located on a physical object using a camera of an electronic device. The machine-readable data representation includes an encoded form of a data value. The method further includes decoding the machine-readable data representation to determine the data value, whereby the data value includes a content identifier and a content source identifier. The method also includes selecting a content source based on the content source identifier, obtaining a content item and content location information based on the content identifier from the content source, determining a content position and a content orientation for the content item relative to the physical object based on the content location information, and displaying a representation of the content item using the electronic device according to the content position and the content orientation.

Locating content in an environment

A method is performed at an electronic device with a one or more processors, a non-transitory memory, and a display. The method includes presenting, on the display, a plurality of objects including a first object and a second object. The method includes obtaining a first display priority value that is associated with the first object. The method includes prioritizing the first object over the second object based on a function of the first display priority value. The method includes, in response to determining that each of the first object and the second object satisfies an offscreen criterion, displaying, on the display, a first offscreen indicator that is associated with the first object according to the prioritization.

Displaying a prioritized offscreen indicator

In one implementation, a method of authenticating a user is performed by a first device including a camera, an eye tracker, one or more processors, and non-transitory memory. The method includes obtaining, via the camera, an image of a physical environment. The method includes detecting, in the image of the physical environment, a second device. The method includes determining, using the eye tracker, a gaze of a user. The method includes determining that the gaze of the user is directed to the second device. The method includes, in response to determining that the gaze of the user is directed to the second device, transmitting authentication credentials to the second device.

Method and device for automatic authentication

In one implementation, a method for spatially designating private content. The method includes: presenting, via a display device, an indication of a private viewing region relative to a location of the computing system; determining a first location for presentation of graphical content; and presenting, via the display device, the graphical content at the first location. The method further includes: transmitting a characterization vector associated with the graphical content to at least one other device for display thereon according to a determination that the first location of the graphical content is outside of the private viewing area; and forgoing transmission of the characterization vector associated with the graphical content to the at least one other device according to a determination that the first location of the graphical content is inside of the private viewing area.

Method and device for spatially designating private content

Growing community of inventors

Foster City, CA, United States of America

Thomas G Salter