Apple Inc.

Other

Anselm Grundhoefer

Arun Srivatsan Rangaprasad

Ralf Habel

Jean-Charles Bernard Marcel Bazin

Sivalogeswaran Ratnasingam

Mohamed Selim Ben Himane

Ivan Kolesov

Daniel R Kurz

Moinul H Khan

Aleksandr M Movshovich

Geoffrey Grant Stahl

Thomas G Salter

Michael J Rockwell

Eshan Verma

Thomas J Moore

Yohan Rajan

Tom Sengelaub

Paolo Di Febbo

Jeffrey S Norris

Chaminda Nalaka Vidanagamachchi

Pedro Manuel Da Silva Quelhas

Bart Colin Trzynadlowski

Daniele Casaburo

Paul Ewers

Omar Elafifi

Siddharth Shankar Hazra

Marwan Hussein

Scott G Wade

Shubham Agrawal

Rahul Nair

Tushar Gupta

Dhruv Aditya Govil

Yue Ma

Shuang Wang

Joseph M Luxton

Phillip N Smith

Dhruv A Govil

Various implementations disclosed herein include devices, systems, and methods that provide point identification techniques for electronic devices such as optical see-through head mounted devices. In some implementations, a line of sight technique is used to identify a 3D position of a point. In some implementations, a touching technique is used to identify a 3D position of a point. In some implementations, different point identification techniques are automatically selected and used to identify a 3D position of a point. In some implementations, a 3D position of a point is associated with user input. In some implementations, a 3D position of a point is identified to determine distances, surface areas, or volumes.

Three-dimensional point selection

In accordance with some implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a see-through display. The method includes determining a plurality of light superposition characteristic values associated with ambient light from a physical environment. The plurality of light superposition characteristic values quantifies the ambient light. The method includes modifying image data in order to generate modified image data, based on a function of the plurality of light superposition characteristic values and a reference perceptual gamut. The method includes transforming the modified image data into display data based on a function of a portion of the plurality of light superposition characteristic values and a reference physical gamut that is associated with the see-through display. The method includes displaying the display data on the see-through display.

Color correction pipeline

Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Visual marker with user selectable appearance

Various implementations disclosed herein include devices, systems, and methods that are capable of adjusting the appearance of computer-generated objects (e.g., augmentations) that are predicted to be outside the current focus of an optical-see-through HMD user's attention. In some implementations, a method includes displaying a first computer-generated object and a second computer-generated object on an optical-see-though display of a HMD. In some implementations, the first computer-generated object is identified based on a prediction that the attention of a user is directed to the first computer-generated object, and an appearance of the second computer-generated object is adjusted to change from a first state to a second state.

Attention-driven rendering for computer-generated objects

A method includes sensing a plurality of light superposition characteristic values associated with ambient light from a physical environment. The ambient light emanates from the physical environment towards one side of a translucent display. The plurality of light superposition characteristic values quantifies interactions with the ambient light. The method includes determining a plurality of display correction values associated with the electronic device based on a function of the plurality of light superposition characteristic values and predetermined display characteristics of a computer-generated reality (CGR) object. The method includes changing one or more display operating parameters associated with the electronic device in accordance with the plurality of display correction values in order to satisfy the predetermined display characteristics of the CGR object within a performance threshold.

Modifying display operating parameters based on light superposition from a physical environment

In one implementation, a method of controlling a dimming level of dimmable optical element based on a predicted ambient light level is performed at a device including one or more processors, non-transitory memory, and a dimmable optical element. The method includes predicting a change, in a first direction, in an ambient light level at a future time. The method includes changing, at a first time in advance of the future time and in the first direction, a transmission coefficient of the dimmable optical element based on the predicted change in the ambient light level. The method includes changing, at a second time after the first time and in a second direction opposite the first direction, the transmission coefficient of the dimmable optical element based on the ambient light level at the second time.

Dimming of optical passthrough displays

Various implementations disclosed herein include devices, systems, and methods that determine a type of environment based on light assessment. For example, an example process may include obtaining a first set of light data from a first sensor (e.g., a multiwavelength ambient light sensor) in a physical environment that includes a light source. The first set of light data may identify diffuse light in each of multiple light wavelength ranges that is received by the first sensor. The process may further include obtaining a second set of data from the second sensor in the physical environment. The process may further include determining, based on the first set of light data and the second set of data, that the physical environment is a first type of environment (e.g., indoor) or a second type of environment (e.g., outdoor) based on a set of criteria (e.g., artificial light vs. sunlight).

Environment classification based on light assessment

Various implementations disclosed herein include devices, systems, and methods that obtain sensor data from one or more sensors in a physical environment, and a context is determined based on the sensor data, where the context includes a location of the physical environment and an occurrence of an activity in the physical environment. Camera parameters are selected based on historical parameter data identified based on the context, where the historical parameter data is identified based on camera parameters previously used in the location during prior occurrences of the activity. Then, a camera is configured to capture an image using the selected camera parameters in the location during the occurrence of the activity. In some implementations, the adjusted camera parameters are selected based on context and shared information from a different electronic device regarding camera parameters that were used in the same physical location and during the same activity.

Activity-driven use-case specific camera properties selection

An apparatus includes a primary camera sensor configured to capture images having a first resolution, a primary processing circuit configured to process images captured by the primary camera sensor, a secondary camera sensor configured to capture images having a second resolution, and a secondary processing circuit configured to process images captured by the secondary camera sensor. In response to a determination that a particular object of interest is included in a particular image, the secondary processing circuit may be further configured to cause the primary processing circuit and the primary camera sensor to exit a reduced power mode. The primary camera sensor may be further configured, in response to the exiting, to capture a different image. The primary processing circuit may also be configured to process the different image to validate the particular object of interest.

Video monitoring using a hierarchical combination of camera sensors

Various implementations disclosed herein assess the blurriness of portions of images depicting shapes such as codes or text that have known structural elements. This may involve determining whether a portion of an image of a code or text is sufficiently clear (not blurry) to be accurately interpreted. Blur may be assessed based on spatial frequency of statistical analysis. Blur may be assessed using a machine learning model that is trained using target blur metrics determined based on spatial frequency (e.g., analysis of high frequency portions of discrete cosine transforms of image portions) or statistical analysis (e.g., based on corner/edge detection in image portions).

Robust blur estimation for shapes with known structural elements

Various implementations disclosed herein include devices, systems, and methods that estimate a location of a light source based on ambient light data. For example, an example process may include acquiring ambient light data from an ambient light sensor (ALS) during movement of a device in a physical environment, acquiring motion data from a motion sensor during the movement of the device, determining, based on the ambient light data and the motion data, estimates of three-dimensional (3D) locations of a light source with respect to the device during the movement of the device, and tracking a location of the device in a 3D coordinate system during the movement of the device based on the estimates of the 3D locations of the light source with respect to the device during the movement of the device.

Ambient light sensor-based localization

Various implementations disclosed herein include devices, systems, and methods that provide XR in which virtual objects are positioned based on the accuracy of localizing an electronic device in a physical environment. In some implementations, the technique assesses the accuracy of localization (e.g., centimeter-level accuracy, room-level accuracy, and building-level accuracy) and dynamically adjusts a display strategy. In some implementations, the technique determines a condition causing inaccuracy (e.g., a semantic condition such as 'too fast', “too far”, “too dark”), and provides a notification (e.g., “too fast-slow down”, “too far-move closer”, “too dark-turn on a light”) at the electronic device based on the condition causing the inaccuracy in the localization.

Localization accuracy response

Various implementations disclosed herein include devices, systems, and methods that provide measurement techniques for electronic devices such as optical see-through head mounted devices. In some implementations, a line of sight technique is used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, different measurement point identification techniques are automatically selected and used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, a 3D position of a measurement point is identified to enable measurement of an object in a 3D environment, where the measurement point is identified by selecting from multiple candidates that are determined using different measurement point selection techniques.

Measurement based on point selection

Various implementations disclosed herein include devices, systems, and methods that provide a visual marker including a plurality of markings arranged in a corresponding plurality of shapes. In some implementations, each marking is formed of a set of sub-markings separated by gaps and arranged according to a respective shape, and the gaps of the plurality of markings are configured to encode data and indicate orientation of the visual marker. In some implementations, the plurality of markings are arranged in a plurality of concentric rings of increasing size. In some implementations, the orientation is encoded in a first set of gaps and data in a second set of gaps of the gaps in the plurality of markings.

Visual marker

Identifying a specular surface, such as a mirror, in a captured scene includes extracting, from one or more images of the scene, a set of natural features and generating, from the image, a set of synthesized 'mirrored' features. One or more correspondences may be determined between the set of natural features in the image and the set of synthesized mirrored features. A first set of features are identified based on the determined one or more correspondences as representing a specular surface (e.g., a mirror) located in the scene, and then a geometry and/or location of the specular surface within the scene may be determined. For example, in some embodiments, the feature from a determined pair of corresponding features in a scene that is determined to be farther away from the device that captured the image(s) of the scene may be determined to be the feature lying on the specular surface.

Growing community of inventors

Saratoga, CA, United States of America

Anselm Grundhoefer