Apple Inc.

Arun Srivatsan Rangaprasad

Anselm Grundhoefer

Mohamed Selim Ben Himane

Marco Cavallo

Jean-Charles Bernard Marcel Bazin

James Christopher Schneider

Tiejian Zhang

Ying Bai

Kieran Dimond

Thomas G Salter

Tom Sengelaub

Jeffrey S Norris

Paul Ewers

Scott G Wade

Shubham Agrawal

Dhruv Aditya Govil

Joseph M Luxton

Dhruv A Govil

Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Visual marker with user selectable appearance

Various implementations disclosed herein include devices, systems, and methods that determine a type of environment based on light assessment. For example, an example process may include obtaining a first set of light data from a first sensor (e.g., a multiwavelength ambient light sensor) in a physical environment that includes a light source. The first set of light data may identify diffuse light in each of multiple light wavelength ranges that is received by the first sensor. The process may further include obtaining a second set of data from the second sensor in the physical environment. The process may further include determining, based on the first set of light data and the second set of data, that the physical environment is a first type of environment (e.g., indoor) or a second type of environment (e.g., outdoor) based on a set of criteria (e.g., artificial light vs. sunlight).

Environment classification based on light assessment

Various implementations disclosed herein assess the blurriness of portions of images depicting shapes such as codes or text that have known structural elements. This may involve determining whether a portion of an image of a code or text is sufficiently clear (not blurry) to be accurately interpreted. Blur may be assessed based on spatial frequency of statistical analysis. Blur may be assessed using a machine learning model that is trained using target blur metrics determined based on spatial frequency (e.g., analysis of high frequency portions of discrete cosine transforms of image portions) or statistical analysis (e.g., based on corner/edge detection in image portions).

Robust blur estimation for shapes with known structural elements

Various implementations disclosed herein include devices, systems, and methods that estimate a location of a light source based on ambient light data. For example, an example process may include acquiring ambient light data from an ambient light sensor (ALS) during movement of a device in a physical environment, acquiring motion data from a motion sensor during the movement of the device, determining, based on the ambient light data and the motion data, estimates of three-dimensional (3D) locations of a light source with respect to the device during the movement of the device, and tracking a location of the device in a 3D coordinate system during the movement of the device based on the estimates of the 3D locations of the light source with respect to the device during the movement of the device.

Ambient light sensor-based localization

Implementations of the subject technology provide visualizations of non-visible features of a physical environment, at the location of the non-visible features in the physical environment. The non-visible features may include wireless communications signals, sounds, airflow, gases, subsonic and/or ultrasonic waves, hidden objects, or the like. A device may store visual contexts for visualizations of particular non-visible features. The device may obtain a depth map that allows the device to determine the location of the non-visible feature in the physical environment and to overlay the visualization on a user's view of that location. In this way, the non-visible feature can be visualized its correct location, orientation, direction and/or strength in the physical environment.

Visualization of non-visible phenomena

Various implementations disclosed herein include devices, systems, and methods that provide a visual marker including a plurality of markings arranged in a corresponding plurality of shapes. In some implementations, each marking is formed of a set of sub-markings separated by gaps and arranged according to a respective shape, and the gaps of the plurality of markings are configured to encode data and indicate orientation of the visual marker. In some implementations, the plurality of markings are arranged in a plurality of concentric rings of increasing size. In some implementations, the orientation is encoded in a first set of gaps and data in a second set of gaps of the gaps in the plurality of markings.

Visual marker

Various implementations disclosed herein include devices, systems, and methods that match image portions of a first set of image portions and image portions of another set of image portions. This may be performed in a such a way that the sum of descriptor distances of the matched image portions is optimized in a globally optimal way. In some implementations, the matching is subject to one or more constraints, e.g., ensuring that each image portion of the first data set is matched with at least one image portion of the second data set and ensuring that each image portion of the second data set is matched with only one image portion of the first data set.

Optimal matching of image portion data sets

Various implementations disclosed herein include methods, electronic devices, and systems for performing perceptual-based color correction based on chromaticity values. To that end, in some implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a see-through display. The method includes determining a chromaticity value associated with ambient light from a physical environment. The chromaticity value quantifies the ambient light. The method includes determining a set of color correction values based on a function of the chromaticity value and image data. The set of color correction values and the chromaticity value together satisfy one or more perceptual criteria. The method includes modifying the image data in order to generate display data based on a function of the set of color correction values. The method includes displaying the display data on the see-through display.

Color correction based on perceptual criteria and ambient light chromaticity

Various implementations disclosed herein include devices, systems, and methods for verifying that an image includes a complete code of a time-varying visual marker that displays codes sequentially on a display. In some implementations, the verification determines that the image include a complete code rather than combinations of sequentially-displayed codes that may be included in an image based on use of a rolling shutter (e.g., in a camera of a detecting electronic device). In some implementations, the verification involves comparing a first verification portion of an image to a second opposing verification portion of an image. Various implementations disclosed herein include devices, systems, and methods for modifying image capture parameters (e.g., frame rate) to ensure capture of all codes of a time-varying visual marker.

Time varying visual marker verification and decoding

Various implementations disclosed herein include devices, systems, and methods that provide color visual markers that include colored markings that encode data, where the colors of the colored markings are determined by scanning (e.g., detecting the visual marker using a sensor of an electronic device) the visual marker itself. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, the visual marker is detected in the image by detecting a predefined shape of a first portion of the visual marker in the image. Then, a color-interpretation scheme is determined for interpreting colored markings of the visual marker that encode data by identifying a set of colors at a corresponding set of predetermined locations on the visual marker. Then, the data of the visual marker is decoded using the colored markings and the set of colors of the color-interpretation scheme.

Various implementations disclosed herein include devices, systems, and methods that select colors for visual markers that include colored markings encoding data. In some implementations, the colors are automatically or semi-automatically selected. In some implementations, the colors are selected to remain sufficiently detectable despite changes in lighting conditions or printing/display conditions. In some implementations, a set of colors selectable for use in a visual marker is obtained. Then, measures of distance between a plurality of colors of the set of colors is determined, and a subset of the set of colors for the visual marker is selected based on the measure of distance between colors of the subset of colors. In some implementations, the visual marker appearance includes graphical elements encoding data using the subset of colors. In some implementations, input is received using a GUI on a display to determine multiple colors based on a source image.

Growing community of inventors

Sunnyvale, CA, United States of America

Arun Srivatsan Rangaprasad