Apple Inc.

Mohammad Haris Baig

Daniel Ulbricht

Angela Blechschmidt

Eshan Verma

Tanmay Batra

Chen-Yu Lee

Amit Kumar K C

Amit Kumar Kc

Praveen Gowda Ippadi Veerabhadre Gowda

Quinton L Petty

Various implementations disclosed herein include devices, systems, and methods that enhance an image based on depth and/or other sensor data. For example, an example process may include obtaining an image of a physical environment from a first sensor, each pixel of the image corresponding to an amount of light, obtaining depth data of the physical environment via a second sensor, and enhancing the image based on the depth sensor data.

Image enhancement

In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Plane detection using semantic segmentation

In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Identity-based inclusion/exclusion in a computer-generated reality experience

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Visual search refinement for computer generated rendering environments

A machine learning model is trained and used to perform a computer vision task such as semantic segmentation or normal direction prediction. The model uses a current image of a physical setting and input generated from three dimensional (3D) anchor points that store information determined from prior assessments of the physical setting. The 3D anchor points store previously-determined computer vision task information for the physical setting for particular 3D points locations in a 3D worlds space, e.g., an x, y, z coordinate system that is independent of image capture device pose. For example, 3D anchor points may store previously-determined semantic labels or normal directions for 3D points identified by simultaneous localization and mapping (SLAM) processes. The 3D anchor points are stored and used to generate input for the machine model as the model continues to reason about future images of the physical setting.

Temporal and geometric consistency in physical setting understanding

A machine learning (ML) model is trained and used to produce a probability distribution associated with a computer vision task. The ML model uses a prior probability distribution associated with a particular image capture condition determined based on sensor data. For example, given that an image was captured by an image capture device at a particular height above the floor and angle relative to the vertical world axis, a prior probability distribution for that particular image capture device condition can be used in performing a computer vision task on the image. Accordingly, the machine learning model is given the image as input as well as the prior probability distribution for the particular image capture device condition. The use of the prior probability distribution can improve the accuracy, efficiency, or effectiveness of the ML learning model for the computer vison task.

Computer vision using a prior probability distribution selected based on an image capture condition

In one implementation, a method of generating a confidence value for a result from a primary task is performed at an image processing system. The method includes obtaining, by a feature extractor portion of the neural network, a set of feature maps for an image data frame; generating a contextual information vector associated with the image data frame based on results from one or more auxiliary tasks performed on the set of feature maps by an auxiliary task sub-network portion of the neural network; performing, by a primary task sub-network portion of the neural network, a primary task on the set of feature maps for the image data frame in order to generate a primary task result; and generating a confidence value based on the contextual information vector, wherein the confidence value corresponds to a reliability metric for the primary task result.

Generating a contextual information vector for improved scene understanding

In one implementation, a method of estimating the orientation of an object in an image is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels at a respective plurality of pixel locations and having a respective plurality of pixel values. The method includes determining a first set of pixels locations corresponding to a 2D boundary surrounding an object represented in the image and determining, based on the first set of pixel locations, a second set of pixel locations corresponding to a 3D boundary surrounding the object.

Object detection and orientation estimation

A system and techniques that use one or more machine learning models to predict a dense depth map (e.g., of depth values for all pixels or at least more pixels than a sparse estimation source (e.g., SLAM)). In some implementations, the machine learning model includes two sub models (e.g., neural networks). The first machine learning model predicts computer vision data such as semantic labels and surface normal directions from an input image. This computer vision data will be used to add to or otherwise improve sparse depth data. Specifically, a second machine learning model takes the semantic labels and surface normal directions from and sparse depth data (e.g., 3D points) from a sparse point estimation source (e.g., SLAM) as inputs and outputs a depth map. The output depth map effectively densities the initial depth data (e.g., from SLAM) by providing depth data for additional pixels of the image.

Densifying sparse depth maps

In one implementation, a method of generating a plane hypothesis is performed by a head-mounted device (HMD) including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method include obtaining a point cloud based on the image of the scene and generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene. The method includes generating a plane hypothesis based on the point cloud and the object classification set.

In one implementation, a method of generating a depth map is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes generating, based on a first image and a second image, a first depth map of the second image. The method includes generating, based on the first depth map of the second image and pixel values of the second image, a second depth map of the second image.

Growing community of inventors

San Jose, CA, United States of America

Mohammad Haris Baig