Apple Inc.

Tanmay Batra

Daniel Ulbricht

Angela Blechschmidt

Eshan Verma

Mohammad Haris Baig

Chen-Yu Lee

Amit Kumar K C

Amit Kumar Kc

Peter Georg Meier

Peter Meier

Stefano Alletto

Bharath Kumar Comandur Jagannathan Raghunathan

In one implementation, a method of body pose estimation is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a plurality of two-dimensional images of a body in a three-dimensional environment at a respective plurality of times. The method includes determining, for each of the plurality of two-dimensional images, the two-dimensional location in the two-dimensional image of one or more joints of the body at the respective plurality of times. The method includes determining, based on the two-dimensional locations, a plurality of three-dimensional locations in the three-dimensional environment of the one or more joints of the body at the respective plurality of times. The method includes determining, based on the three-dimensional locations, a plurality of updated three-dimensional locations in the three-dimensional environment of the one or more joints of the body at the respective plurality of times.

Two-stage body pose estimation

In some implementations initially training a first neural network includes inputting the training inputs and corresponding training labels into the first neural network to produce output labels, comparing the output labels to the corresponding training labels using a second neural network that learns and applies a comparison metric, and adjusting parameters of the first neural network based on the comparing. The device then inputs additional inputs into the first neural network to produce additional output labels and corresponding confidence values from the second neural network. The device selects, based on the confidence values, an automatically-labeled training set of data including a subset of the additional inputs and a corresponding subset of the additional output labels. During a second training stage, the device trains the first neural network and the second neural network using the automatically-labeled training set of data.

Iterative neural network training using quality assurance neural network

In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Plane detection using semantic segmentation

In some implementations a neural network is trained to perform a main task using a clustering constraint, for example, using both a main task training loss and a clustering training loss. Training inputs are inputted into a main task neural network to produce output labels predicting locations of the parts of the objects in the training inputs. Data from pooled layers of the main task neural network is inputted into a clustering neural network. The main task neural network and the clustering neural network are trained based on a main task loss from the main task neural network and a clustering loss from the clustering neural network. The main task loss is determined by comparing differences between the output labels and the training labels. The clustering loss encourages the clustering network to learn to label the parts of the objects individually, e.g., to learn groups corresponding to the object parts.

Semi-supervised learning using clustering as an additional constraint

In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Identity-based inclusion/exclusion in a computer-generated reality experience

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Visual search refinement for computer generated rendering environments

In some implementations at an electronic device, training a dual EDNN includes defining a data structure of attributes corresponding to defined parts of a task, processing a first instance of an input using a first EDNN to produce a first output while encoding a first set of the attributes in a first latent space, and processing a second instance of the input using a second EDNN to produce a second output while encoding attribute differences from attribute averages in a second latent space. The device then determines a second set of the attributes based on the attribute differences and the attribute averages. The device then adjusts parameters of the first and second EDNNs based on comparing the first instance of the input to the first output, the second instance of the input to the second output, and the first set of attributes to the second set of attributes.

Unsupervised annotation using dual network system with pre-defined structure

In one implementation, a method of estimating the orientation of an object in an image is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels at a respective plurality of pixel locations and having a respective plurality of pixel values. The method includes determining a first set of pixels locations corresponding to a 2D boundary surrounding an object represented in the image and determining, based on the first set of pixel locations, a second set of pixel locations corresponding to a 3D boundary surrounding the object.

Object detection and orientation estimation

In one implementation, a method of generating a plane hypothesis is performed by a head-mounted device (HMD) including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method include obtaining a point cloud based on the image of the scene and generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene. The method includes generating a plane hypothesis based on the point cloud and the object classification set.

In one implementation, a method of generating a depth map is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes generating, based on a first image and a second image, a first depth map of the second image. The method includes generating, based on the first depth map of the second image and pixel values of the second image, a second depth map of the second image.

Depth map generation

Identity-based inclusion/exclusion in a computer generated reality experience

In one implementation, a method is disclosed for controlling attention diversions while presenting computer-generated reality (CGR) environments on an electronic device. The method includes presenting content representing a view of CGR environment on a display. While presenting the content, an object is detected in a physical environment in which the electronic device is located using an image sensor of the electronic device. The method further includes determining whether the object exhibits a characteristic indicative of attention-seeking behavior. In accordance with a determination that the object exhibits the characteristic, a visual cue corresponding to the object is presented on a first portion of the display without modifying the presentation of the content on a second portion of the display.

Growing community of inventors

Mountain View, CA, United States of America

Tanmay Batra