Apple Inc.

Magic Leap, Inc.

Chen-Yu Lee

Daniel Ulbricht

Angela Blechschmidt

Eshan Verma

Mohammad Haris Baig

Tanmay Batra

Amit Kumar K C

Andrew Rabinovich

Vijay Badrinarayanan

Tomasz Jan Malisiewicz

Srivignesh Rajendran

Methods and systems for meta-learning are described for automating learning of child tasks with a single neural network. The order in which tasks are learned by the neural network can affect performance of the network, and the meta-learning approach can use a task-level curriculum for multi-task training. The task-level curriculum can be learned by monitoring a trajectory of loss functions during training. The meta-learning approach can learn to adapt task loss balancing weights in the course of training to get improved performance on multiple tasks on real world datasets. Advantageously, learning to dynamically balance weights among different task losses can lead to superior performance over the use of static weights determined by expensive random searches or heuristics. Embodiments of the meta-learning approach can be used for computer vision tasks or natural language processing tasks, and the trained neural networks can be used by augmented or virtual reality devices.

Meta-learning for multi-task learning for neural networks

In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Identity-based inclusion/exclusion in a computer-generated reality experience

Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.

Room layout estimation methods and techniques

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Visual search refinement for computer generated rendering environments

In one implementation, a method of estimating the orientation of an object in an image is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels at a respective plurality of pixel locations and having a respective plurality of pixel values. The method includes determining a first set of pixels locations corresponding to a 2D boundary surrounding an object represented in the image and determining, based on the first set of pixel locations, a second set of pixel locations corresponding to a 3D boundary surrounding the object.

Object detection and orientation estimation

Methods and systems that train a neural network to classify inputs using a first set of labeled inputs corresponding to a source domain and adapt that neural network to classify inputs from another domain. The neural network includes a generator network and two or more classifier networks. The generator network is trained to receive inputs and generate features. The two or more classifier networks are trained to classify those features into classes to obtain class probability predictions. The neural network is adapted to a target domain, for example, by training the classifier networks to maximize a Wasserstein distance-based discrepancy between the class probability predictions of the classifier networks, by training the classifier networks to increase Wasserstein distance-based discrepancy or by training the generator network to minimize the Wasserstein distance-based discrepancy between the class probability predictions of the multiple classifier networks, or both.

Domain adaptation using probability distribution distance

Systems, methods, and computer readable media to implementing an end-to-end room layout estimation are described. A room layout estimation engine performs feature extraction on an image frame to generate a first set of coefficients for a first room layout class and a second set of coefficients for a second room layout class. Afterwards, the room layout estimation engine generates a first set of planes according to the first set of coefficients and a second set of planes according to the second set of coefficients. The room layout estimation engine generates a first prediction plane according to the first set of planes and a second prediction plane according to the second set of planes. Afterwards, the room layout estimation engine merges the first prediction plane and the second prediction plane to generate a predicted room layout for the room.

End-to-end room layout estimation

In one implementation, a method of generating a depth map is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes generating, based on a first image and a second image, a first depth map of the second image. The method includes generating, based on the first depth map of the second image and pixel values of the second image, a second depth map of the second image.

Growing community of inventors

Sunnyvale, CA, United States of America

Chen-Yu Lee