Apple Inc.

Amazon Technologies, Inc.

A9.com, Inc.

Amit Kumar K C

Daniel Ulbricht

Chen-Yu Lee

Angela Blechschmidt

Eshan Verma

Tanmay Batra

Mohammad Haris Baig

Arnab S Dhua

Yu Lou

Xing Zhang

Geng Yan

Mohammed Haris Baig

Doug Gray

The present disclosure presents approaches for accurately detecting one or more objects of interest in an input image. A bounding box may be generated around the object of interest and may be processed to predict a classification of the object of interest. One or more spatial features may also be determined based, at least in part, upon additional image data located within a second bounding box, relative to the original bounding box. Using the spatial features, a second prediction vector may be generated and combined with the first prediction vector to generate a third prediction vector which may help to accurately classify the object of interest. Using the third prediction vector, recommended content may be determined and provided for display, among other such options.

Object classification for visual search

In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Identity-based inclusion/exclusion in a computer-generated reality experience

Devices, systems, and methods are provided for real-time object placement guidance in augmented reality experience. An example method may include receiving, by a device having a sensor, an indication of an object to be viewed in an physical environment of the device. The example method may also include determining a 3D model of the physical environment using data of the physical environment captured by the sensor. The example method may also include determining that a first surface in the 3D model of the environment is a first floor space, and a second surface in the 3D model of the environment is a first wall space. The example method may also include determining that a portion of the first surface is unoccupied and sized to fit the object. The example method may also include determining a first location in the 3D model of the physical environment for placement of a virtual representation of the object based on a 3D model of the object, wherein the first location corresponds to the portion of the first floor space. The example method may also include generating the virtual representation of the object for display at the first location, the virtual representation of the object having a first orientation, wherein the first orientation is based on a second orientation of the second surface. The example method may also include generating a first real-time view of the physical environment comprising the virtual representation of the object within the portion of the first location and in the first orientation. In some cases, a real-time virtual overlay may also be generated in the physical environment, the real-time virtual overlay indicating a location of a floor space in the physical environment.

Systems and methods for real-time object placement in augmented reality experience

In one implementation, a method of estimating the orientation of an object in an image is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels at a respective plurality of pixel locations and having a respective plurality of pixel values. The method includes determining a first set of pixels locations corresponding to a 2D boundary surrounding an object represented in the image and determining, based on the first set of pixel locations, a second set of pixel locations corresponding to a 3D boundary surrounding the object.

Object detection and orientation estimation

Systems, methods, and computer readable media to implementing an end-to-end room layout estimation are described. A room layout estimation engine performs feature extraction on an image frame to generate a first set of coefficients for a first room layout class and a second set of coefficients for a second room layout class. Afterwards, the room layout estimation engine generates a first set of planes according to the first set of coefficients and a second set of planes according to the second set of coefficients. The room layout estimation engine generates a first prediction plane according to the first set of planes and a second prediction plane according to the second set of planes. Afterwards, the room layout estimation engine merges the first prediction plane and the second prediction plane to generate a predicted room layout for the room.

End-to-end room layout estimation

In one implementation, a method of generating a depth map is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes generating, based on a first image and a second image, a first depth map of the second image. The method includes generating, based on the first depth map of the second image and pixel values of the second image, a second depth map of the second image.

Depth map generation

In one implementation, a method of training a type-agnostic object segmentation system is performed in a type-agnostic object segmentation system including one or more processors, and a non-transitory memory. The method includes obtaining a data set; generating a respective embedding vector for each of a plurality of pixels of the image frame; mapping a plurality of embedding vector representations into a dataspace defined by a dimensionality of the plurality of embedding vector representations; comparing the at least one object instance representation candidate against the respective segmentation mask; and adjusting the type-agnostic object segmentation system in order to satisfy an error threshold across the plurality of image data frames according to a determination that the at least one object instance representation candidate and the respective segmentation mask differ by a threshold value.

Growing community of inventors

Sunnyvale, CA, United States of America

Amit Kumar K C