Qualcomm Incorporated

Qualcomm Technologies, Inc.

Ecole Polytechnique Federale De Lausanne

Vincent Lepetit

Markus Oberweger

Mahdi Rad

Clemens Arth

Peter Michael Roth

Friedrich Fraundorfer

Sinisa Stekovic

Alexander Grabner

Shreyas Hampali

Paul Wohlhart

Pascal Fua

Kwang Moo Yi

Martin Hirzer

Eduard Fortuny Trulls

Systems and techniques are provided for determining one or more poses of one or more objects. For example, a process can include determining, using a machine learning system, a plurality of keypoints from an image. The plurality of keypoints are associated with at least one object in the image. The process can include determining a plurality of features from the machine learning system based on the plurality of keypoints. The process can include classifying the plurality of features into a plurality of joint types. The process can include determining pose parameters for the at least one object based on the plurality of joint types.

Keypoint-based sampling for pose estimation

Systems and techniques are provided for determining environmental layouts. For example, based on one or more images of an environment and depth information associated with the one or more images, a set of candidate layouts and a set of candidate objects corresponding to the environment can be detected. The set of candidate layouts and set of candidate objects can be organized as a structured tree. For instance, a structured tree can be generated including nodes corresponding to the set of candidate layouts and the set of candidate objects. A combination of objects and layouts can be selected in the structured tree (e.g., based on a search of the structured tree, such as using a Monte-Carlo Tree Search (MCTS) algorithm or adapted MCTS algorithm). A three-dimensional (3D) layout of the environment can be determined based on the combination of objects and layouts in the structured tree.

Scene layout estimation

A method is described. The method includes mapping features extracted from an unannotated red-green-blue (RGB) image of the object to a depth domain. The method further includes determining a three-dimensional (3D) pose of the object by providing the features mapped from the unannotated RGB image of the object to the depth domain to a trained pose estimator network.

Systems and methods for three-dimensional pose determination

Techniques are provided for one or more three-dimensional models representing one or more objects. For example, an input image including one or more objects can be obtained. From the input image, a location field can be generated for each object of the one or more objects. A location field descriptor can be determined for each object of the one or more objects, and a location field descriptor for an object of the one or more objects can be compared to a plurality of location field descriptors for a plurality of three-dimensional models. A three-dimensional model can be selected from the plurality of three-dimensional models for each object of the one or more objects. A three-dimensional model can be selected for the object based on comparing a location field descriptor for the object to the plurality of location field descriptors for the plurality of three-dimensional models.

Model retrieval for objects in images using field descriptors

Techniques are provided for determining one or more environmental layouts. For example, one or more planes can be detected in an input image of an environment. The one or more planes correspond to one or more objects in the input image. One or more three-dimensional parameters of the one or more planes can be determined. One or more polygons can be determined using the one or more planes and the one or more three-dimensional parameters of the one or more planes. A three-dimensional layout of the environment can be determined based on the one or more polygons.

Layout estimation using planes

Techniques are provided for selecting a three-dimensional model. An input image including an object can be obtained, and a pose of the object in the input image can be determined. One or more candidate three-dimensional models representing one or more objects in the determined pose can be obtained. From the one or more candidate three-dimensional models, a candidate three-dimensional model can be determined to represent the object in the input image.

Pose estimation and model retrieval for objects in images

A method for training a feature detector of an image processing device, including the steps of detecting features in the image to generate a score map, computing a center of mass on the score map to generate a location, extracting a patch from the image at the location by a first spatial transformer, estimating an orientation of the patch, rotating the patch in accordance with the patch orientation with a second spatial transformer, and describing the rotated patch to create a description vector.

Method, system, and device for learned invariant feature transform for computer images

A method determines a pose of an image capture device. The method includes accessing an image of a scene captured by the image capture device. A semantic segmentation of the image is performed, to generate a segmented image. An initial pose of the image capture device is generated using a three-dimensional (3D) tracker. A plurality of 3D renderings of the scene are generated, each of the plurality of 3D renderings corresponding to one of a plurality of poses chosen based on the initial pose. A pose is selected from the plurality of poses, such that the 3D rendering corresponding to the selected pose aligns with the segmented image.

Pose determination with semantic segmentation

The present disclosure describes methods, apparatuses, and non-transitory computer-readable mediums for estimating a three-dimensional ('3D') pose of an object from a two-dimensional ('2D') input image which contains the object. Particularly, certain aspects of the disclosure are concerned with 3D pose estimation of a symmetric or nearly-symmetric object. An image or a patch of an image includes the object. A classifier is used to determine whether a rotation angle of the object in the image or the patch of the image is within a first predetermined range. In response to a determination that the rotation angle is within the first predetermined range, a mirror image of the object is determined. Two-dimensional (2D) projections of a three-dimensional (3D) bounding box of the object are determined by applying a trained regressor to the mirror image of the object in the image or the patch of the image. The 3D pose of the object is estimated based on the 2D projections.

Three-dimensional pose estimation of symmetrical objects

A method for normalizing an image by an electronic device is described. The method includes obtaining an image including a target object. The method also includes determining a set of windows of the image. The method further includes, for each window of the set of windows of the image, predicting parameters of an illumination normalization model adapted to the window using a first convolutional neural network (CNN), and applying the illumination normalization model to the window to produce a normalized window.

Systems and methods for normalizing an image

Techniques are provided for estimating a three-dimensional pose of an object. An image including the object can be obtained, and a plurality of two-dimensional (2D) projections of a three-dimensional bounding (3D) box of the object in the image can be determined. The plurality of 2D projections of the 3D bounding box can be determined by applying a trained regressor to the image. The trained regressor is trained to predict two-dimensional projections of the 3D bounding box of the object in a plurality of poses, based on a plurality of training images. The three-dimensional pose of the object is estimated using the plurality of 2D projections of the 3D bounding box.

Methods and systems of performing object pose estimation

A computer-implemented method, apparatus, computer readable medium and mobile device for determining a 6DOF pose from an input image. The process of determining 6DOF pose may include processing an input image to create one or more static representations of the input image, creating a dynamic representation of the input image from an estimated 6DOF pose and a 2.5D reference map, and measuring correlation between the dynamic representation and the one or more static representations of the input image. The estimated 6DOF may be iteratively adjusted according to the measured correlation error until a final adjusted dynamic representation meets an output threshold.

Growing community of inventors

Graz, Austria

Vincent Lepetit