Google Inc.

Sony Corporation

Sony Group Corporation

Jianing Wei

Matthias Grundmann

Xue Tu

Alexander Berestov

Tingbo Hou

Adel Ahmadyan

Xiaoling Wang

Chia-Kai Liang

Pingshan Li

Liangkai Zhang

Lun-Cheng Chu

Akira Matsui

Takami Mizukura

Cheng Yong Yang

Chuen-Chien Lee

Keir Banks Mierle

Bryan Woods

Tomonori Shuda

Artsiom Ablavatski

Keith Schaefer

Sundeep Vaddadi

Konstantine Tsotsos

Leon Wong

The present disclosure is directed to systems and methods for performing object detection and pose estimation in 3D from 2D images. Object detection can be performed by a machine-learned model configured to determine various object properties. Implementations according to the disclosure can use these properties to estimate object pose and size.

Systems and methods for object detection including pose and size estimation

The present disclosure is directed to systems and methods for generating synthetic training data using augmented reality (AR) techniques. For example, images of a scene can be used to generate a three-dimensional mapping of the scene. The three-dimensional mapping may be associated with the images to indicate locations for positioning a virtual object. Using an AR rendering engine, implementations can generate an and orientation. The augmented image can then be stored in a machine learning dataset and associated with a label based on aspects of the virtual object.

AR-assisted synthetic data generation for training machine learning models

The present disclosure provides systems and methods for calibration-free instant motion tracking useful, for example, for rending virtual content in augmented reality settings. In particular, a computing system can iteratively augment image frames that depict a scene to insert virtual content at an anchor region within the scene, including situations in which the anchor region moves relative to the scene. To do so, the computing system can estimate, for each of a number of sequential image frames: a rotation of an image capture system that captures the image frames; and a translation of the anchor region relative to an image capture system, thereby providing sufficient information to determine where and at what orientation to render the virtual content within the image frame.

Calibration-free instant motion tracking for augmented reality

A method includes receiving a video comprising images representing an object, and determining, using a machine learning model, based on a first image of the images, and for each respective vertex of vertices of a bounding volume for the object, first two-dimensional (2D) coordinates of the respective vertex. The method also includes tracking, from the first image to a second image of the images, a position of each respective vertex along a plane underlying the bounding volume, and determining, for each respective vertex, second 2D coordinates of the respective vertex based on the position of the respective vertex along the plane. The method further includes determining, for each respective vertex, (i) first three-dimensional (3D) coordinates of the respective vertex based on the first 2D coordinates and (ii) second 3D coordinates of the respective vertex based on the second 2D coordinates.

Object pose estimation and tracking using machine learning

In a general aspect, a method can include receiving data defining an augmented reality (AR) environment including a representation of a physical environment, and changing tracking of an AR object within the AR environment between region-tracking mode and plane-tracking mode.

Hybrid placement of objects in an augmented reality environment

Example embodiments allow for fast, efficient determination of bounding box vertices or other pose information for objects based on images of a scene that may contain the objects. An artificial neural network or other machine learning algorithm is used to generate, from an input image, a heat map and a number of pairs of displacement maps. The location of a peak within the heat map is then used to extract, from the displacement maps, the two-dimensional displacement, from the location of the peak within the image, of vertices of a bounding box that contains the object. This bounding box can then be used to determine the pose of the object within the scene. The artificial neural network can be configured to generate intermediate segmentation maps, coordinate maps, or other information about the shape of the object so as to improve the estimated bounding box.

Real-time pose estimation for unseen objects

The disclosed technology includes techniques for providing improved video stabilization on a mobile device. Using gyroscope data of the mobile device, the physical camera orientation of the mobile device may be estimated over time. Using the physical camera orientation and historical data, corresponding virtual camera orientations representing a camera orientation with undesired rotational movement removed may be modeled using a non-linear filter to provide for mapping of a real image to a stabilized virtual image. The virtual camera orientation may be modified to prevent undefined pixels from appearing in the output image.

Video stabilization for mobile devices

Focus detection is to determine whether an image is in focus or not. Focus detection is able to be used for improving camera autofocus performance. Focus detection by using only one feature does not provide enough reliability to distinguish in-focus and slightly out-of-focus images. A focus detection algorithm of combining multiple features used to evaluate sharpness is described herein. A large image data set with in-focus and out-of-focus images is used to develop the focus detector for separating the in-focus images from out-of-focus images. Many features such as iterative blur estimation, FFT linearity, edge percentage, wavelet energy ratio, improved wavelet energy ratio, Chebyshev moment ratio and chromatic aberration features are able to be used to evaluate sharpness and determine big blur images.

Focus detection

A defocus estimation algorithm is described herein. The defocus estimation algorithm utilizes a single image. A Laplacian of Gaussian approximation is determined by computing a difference of Gaussian. Defocus blur is able to be estimated by computing a blur difference between two images using the difference of Gaussian.

Defocus estimation from single image based on Laplacian of Gaussian approximation

Apparatus and methods for estimating defocus direction from a single image obtained, such as in a digital camera apparatus, are presented. To determine defocus direction, point spread function (PSF) differences for the image are evaluated in the frequency domain, with frequency pairs being found having largest difference in their Fourier transform magnitudes, from which a direction estimate feature is extracted, and defocus direction estimated based on relation of estimated feature and statistics derived from camera image tests. The method can be applied for controlling autofocus mechanisms in cameras, or other applications requiring rapid determination of defocus directions, such as from a single image.

Camera defocus direction estimation

A method for generating a depth map for a 2D image and video includes receiving the 2D image and video; defining a plurality of object classes; analyzing content of the received 2D image and video; calculating probabilities that the received 2D image belongs to the object classes; and determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.

Depth map generation based on soft classification

A method for converting a 2D image into a 3D image includes receiving the 2D image; determining whether the received 2D image is a portrait, wherein the portrait can be a face portrait or a non-face portrait; if the received 2D image is determined to be a portrait, creating a disparity between a left eye image and a right eye image based on a local gradient and a spatial location; generating the 3D image based on the created disparity; and outputting the generated 3D image.

Growing community of inventors

San Jose, CA, United States of America

Jianing Wei