A9.com, Inc.

Motorola Solutions, Inc.

Himanshu Arora

Arnab Sanat Kumar Dhua

Sunil Ramesh

Radek Grzeszczuk

Xi Zhang

Yu Lou

Arnab S Dhua

Chun-Kai Wang

Colin Jon Taylor

David Creighton Mott

Karl Hillesland

Divyansh Agarwal

Chun Kai Wang

Sudeshna Pantham

Adam Wiggen Kraft

Boaz J Super

Eran Borenstein

Feng Guo

Xing Zhang

Atul Kumar

Anish Ramdas Nair

Chetan Pitambar Bhole

Erick Cantu-Paz

Paulo Ricardo Dos Santos Mendonca

Yifan Xing

Max Delgadillo, Jr

Jason Canada

Tomas Francisco Yago Vicente

Jesse Chang

Nicholas Corso

Ting-Hsiang Hwang

Brian Graham

Kenan Deng

Sundar Vedula

Chen Fang

Yuting Wang

Approaches described and suggested herein relate to generating three-dimensional representations of objects to be used to render virtual reality and augmented reality effects on personal devices such as smartphones and personal computers, for example. An initial surface mesh of an object is obtained. A plurality of silhouette masks of the object taken from a plurality of viewpoints is also obtained. A plurality of depth maps are generated from the initial surface mesh. Specifically, the plurality of depth maps are taken from the same plurality of viewpoints from which the silhouette images are taken. A volume including the object is discretized into a plurality of voxels. Each voxel is then determined to be either inside the object or outside of the object based on the silhouette masks and the depth data. A final mesh is then generated from the voxels that are determined to be inside the object.

Depth hull for rendering three-dimensional models

Images of various views of objects can be captured. An object mesh structure can be created based at least in part on the object images. The object mesh structure represents the three-dimensional shape of the object. Alpha masks indicating which pixels are associated with the object can be used to refine the object mesh structure. A request can be made to view the object from an arbitrary viewpoint which differs from the viewpoints associated with the captured images. A subset of the captured images can be used to create a synthetic image. Different weights can be assigned to the captured image to render a synthetic image that represents the view from the arbitrary viewpoint selected. The input images for the synthetic image can be prefetched, or loaded into memory before the arbitrary view is requested. The images can also be cached for future use or to avoid reloading them for another synthetic image.

Rendering three-dimensional models on mobile devices

Approaches described and suggested herein relate to generating an enhanced point cloud representation of an objection and generating a surface mesh from the enhanced point cloud. The surface mesh can be used to render three-dimensional representations of objects on personal devices such as smartphones and personal computers, for example. Generating an enhanced point cloud of an object includes capturing a plurality of images of the object from a plurality of viewpoints about the object, generating an initial point cloud representation of the object from the plurality of images, generating a preliminary surface mesh from the point cloud using a Delauney-based meshing algorithm, and sampling points from the preliminary surface mesh. The sampled points are then added to the point cloud to form the enhanced point cloud. A final surface mesh can then be generated from the enhanced point cloud using a Poisson-based meshing algorithm.

Enhanced point cloud for three-dimensional models

A machine learning system receives a reference image and generates a series of projected view images of a physical object represented in the images. Parallel neural networks may receive the reference image and series of projected view images for analysis to determine one or more features of the physical object. By pooling the results from the parallel network, a single output may be provided to a set of decodes that are trained to identify a material property of the one or more items. As a result, a three-dimensional model may be generated that includes a graphical representation of the object as a function of its material properties to enable improved rendering.

Method and system for generating models from multiple views

Approaches herein provide for multi-camera calibration and subsequent application in augmented reality applications. The approach obtains image data of a three-dimensional (3D) calibration object from different directions using the at least one camera. A determination is made for extrinsic parameters associated with a relative position of the 3D calibration object to the at least one camera. A further determination is made for intrinsic parameters associated with the at least one camera relative to the 3D calibration object facing in the different positions. Revised intrinsic parameters are generated from the intrinsic parameters. The revised intrinsic parameters provide a calibration output that may be applied to generate views of an item in an augmented reality application.

Camera calibration for augmented reality

A complex three-dimensional virtual representation of an object can be rendered. Virtual images can be captured representing a plurality of views of the complex virtual representation. The virtual images can be converted into binary masks depicting the object pixels and non-object pixels in the virtual images. The binary masks can be used to create a three-dimensional representation of the object having lower complexity than the first three-dimensional virtual representation of the object. In embodiments, the low complexity three-dimensional virtual representation of the object and the virtual images are sent to a mobile device to render a low payload representation of the object on the mobile device.

Systems and methods for displaying 3D containers in a computer generated environment are described. A computing device may provide a user with a catalog of objects which may be purchased. In order to view what an object may look like prior to purchasing the object, a computing device may show a 3D container that has the same dimensions as the object. As discussed herein, the 3D container may be located and oriented based on a two-dimensional marker. Moreover, some 3D containers may contain a representation of an object, which may be a 2D image of the object.

Augmented reality product preview

Objects can be rendered in three-dimensions and viewed and manipulated in an augmented reality environment. Background images are subtracted from object images from multiple viewpoints to provide baseline representations of the object. Morphological operations can be used to remove errors caused by misalignment of an object image and background image. Using two different contrast thresholds, pixels can be identified that can be said at two different confidence levels to be object pixels. An edge detection algorithm can be used to determine object contours. Low confidence pixels can be associated with the object if they can be connected to high confidence pixels without crossing an object contour. Segmentation masks can be created from high confidence pixels and properly associated low confidence pixels. Segmentation masks can be used to create a three-dimensional representation of the object.

Creating multi-dimensional object representations

Various embodiments provide for the generation of 3D models of objects. For example, depth data and color image data can be captured from viewpoints around an object using a sensor. A camera having a higher resolution can simultaneously capture image data of the object. Features between images captured by the image sensor and the camera can be extracted and compared to determine a mapping between the camera and the image. Once the mapping between the camera and the image sensor is determined, a second mapping between adjacent viewpoints can be determined for each image around the object. In this example, each viewpoint overlaps with an adjacent viewpoint and features extracted from two overlapping viewpoints are matched to determine their relative alignment. Accordingly, a 3D point cloud can be generated and the images captured by the camera can be projected on the surface of the 3D point cloud to generate the 3D model.

3-D model generation using multiple cameras

A computing device can determine visual words for images in a first set of images associated with a description that includes a term. Each visual word can describe pixels in an image. The device can determine visual words for images in a second set of images that are associated with a description that does not include the term. The device can determine a set of visual words that are associated with the term. The device can determine regions in images corresponding to visual words in the set of visual words. The device can associate the respective determined regions for the images with the term in a database. When a search query is received that includes an image that is determined to be associated with the term, the database can be utilized to determine images responsive to the query by identifying images that are also associated with the term.

Approaches for associating terms with image regions

Various embodiments provide a method for computing color descriptors of product images. For example, a number of fine color representatives can be determined to describe color variation in an image as a histogram by assigning a saturation value and a brightness value to a plurality of color hues. For each pixel of the image, the closest color among a defined fine color representative set is computed. In this example, each of the pixels is assigned a color ID corresponding to their closest matching fine color representative and at least one family color ID corresponding one or more pure color families. In this example, a histogram of the color representatives and a histogram for the color families are computed. A single color vector descriptor for the image is then determined by combining the family histogram with the color representative histogram.

Visual search utilizing color descriptors

Growing community of inventors

San Jose, CA, United States of America

Himanshu Arora