A9.com, Inc.

Amazon Technologies, Inc.

Arnab S Dhua

Sunil Ramesh

Gurumurthy D Ramkumar

Himanshu Arora

Nalin Senthamil

Bryan E Feldman

Yu Lou

Raghavan Manmatha

Gautam Bhargava

Max Delgadillo

Divyansh Agarwal

Douglas Ryan Gray

Xing Zhang

Amit Kumar K C

Max Delgadillo, Jr

Chun Kai Wang

Supratik Bhattacharyya

William F Stasior

Leslie Wang

Stephen S Tompkins

Ketan Ramesh Deshpande

Michael Q Phung

Geng Yan

Catherine L Wah

Matthew W Amacker

Xi Zhang

Karl Hillesland

Son Dinh Tran

Keshav Menon

Sonja E Hyde-Moyer

Jason Canada

Jesse Chang

Tomas Francisco Yago Vicente

Nicholas Corso

Xinliang Zhu

Aleksandr V Segal

Kenan Deng

Doug Gray

Ella Day Bench

Brian Graham

Sundar Vedula

Ting-Hsiang Hwang

The present disclosure presents approaches for accurately detecting one or more objects of interest in an input image. A bounding box may be generated around the object of interest and may be processed to predict a classification of the object of interest. One or more spatial features may also be determined based, at least in part, upon additional image data located within a second bounding box, relative to the original bounding box. Using the spatial features, a second prediction vector may be generated and combined with the first prediction vector to generate a third prediction vector which may help to accurately classify the object of interest. Using the third prediction vector, recommended content may be determined and provided for display, among other such options.

Object classification for visual search

Combined feature vectors may be generated to map features of two or more search queries to a common embedding space. A user may provide an initial input query and then provide a refinement query. Independent feature vectors may be generated for each of the initial input query and the refinement query, may be weighted, and then may be combined to form a combined feature vector. The combined feature vector aligns different search modalities within the common embedding space that may be executed against an index.

Multimodal search and refinement

Approaches described and suggested herein relate to generating three-dimensional representations of objects to be used to render virtual reality and augmented reality effects on personal devices such as smartphones and personal computers, for example. An initial surface mesh of an object is obtained. A plurality of silhouette masks of the object taken from a plurality of viewpoints is also obtained. A plurality of depth maps are generated from the initial surface mesh. Specifically, the plurality of depth maps are taken from the same plurality of viewpoints from which the silhouette images are taken. A volume including the object is discretized into a plurality of voxels. Each voxel is then determined to be either inside the object or outside of the object based on the silhouette masks and the depth data. A final mesh is then generated from the voxels that are determined to be inside the object.

Depth hull for rendering three-dimensional models

Devices, systems, and methods are provided for real-time object placement guidance in augmented reality experience. An example method may include receiving, by a device having a sensor, an indication of an object to be viewed in an physical environment of the device. The example method may also include determining a 3D model of the physical environment using data of the physical environment captured by the sensor. The example method may also include determining that a first surface in the 3D model of the environment is a first floor space, and a second surface in the 3D model of the environment is a first wall space. The example method may also include determining that a portion of the first surface is unoccupied and sized to fit the object. The example method may also include determining a first location in the 3D model of the physical environment for placement of a virtual representation of the object based on a 3D model of the object, wherein the first location corresponds to the portion of the first floor space. The example method may also include generating the virtual representation of the object for display at the first location, the virtual representation of the object having a first orientation, wherein the first orientation is based on a second orientation of the second surface. The example method may also include generating a first real-time view of the physical environment comprising the virtual representation of the object within the portion of the first location and in the first orientation. In some cases, a real-time virtual overlay may also be generated in the physical environment, the real-time virtual overlay indicating a location of a floor space in the physical environment.

Systems and methods for real-time object placement in augmented reality experience

System and methods are provided that generate a three-dimensional model from a physical space. While a user is scanning and/or recording the physical space with a user computing device, user speech describing the physical space is recorded. A transcript is generated from the audio captured during the scan and/or image recording of the physical space. Keywords from the transcript are used to improve computer-vision object identification, which is incorporated in the three-dimensional model.

Three-dimensional room analysis with audio input

Approaches described and suggested herein relate to generating an enhanced point cloud representation of an objection and generating a surface mesh from the enhanced point cloud. The surface mesh can be used to render three-dimensional representations of objects on personal devices such as smartphones and personal computers, for example. Generating an enhanced point cloud of an object includes capturing a plurality of images of the object from a plurality of viewpoints about the object, generating an initial point cloud representation of the object from the plurality of images, generating a preliminary surface mesh from the point cloud using a Delauney-based meshing algorithm, and sampling points from the preliminary surface mesh. The sampled points are then added to the point cloud to form the enhanced point cloud. A final surface mesh can then be generated from the enhanced point cloud using a Poisson-based meshing algorithm.

Enhanced point cloud for three-dimensional models

A machine learning system receives a reference image and generates a series of projected view images of a physical object represented in the images. Parallel neural networks may receive the reference image and series of projected view images for analysis to determine one or more features of the physical object. By pooling the results from the parallel network, a single output may be provided to a set of decodes that are trained to identify a material property of the one or more items. As a result, a three-dimensional model may be generated that includes a graphical representation of the object as a function of its material properties to enable improved rendering.

Method and system for generating models from multiple views

A complex three-dimensional virtual representation of an object can be rendered. Virtual images can be captured representing a plurality of views of the complex virtual representation. The virtual images can be converted into binary masks depicting the object pixels and non-object pixels in the virtual images. The binary masks can be used to create a three-dimensional representation of the object having lower complexity than the first three-dimensional virtual representation of the object. In embodiments, the low complexity three-dimensional virtual representation of the object and the virtual images are sent to a mobile device to render a low payload representation of the object on the mobile device.

Rendering three-dimensional models on mobile devices

A user can capture image information about an object in order to identify the object. In addition to obtaining information about the object, the user can obtain additional views of the object. For example, if the user captures an image of packaging containing the object, the user can receive views of the object when out of the packaging, as well as a view showing the relative size of the object within the packaging. If the object requires assembly, the views can include views of the object at the component, part, or ingredient level, as well as views during assembly and one or more views of the finished product. A user also can obtain a virtual x-ray view that enables the user to move the client device with respect to the object in order to obtain different views of what is contained within the packaging.

Providing augmented reality view of objects

Approaches are described for rendering augmented reality overlays on an interface displaying the active field of view of a camera. The interface can display to a user an image or video, for example, and the overlay can be rendered over, near, or otherwise positioned with respect to any text or other such elements represented in the image. The overlay can have associated therewith at least one function or information, and when an input associated with the overlay is selected, the function can be performed (or caused to be performed) by the portable computing device.

Providing overlays based on text in a live camera view

Techniques for augmenting an image of an object captured and displayed in real time with associated content are disclosed. In one embodiment, the method for augmenting the image includes receiving information defining a sampled frame of a video being captured by an electronic device in substantially real time, determining information representative of an object captured in the sampled frame based on the received information, causing the determined information to match stored information defining a plurality of items to locate an item matched to the captured object, retrieving content associated with the matched item, and providing the retrieved content for display with the captured image on the electronic device. The retrieved content may be rendered in an overlay element that overlays the captured image displayed on the electronic device. The rendered content is configured to enable a user to interact with the content.

Augmenting a live view

Techniques for facilitating a shape-based search of content items are disclosed. In some situations, the techniques include receiving a request for search of content that includes an image of an item and a glyph input by the user on the image that indicates a shape of the first item, extracting features associated with the item from an area of the image on which the glyph is input; identifying a shape category of the first item based at least in part on the indicated shape of the glyph; and conducting a search for items having features that match the extracted features of the first item, the search being conducted among a plurality of items associated with the identified shape category. The retrieved search results may be returned to a user.

Techniques for shape-based search of content

Visual incongruity in search result sets may be reduced at least in part by searching an optimized visually significant subset of a category tree that categorizes a collection of content. The category tree may be optimized at build time at least in part by pruning with respect to visual coherence and by the size of the content collection subset referenced by particular categories. Content collection subset sizes both too large and too small can detract from the visual significance of a particular category. The visually significant subset of the category tree may be further optimized at query time by intersecting the visually significant subset with the query-associated sub-tree(s) and further pruning categories in the visually significant subset that have child categories in the visually significant subset. Searching with respect to the optimized visually significant subset can also improve search efficiency.

Refinement shape content search

Growing community of inventors

Sunnyvale, CA, United States of America

Arnab S Dhua