University of Illinois

University of California

David Alexander Forsyth

Kevin Karsch

Ryan J White

Derek Edward Hoiem

Zicheng Liao

Mani Golparvar-Fard

Varsha Chandrashekhar Hedau

Kedan Li

Ranjitha Kumar

Mariya I Vasileva

Yuan Shen

Shreya Rajpal

Bryan Plummer

Krishna Dusad

A search engine server includes a communication interface through which to receive a multi-modal query from a browser of a client device, the multi-modal query including at least a first image of an item. A processing device, coupled to the communication interface, is to: execute a neural network (NN) regressor model on the first image to identify a plurality of second items that are similar to and compatible with the item depicted in the first image, wherein a set of images correspond to the plurality of second items; generate structured text that explains, within one of a phrase or a sentence, why the set of images are relevant to the item; and return, to the browser of the client device via the communication interface, a set of search results comprising the set of images and the structured text.

Search engine use of neural network regressor for multi-modal item recommendations based on visual semantic embeddings

A system and method are disclosed for, using structure-from-motion techniques, projecting a building information model (BIM) into images from photographs taken of a construction site, to generate a 3D point cloud model using the BIM and, when combined with scheduling constraints, facilitates 4D visualizations and progress monitoring. One of the images acts as an anchor image. Indications are received of first points in the anchor image that correspond to second points in the BIM. Calibration information for an anchor camera is calculated based on the indications and on metadata extracted from the anchor image, to register the anchor image in relation to the BIM. A homography transformation is determined between the images and the anchor camera using the calibration information, to register the rest of the images with the BIM, where some of those images are taken from different cameras and from different angles to the construction site.

4D vizualization of building design and construction modeling with photographs

A fragment is relit for insertion into a target scene of an image by obtaining a fragment model for the fragment. A set of detail maps for the fragment model are generated, each of which encodes fine-scale shading effects from the surface detail of the fragment. A target scene model is obtained for the target scene, and the fragment model is inserted into the target scene model. The target scene model with inserted fragment model is rendered, and a composited target scene is generated. A modified target scene is generated by combining the composited target scene and the set of detail maps. Weights assigned to the different detail maps can be changed by the user, allowing the modified target scene to be readily altered without re-rendering the target scene model with the inserted fragment model.

Relighting fragments for insertion into content

An image into which one or more objects are to be inserted is obtained. Based on the image, both a 3-dimensional (3D) representation and a light model of the scene in the image are generated. One or more objects are added to the 3D representation of the scene. The 3D representation of the scene is rendered, based on the light model, to generate a modified image that is the obtained image modified to include the one or more objects.

Inserting objects into content

A method which makes it possible to replace a reference pattern (aka texture) on a video of clothing is described. For example, a video of a T-shirt with the text 'UCLA' could be edited to say 'UC Berkeley' with minimal user intervention. The method is novel because it separates the problem into two parts: erasing the old texture (while properly accounting for lighting) and adding the new texture (properly deformed to appear on the shirt. This decomposition creates realistic images with simple tracking models of the cloth. It is believed that this method may be used for targeted audience sales techniques, with static or dynamic replacement images replacing images on warping or otherwise deforming surfaces, in addition to other perspective deformations. Potentially, real-time image replacements using the techniques taught here could be used for live activities or real-time video feeds.

Automated image replacement using deformation and illumination estimation

Growing community of inventors

Urbana, IL, United States of America

David Alexander Forsyth