Adobe, Inc.

Amazon Technologies, Inc.

UniversitÈ Laval

Sunil Sharadchandra Hadap

Kalyan Krishna Sunkavalli

Stephen J DiVerdi

Nathan Aaron Carr

Zhuo Hui

Emiliano Gambaretto

Yannick Hold-Geoffroy

Hailin Jin

Amit Kumar Agrawal

Aravind Krishnaswamy

Apoorv Chaudhri

Ashwin Swaminathan

Jonathan Eisenmann

Prakash Ramu

Visesh Uday Kumar Chari

Brandon Michael Smith

Jinjin Li

Ram Sever

Yuelong Li

Gitika Karumuri

Elya Shechtman

Byungmoon Kim

Joon-Young Lee

Zhixin Shu

Zexiang Xu

Hyeongwoo Kim

Mathieu Garon

Miriam Bellver Bueno

Matthew David Fisher

Mehmet Yumer

Rohith Mysore Vijaya Kumar

Jerry Glen Harris

Vimal Bhat

Jean-Francois Lalonde

Kevin Karsch

Michael W Tao

Abhinav Jain

Vidya Narayanan

Rajesh Gautam

JaeGwang Lim

Fabian Langguth

Alexandros Panagopoulos

Jorge F Lopez Moreno

Dylan John Drover

Jorge Moreno Lopez

Xin Shen

Jinsong Zhang

Amogh Gupta

Linjie Luo

Menglei Chai

Nancy Yi Liang

Darin C Tomack

Raja Bala

Walter Michael Shaw

Halin Jin

Peter Borosan

Xiang Huang

Rahul Narain

Zoe Rachel Sherman

Seungbae Bang

Yuxuan Mei

A first region within image data corresponding to a first portion of a user is determined, and a classifier associated with the first portion of the user is determined. A second region within the image data that includes the first region is determined. Within the second region, first pixels corresponding to the first portion of the user, second pixels corresponding to a second portion of the user adjoined to the first portion of the user, and third pixels corresponding to an environment of the user are identified. Based on identifying the second pixels, the classifier associated with the first portion of the user is confirmed. Upon the confirmation an item is rendered on the first portion of the user.

Reducing false positive detections based on classification and segmentation cues

Systems and methods are provided for generating a flat garment pattern and/or 3D mesh representation of a garment from one or more images depicting the garment laid flat or hung up. A system may obtain both a front image depicting a front view of a garment and a back image depicting a back view of the garment. A front and back silhouette of the garment may then be generated, which may include segmenting the garment depiction from background image content. A parametric representation of the garment may then be generated based on the front and back silhouettes, which may be implemented by iteratively optimizing, using differentiable rendering techniques, a garment representation within a parametric garment space previously learned for the particular garment type. A 3D mesh garment representation may then be generated based on the parametric representation, from which a flat sewing pattern may subsequently be generated if desired.

Garment pattern generation from image data

Described are systems and method directed to generation of a dimensionally accurate three-dimensional ('3D') model of a body, such as a human body, based on two-dimensional (“2D”) images of that body. A user may use a 2D camera, such as a digital camera typically included in many of today's portable devices (e.g., cell phones, tablets, laptops, etc.) and obtain a series of 2D body images of their body from different views with respect to the camera. The 2D body images may then be used to generate a plurality of predicted body parameters corresponding to the body represented in the 2D body images. Those predicted body parameters may then be further processed to generate a dimensionally accurate 3D model or avatar of the body of the user.

Three-dimensional body model from a two-dimensional body image

The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes predicting, using a machine learning model, a first plurality of pixels including pixels of an image corresponding to a foot in the image and adding a first virtual cylindrical object to a three-dimensional model of a shoe such that a portion of the first virtual cylindrical object is positioned within the three-dimensional model of the shoe. The method also include determining a set of pixels including pixels of the first plurality of pixels that intersect with the first virtual cylindrical object and removing, from the three-dimensional model of the shoe, a first portion of the three-dimensional model of the shoe that intersects with the set of pixels.

Virtual shoe try-on

The present disclosure describes a system and method for implementing a virtual shoe try-on feature. The method includes receiving, from a database, a three-dimensional model of a shoe with a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe. The method also includes predicting, using a machine learning model and based on an image of a foot, one or more features of the foot, generating, based on the first annotated point, the second annotated point, the third annotated point, and the one or more features, a two-dimensional model of the shoe using the three-dimensional model of the shoe, and superimposing the two-dimensional model of the shoe onto the image of the foot.

Systems and methods are provided for generating custom garment patterns for producing clothing sized to fit any newly input human body representation. A set of reference sample bodies may be selected for which custom designed garment patterns are then obtained. Once reference patterns are obtained, a custom garment pattern for a particular new body may be automatically created by blending two or more of the reference patterns. For example, the neighboring reference bodies to the new body may be identified within a low-dimensional embedding space, and interpolation of garment parameters for the previously designed garment patterns for these reference bodies may be performed to produce a custom garment pattern.

Custom garment pattern blending based on body data

A method includes extracting, using a backbone of a machine learning model, a plurality of features from an image of a foot and predicting, using a first portion of the machine learning model and based on one or more features of the plurality of features, a first aspect of the foot. The method also includes predicting, using a second portion of the machine learning model and based on one or more features of the plurality of features, a second aspect of the foot different from the first aspect, generating, using at least the first aspect and the second aspect, a two-dimensional model of a shoe, and superimposing the two-dimensional model of the shoe onto the image of the foot.

According to an embodiment, a method includes estimating, based at least in part on a first location of a bounding box for a foot in a first frame of a sequence of frames and a second location of the bounding box in a second frame of the sequence of frames, a smoothing transformation that characterizes a movement of the foot in the sequence of frames and smoothing, based at least in part on the smoothing transformation, a movement of a structure of the foot from a first location of the structure in the first frame to a second location of the structure in the second frame. The method also includes superimposing a model of a shoe onto the foot in the sequence of frames based at least in part on the smoothed movement.

Described are systems and methods directed to generation of a personalized three-dimensional ('3D') body model of a body, such as a human body, based on two-dimensional ('2D') images of that body and the generation and presentation of predicted personalized 3D body models of the body when one or more body measurements (e.g., body fat, body weight, muscle mass) are changed. For example, a user may provide a target body measurement value and the implementations will generate one or more predicted personalized 3D body models representative of a predicted appearance of the body with the target body measurement value.

Generation and presentation of predicted personalized three-dimensional body models

Methods and systems are provided for performing material capture to determine properties of an imaged surface. A plurality of images can be received depicting a material surface. The plurality of images can be calibrated to align corresponding pixels of the images and determine reflectance information for at least a portion of the aligned pixels. After calibration, a set of reference materials from a material library can be selected using the calibrated images. The set of reference materials can be used to determine a material model that accurately represents properties of the material surface.

Material capture using imaging

Techniques are disclosed for performing video synthesis of audiovisual content. In an example, a computing system may determine first facial parameters of a face of a particular person from a first frame in a video shot, whereby the video shot shows the particular person speaking a message. The system may determine second facial parameters based on an audio file that corresponds to the message being spoken in a different way from the video shot. The system may generate third facial parameters by merging the first and the second facial parameters. The system may identify a region of the face that is associated with a difference between the first and second facial parameters, render the region of the face based on a neural texture of the video shot, and then output a new frame showing the face of the particular person speaking the message in the different way.

Facial synchronization utilizing deferred neural rendering

Techniques are disclosed for performing video synthesis of audiovisual content. In an example, a computing system may determine first parameters of a face and body of a source person from a first frame in a video shot. The system also determines second parameters of a face and body of a target person. The system determines that the target person is a replacement for the source person in the first frame. The system generates third parameters of the target person based on merging the first parameters with the second parameters. The system then performs deferred neural rendering of the target person based on a neural texture that corresponds to a texture space of the video shot. The system then outputs a second frame that shows the target person as the replacement for the source person.

Person replacement utilizing deferred neural rendering

Described are systems and method directed to generation of a dimensionally accurate three-dimensional ('3D') model of a body, such as a human body, based on two-dimensional ('2D') images of that body. A user may use a 2D camera, such as a digital camera typically included in many of today's portable devices (e.g., cell phones, tablets, laptops, etc.) and obtain a series of 2D body images of their body from different views with respect to the camera. The 2D body images may then be used to generate a plurality of predicted body parameters corresponding to the body represented in the 2D body images. Those predicted body parameters may then be further processed to generate a dimensionally accurate 3D model or avatar of the body of the user.

Three-dimensional body composition from two-dimensional images

The present disclosure relates to using an object relighting neural network to generate digital images portraying objects under target lighting directions based on sets of digital images portraying the objects under other lighting directions. For example, in one or more embodiments, the disclosed systems provide a sparse set of input digital images and a target lighting direction to an object relighting neural network. The disclosed systems then utilize the object relighting neural network to generate a target digital image that portrays the object illuminated by the target lighting direction. Using a plurality of target digital images, each portraying a different target lighting direction, the disclosed systems can also generate a modified digital image portraying the object illuminated by a target lighting configuration that comprises a combination of the different target lighting directions.

Relighting digital images illuminated from a target lighting direction

This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to estimate lighting parameters for specific positions within a digital scene for augmented reality. For example, based on a request to render a virtual object in a digital scene, a system uses a local-lighting-estimation-neural network to generate location-specific-lighting parameters for a designated position within the digital scene. In certain implementations, the system also renders a modified digital scene comprising the virtual object at the designated position according to the parameters. In some embodiments, the system generates such location-specific-lighting parameters to spatially vary and adapt lighting conditions for different positions within a digital scene. As requests to render a virtual object come in real (or near real) time, the system can quickly generate different location-specific-lighting parameters that accurately reflect lighting conditions at different positions within a digital scene in response to render requests.

Estimating lighting parameters for positions within augmented-reality scenes

Growing community of inventors

Dublin, CA, United States of America

Sunil Sharadchandra Hadap