Sony Corporation

Sony Group Corporation

Sony Corporation of America

Sony Electronics Inc

Mohammad Gharavi-Alkhansari

Hiroyuki Takeda

Wei Liu

Jie Hu

Ali Jezu Tabatabai

Jie Ni

Ehsan Maani

Golnaz Abdollahian

Daniel A Usikov

Chen Fu

Yoichi Yagasaki

Wei Ji

Alexander Berestov

Tamaki Kojima

Hye-Yeon Cheong

Mohammed Zubair Visharam

Jaewon Shin

Kendai Furukawa

Yuelong Li

An electronic device and method for lighting control for multi-view image capture is provided. The electronic device selects, from a plurality of predefined lighting patterns, a set of lighting patterns to capture a set of images of a subject. The electronic device controls a lighting system to illuminate the subject with a first pulse and a second pulse. An illumination intensity increases from a first intensity to a second intensity, associated with the first pulse and the second pulse respectively. The electronic device controls the lighting system to illuminate the subject with flash pulses corresponding to the selected set of lighting patterns, based on illumination of the subject with the first pulse and the second pulse. The electronic device controls a set of multi-view image capture devices to capture the set of images of the subject, based on illumination of the subject with the selected set of flash pulses.

Lighting control for multi-view image capture

An electronic device and method for generation of reflectance maps for relightable 3D models is disclosed. The electronic device acquires multi-view image data that includes a set of images of an object and generates a 3D mesh of the object based on the multi-view image data. The electronic device obtains a set of motion-corrected images based on a minimization of a rigid motion associated with the object between images of the set of images and generates texture maps in a UV space based on the set of motion-corrected images and the 3D mesh. The electronic device obtains specular and diffuse reflectance maps based on a separation of specular and diffuse reflectance components from the texture maps, and obtains a relightable 3D model of the object based on the specular and diffuse reflectance maps.

Generation of reflectance maps for relightable 3D models

An electronic device and method for 3D human modeling under a specific body-fitting of clothes is disclosed. A set of depth images of a person wearing clothes is acquired and an initial body model is generated based on a first shape-fitting of a human body prior with at least a first depth scan. A posed-body model is obtained based on a pose-fitting of the initial body model with at least one depth scan. In each depth scan, clothed scan points and unclothed scan points are determined. A final body model is determined based on a second shape-fitting of vertices which belong to the posed-body model and correspond to an under-cloth skin portion of the body of the person. The second shape-fitting is based on the clothed scan points and the unclothed scan points. The final body model is textured based on a skin texture map for the body.

Three-dimensional (3D) human modeling under specific body-fitting of clothes

A system and method for neural-network based image-to-image translation is provided. The system includes a generator network which receives an image and generates a first synthetic image based on the received image. The first synthetic image is different from the image in appearance, style, or domain. The system includes a feature-to-image generator that extracts a feature map from an intermediate layer of the generator network and generates a second synthetic image. The system further includes a first discriminator network that generates a result indicative of whether the first synthetic image is real or fake in comparison to a first reference image and a second discriminator network that generates a result indicative of whether the second synthetic image is real or fake in comparison to a second reference image. The system further includes a processor which trains the first generator network in adversarial manner based on losses associated with the results.

Neural network-based image-to-image translation

An electronic device and method for positioning an eyeball into a 3D mesh of a head portion of an object is provided. Images of an eye of an object, the 3D mesh of the head portion of the object, and a 3D template mesh of an eyeball are acquired. 3D feature points for eye regions are extracted from the images and fit to a sphere. An initial pose transformation between the 3D template mesh and the sphere is estimated. Based on the 3D template mesh, a first set of points corresponding to the eye regions are interpolated. Based on sampling parameters for the first set of points, a second set of points are determined. A final pose transformation is determined based on minimization of difference between the first and the second set of points and the 3D template mesh is fit into an eyeball socket of the 3D mesh.

Eyeball positioning for 3D head modeling

An electronic device and method for shape refinement of a 3D mesh reconstructed from images is disclosed. A set of images of an object is acquired and used to estimate a first 3D mesh of a head portion of the object. A first set of operations is executed on the first 3D mesh to generate a second 3D mesh. The first set of operations includes a removal of one or more regions which are unneeded for head-shape estimation and/or a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh. A 3D template mesh is processed to determine a set of filling patches which corresponds to a set of holes in the second 3D mesh. Based on the second 3D mesh and the set of filling patches, a hole filling operation is executed to generate a final 3D mesh.

Shape refinement of three-dimensional (3D) mesh reconstructed from images

A system and method for three-dimensional (3D) microgeometry and reflectance modeling is provided. The system receives images comprising a first set of images of a face and a second set of images of the face. The faces in the first set of images and the second set of images are exposed to omni-directional lighting and directional lighting, respectively. The system generates a 3D face mesh based on the received images and executes a set of skin-reflectance modeling operations by using the generated 3D face mesh and the second set of images, to estimate a set of texture maps for the face. Based on the estimated set of texture maps, the system texturizes the generated 3D face mesh. The texturization includes an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh.

3D microgeometry and reflectance modeling

An electronic device and method of preservation of geometry details in tracked meshes is provided. A set of three-dimensional (3D) scans and a sequence of tracked meshes of an object of interest are acquired. The sequence of tracked meshes comprises a set of tracked meshes which temporally corresponds to the set of 3D scans. Based on differences between the set of tracked meshes and the set of 3D scans, a set of displacement maps is generated. A plurality of vectors is computed, each of which includes surface tension values associated with mesh vertices of a corresponding tracked mesh. Based on the set of displacement maps and a corresponding set of vectors, a neural network is trained on a task of displacement map generation. The trained neural network model is applied on the plurality of vectors to generate displacement maps. Each tracked mesh is updated based on a corresponding displacement map.

Preserving geometry details in a sequence of tracked meshes

An electronic device and method for texture map generation are disclosed. A set of color images from a set of viewpoints and depth information of a subject's face are acquired. A 3D model is generated based on a color image and the depth information. A set of viewpoint-specific projections of the 3D model are generated along the set of viewpoints. The 3D model is refined based on minimization of difference between each viewpoint-specific projection and a corresponding color image. A set of texture maps corresponding to the set of viewpoints is generated, based on the refined 3D model, and set of color images. A dynamic programming workflow is executed to determine seams along which respective portions of the set of texture maps are to be stitched and the respective portions are stitched along the seams to generate a final texture map for the refined 3D model of the subject's face.

Texture map generation using multi-viewpoint color images

The mesh tracking described herein involves mesh tracking on 3D face models. In contrast to existing mesh tracking algorithms which generally require user intervention and manipulation, the mesh tracking algorithm is fully automatic once a template mesh is provided. In addition, an eye and mouth boundary detection algorithm is able to better reconstruct the shape of eyes and mouths.

Automatic mesh tracking for 3D face modeling

An electronic device and method for 3D modeling based on 2D warping is disclosed. The electronic device acquires a color image of a face of a user, depth information corresponding to the color image, and a point cloud of the face. A 3D mean-shape model of a reference 3D face is acquired, and rigid aligned with the point cloud. A 2D projection of the aligned 3D mean-shape model is generated. The 2D projection includes a set of landmark points associated with the aligned 3D mean-shape model. The 2D projection is warped such that the set of landmark points in the 2D projection is aligned with a corresponding set of feature points in the color image. A 3D correspondence between the aligned 3D mean-shape model and the point cloud is determined for a non-rigid alignment of the aligned 3D mean-shape model, based on the warped 2D projection and the depth information.

Three-dimensional (3D) shape modeling based on two-dimensional (2D) warping

An electronic apparatus and method for image-based mask frame interpolation is provided. The electronic apparatus receives a sequence of image frames, including a group of image frames and an intermediate image frame. The electronic apparatus generates foreground (FG) masks based on application of a trained Neural Network on the group of image frames and selects a Region-of-Interest (RoI) from a first image frame of the group of image frames. The electronic apparatus extracts first feature vectors for pixels within the selected RoI and second feature vector for a first pixel of the intermediate image frame. The electronic apparatus estimates a first mask value for the first pixel based on application of a statistical function on the first feature vectors and the second feature vector and generates an intermediate FG mask based on the estimated first mask value and re-estimation of mask values for remaining pixels of the intermediate image frame.

Image-based mask frame interpolation

An electronic device and method for image mask generation using a deep neural network is provided. The electronic device stores an input image that includes an object of interest generates a pilot mask for the input image by application of a trained deep neural network (DNN) on the input image. The electronic device extracts, from the input image, a region of interest that includes the object of interest, based on the generated pilot mask and resizes the extracted region of interest such that a size of the resized region of interest equals a threshold scaling size. The electronic device further generates a final mask by reapplication of the trained DNN on the resized region of interest.

Image mask generation using a deep neural network

An electronic apparatus and method is provided for shape-refinement of a triangular 3D mesh using a modified Shape from Shading (SFS) scheme. The electronic apparatus generates a flat two-dimensional (2D) mesh based on an orthographic projection of an initial three-dimensional (3D) triangular mesh on an image plane that includes a plurality of square grid vertices. The electronic apparatus estimates a final grid depth value for each square grid vertex of the flat 2D mesh based on a modified SFS scheme. The modified SFS scheme corresponds to an objective relationship among a reference grid image intensity value, an initial grid depth value, and a grid albedo value for each square grid vertex of the plurality of square grid vertices. The electronic apparatus estimates a final 3D triangular mesh based on the initial 3D triangular mesh and the estimated final grid depth value.

Shape-refinement of triangular three-dimensional mesh using a modified shape from shading (SFS) scheme

An electronic apparatus for facial texture map generation using single color image and depth information is provided. The electronic apparatus stores a first color image and corresponding depth information of a face, an un-textured three-dimensional (3D) face model of the face and a reference facial texture model. The electronic apparatus generates a first texture map of the face based on the un-textured 3D face model, the first color image, and the depth information. The electronic apparatus further generates a second texture map by texture model fitting of the reference facial texture model on a plurality of first effective points on the first texture map. The electronic apparatus refines the second texture map based on minimization of an objective function for the second texture map and further generates a final texture map based on the first texture map and the refined second texture map.

Growing community of inventors

San Jose, CA, United States of America

Mohammad Gharavi-Alkhansari