Sony Corporation

Sony Group Corporation

Sony Corporation of America

Jie Hu

Mohammad Gharavi-Alkhansari

An electronic device and method for positioning an eyeball into a 3D mesh of a head portion of an object is provided. Images of an eye of an object, the 3D mesh of the head portion of the object, and a 3D template mesh of an eyeball are acquired. 3D feature points for eye regions are extracted from the images and fit to a sphere. An initial pose transformation between the 3D template mesh and the sphere is estimated. Based on the 3D template mesh, a first set of points corresponding to the eye regions are interpolated. Based on sampling parameters for the first set of points, a second set of points are determined. A final pose transformation is determined based on minimization of difference between the first and the second set of points and the 3D template mesh is fit into an eyeball socket of the 3D mesh.

Eyeball positioning for 3D head modeling

An electronic device and method for shape refinement of a 3D mesh reconstructed from images is disclosed. A set of images of an object is acquired and used to estimate a first 3D mesh of a head portion of the object. A first set of operations is executed on the first 3D mesh to generate a second 3D mesh. The first set of operations includes a removal of one or more regions which are unneeded for head-shape estimation and/or a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh. A 3D template mesh is processed to determine a set of filling patches which corresponds to a set of holes in the second 3D mesh. Based on the second 3D mesh and the set of filling patches, a hole filling operation is executed to generate a final 3D mesh.

Shape refinement of three-dimensional (3D) mesh reconstructed from images

An electronic device and method for texture map generation are disclosed. A set of color images from a set of viewpoints and depth information of a subject's face are acquired. A 3D model is generated based on a color image and the depth information. A set of viewpoint-specific projections of the 3D model are generated along the set of viewpoints. The 3D model is refined based on minimization of difference between each viewpoint-specific projection and a corresponding color image. A set of texture maps corresponding to the set of viewpoints is generated, based on the refined 3D model, and set of color images. A dynamic programming workflow is executed to determine seams along which respective portions of the set of texture maps are to be stitched and the respective portions are stitched along the seams to generate a final texture map for the refined 3D model of the subject's face.

Texture map generation using multi-viewpoint color images

An electronic device and method for 3D modeling based on 2D warping is disclosed. The electronic device acquires a color image of a face of a user, depth information corresponding to the color image, and a point cloud of the face. A 3D mean-shape model of a reference 3D face is acquired, and rigid aligned with the point cloud. A 2D projection of the aligned 3D mean-shape model is generated. The 2D projection includes a set of landmark points associated with the aligned 3D mean-shape model. The 2D projection is warped such that the set of landmark points in the 2D projection is aligned with a corresponding set of feature points in the color image. A 3D correspondence between the aligned 3D mean-shape model and the point cloud is determined for a non-rigid alignment of the aligned 3D mean-shape model, based on the warped 2D projection and the depth information.

Three-dimensional (3D) shape modeling based on two-dimensional (2D) warping

An electronic apparatus and method is provided for shape-refinement of a triangular 3D mesh using a modified Shape from Shading (SFS) scheme. The electronic apparatus generates a flat two-dimensional (2D) mesh based on an orthographic projection of an initial three-dimensional (3D) triangular mesh on an image plane that includes a plurality of square grid vertices. The electronic apparatus estimates a final grid depth value for each square grid vertex of the flat 2D mesh based on a modified SFS scheme. The modified SFS scheme corresponds to an objective relationship among a reference grid image intensity value, an initial grid depth value, and a grid albedo value for each square grid vertex of the plurality of square grid vertices. The electronic apparatus estimates a final 3D triangular mesh based on the initial 3D triangular mesh and the estimated final grid depth value.

Shape-refinement of triangular three-dimensional mesh using a modified shape from shading (SFS) scheme

An electronic apparatus for facial texture map generation using single color image and depth information is provided. The electronic apparatus stores a first color image and corresponding depth information of a face, an un-textured three-dimensional (3D) face model of the face and a reference facial texture model. The electronic apparatus generates a first texture map of the face based on the un-textured 3D face model, the first color image, and the depth information. The electronic apparatus further generates a second texture map by texture model fitting of the reference facial texture model on a plurality of first effective points on the first texture map. The electronic apparatus refines the second texture map based on minimization of an objective function for the second texture map and further generates a final texture map based on the first texture map and the refined second texture map.

Facial texture map generation using single color image and depth information

An electronic apparatus for shape refinement of a three-dimensional (3D) shape model is provided. The electronic apparatus generates a back-projected image for an object portion based on an initial 3D shape model of the object portion and a texture map of the object portion. The electronic apparatus computes an optical flow map between the back-projected image and a two-dimensional (2D) color image of the object portion. The electronic apparatus determines a plurality of 3D correspondence points for a corresponding plurality of vertices of the initial 3D shape model, based on the optical flow map and a depth image of the object portion. The electronic apparatus estimates a final 3D shape model that corresponds to a shape-refined 3D model of the object portion based on the initial 3D shape model and the plurality of 3D correspondence points for the corresponding plurality of vertices of the initial 3D shape model.

Shape refinement of three dimensional shape model

A virtual reality-based apparatus and method to generate a 3D human face model includes, storage of a 3D face model that is an existing 3D face model or a mean-shape face model in at least a neutral expression. A point cloud of the face of the first user is generated based on the plurality of color images and depth information of the face of the first user. A first 3D face model of the first user having neutral expression is generated by a shape-based model-fitment on the stored 3D face model. A shape of the first 3D face model is refined based on a difference between the first 3D face model, the shape-based model-fitment, and the generated point cloud. The display of the refined first 3D face model is controlled to exhibit a minimum deviation from the shape and appearance of the face of the first user.

Virtual reality-based apparatus and method to generate a three dimensional (3D) human face model using image and depth data

Apparatus ad method for texturing a 3D model by UV map in-painting, generates a first UV map from the 3D model and applies color values to the first UV map based on correspondence between a first color image and the first UV map mapped to the first region of the 3D model. A first point is detected on a boundary between a textured region and a first non-textured region within the textured region of the first UV map and a region-of-interest is detected around the first point on the boundary that is utilized to determine texture information of a specific patch of the textured region. A first portion of the first non-textured region of the detected region-of-interest is texturized to obtain a textured region-of-interest in the first UV map, which is utilized to generate a second UV map and texture the at least the first region of the 3D model.

Growing community of inventors

San Jose, CA, United States of America

Jie Hu