Personify, Inc.

Verizon Patent and Licensing Inc.

Cong Nguyen

Simion Venshtain

Yi Zhang

Quang Nguyen

Dennis Lin

Gia Dang

Long Dang

Minh Do

An exemplary method includes maintaining a receiver-side mesh-vertices list, receiving duplicative-vertex information from a sender, and responsively reducing the receiver-side mesh-vertices list in accordance with the received duplicative-vertex information, and rendering, using the reduced receiver-side mesh-vertices list, viewpoint-adaptive three-dimensional (3D) personas of a subject at least in part by weighting video pixel colors from different video-camera vantage points of video cameras that capture video streams of the subject, the weighting being performed according to a respective geometric relationship of each video-camera vantage point to a user-selected viewpoint.

Systems and methods for reconstruction and rendering of viewpoint-adaptive three-dimensional (3D) personas

An exemplary method includes generating a 3D mesh of a subject based on frames of time-synchronized video streams of a subject, the frames associated with a first time and generating a transformed facial-mesh model based on a facial portion of the 3D mesh and a facial-mesh model. The method further includes generating a hybrid mesh by combining the transformed facial-mesh model and at least a portion of the 3D mesh. The method further includes generating a current 3D mesh based on frames of the time-synchronized video streams associated with a second time that temporally follows the first time. The method further includes generating a deformed historical 3D mesh by applying a non-rigid deformation process to the hybrid mesh based on the current 3D mesh. The method further includes compressing the deformed historical 3D mesh to form at least one triangle-based 3D submesh including a plurality of submesh triangles.

Systems and methods compression, transfer, and reconstruction of three-dimensional (3D) data meshes

Systems and methods relate to encoded video streams including geometric-data streams transmitted to a receiver for rendering of a viewpoint-adaptive 3D persona. A method includes obtaining at least one triangle-based three-dimensional (3D) submesh of a subject, wherein the obtained triangle-based 3D submesh includes a plurality of submesh vertices that define a plurality of submesh triangles, identifying a plurality of strips of the submesh triangles, generating triangle-strip data representing the identified strips of submesh triangles, generating compressed-submesh data that includes the triangle-strip data, and transmitting the compressed-submesh data to a receiver for reconstruction of the triangle-based 3D submesh of the subject.

Systems and methods relate receiving video streams captured of a subject by video cameras, each video stream including video frames that are time-synchronized with the video, each video camera having a known vantage point in a predetermined coordinate system; obtaining at least one three-dimensional (3D) mesh of the subject, the mesh being time-synchronized and including a plurality of mesh vertices with known locations; identifying a user-selected viewpoint, and identifying a viewpoint-specific subset of the mesh vertices visible; generating 3D submeshes of the subject by calculating visible-vertices lists from the vantage point of each video camera from which the viewpoint-specific subset of mesh vertices is visible; projecting mesh vertices from the calculated visible-vertices lists on to video pixels; and rendering viewpoint-adaptive 3D personas of the subject by weighting video pixel colors from different video-camera vantage points according to the geometric relationship of each video-camera vantage point to the user-selected viewpoint.

Disclosed herein are methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data. One embodiment takes the form of a process that includes obtaining video data depicting at least a portion of a user. The process also includes obtaining short-range depth data associated with the video data. The process also includes obtaining long-range depth data associated with the video data. The video data, short-range depth data, and long-range depth data may be obtained via a single 3-D video camera. The process also includes classifying pixels of the video data as foreground based at least in part on both the short-range depth data and the long-range depth data. In some embodiments, classifying pixels of the video data as foreground comprises employing an alpha mask. The alpha mask may comprise binary foreground (hard) indicators. The alpha mask may comprise foreground-likelihood (soft) indicators.

Methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data

Disclosed herein are methods and systems for assigning pixels distance-cost values using a flood fill technique. One embodiment takes the form of a process that includes obtaining video data depicting a head of a user, obtaining depth data associated with the video data, and selecting seed pixels for a flood fill at least in part by using the depth information. The process also includes performing the flood fill from the selected seed pixels. The flood fill assigns respective distance-cost values to pixels of the video data based on position-space cost values and color-space cost values. In some embodiments, the process also includes classifying pixels of the video data as foreground based at least in part on the assigned distance-cost values. In some other embodiments, the process also includes assigning pixels of the video data foreground-likelihood values based at least in part on the assigned distance-cost values.

Methods and systems for assigning pixels distance-cost values using a flood fill technique

Systems and methods for using geometric primitives to identify background in video data. In one embodiment, a method obtains video data depicting at least a portion of a user. The video data is processed with at least one persona identification module comprising a geometric primitive module for generating a first persona probability map at least in part by: detecting at least one geometric primitive within the video data; identifying a respective region within the video data associated with each of the at least one detected geometric primitives; and assigning the respective regions an increased background-probability in the first persona probability map; and outputting a persona image by extracting pixels from the video data based on the persona probability map.

Methods and systems for identifying background in video data using geometric primitives

Methods and systems for real-time user extraction using deep learning networks. In one embodiment, user extraction comprises obtaining a given frame of color pixel data, checking whether a reset flag is cleared or set, and generating a trimap for the given frame. If the reset flag is cleared, generating the trimap comprises: obtaining a user-extraction contour based on a preceding frame; and generating the trimap based on the obtained user-extraction contour. If the reset flag is set, generating the trimap comprises: detecting at least one persona feature in the given frame; generating an alpha mask by aligning an intermediate contour with the detected persona feature(s), wherein the intermediate contour is based on a color-based flood-fill operation performed on a previous frame which was segmented by a machine-learning-segmentation process; and generating the trimap based on the generated alpha mask. The generated trimap is output for extracting a user persona.

Growing community of inventors

Ho Chi Minh, Vietnam

Cong Nguyen