Verizon Patent and Licensing Inc.

Personify, Inc.

Other

Simion Venshtain

Cong Nguyen

Yi Zhang

Dennis Lin

Quang Nguyen

Gia Dang

Arthur A Van Hoff

Solmaz Hajmohammadi

Daniel Kopeinigg

Sourabh Khire

Long Dang

Philip Lee

Po-Han Huang

Sanjay J Patel

Andrew Walkingshaw

Minh Do

Glenn Francisco

Christopher Redmann

Charles LePere

An exemplary method includes maintaining a receiver-side mesh-vertices list, receiving duplicative-vertex information from a sender, and responsively reducing the receiver-side mesh-vertices list in accordance with the received duplicative-vertex information, and rendering, using the reduced receiver-side mesh-vertices list, viewpoint-adaptive three-dimensional (3D) personas of a subject at least in part by weighting video pixel colors from different video-camera vantage points of video cameras that capture video streams of the subject, the weighting being performed according to a respective geometric relationship of each video-camera vantage point to a user-selected viewpoint.

Systems and methods for reconstruction and rendering of viewpoint-adaptive three-dimensional (3D) personas

An exemplary method includes generating a 3D mesh of a subject based on frames of time-synchronized video streams of a subject, the frames associated with a first time and generating a transformed facial-mesh model based on a facial portion of the 3D mesh and a facial-mesh model. The method further includes generating a hybrid mesh by combining the transformed facial-mesh model and at least a portion of the 3D mesh. The method further includes generating a current 3D mesh based on frames of the time-synchronized video streams associated with a second time that temporally follows the first time. The method further includes generating a deformed historical 3D mesh by applying a non-rigid deformation process to the hybrid mesh based on the current 3D mesh. The method further includes compressing the deformed historical 3D mesh to form at least one triangle-based 3D submesh including a plurality of submesh triangles.

Systems and methods compression, transfer, and reconstruction of three-dimensional (3D) data meshes

An illustrative method includes obtaining a three-dimensional (3D) mesh of a subject, obtaining a mesh model, and generating a hybrid mesh of the subject. The generating includes replacing a portion of the 3D mesh with the mesh model such that the hybrid mesh includes a non-replaced portion of the 3D mesh represented at a first resolution and the mesh model representing the replaced portion of the 3D mesh at a second resolution.

Systems and methods for model-based modification of a three-dimensional (3D) mesh

An illustrative volumetric capture system accesses a machine learning model associated with bodies of a particular body type, as well as a two-dimensional (2D) image captured by a capture device located at a real-world scene. The 2D image depicts a body of the particular body type that is present at the real-world scene. Using the machine learning model and based on the 2D image, the volumetric capture system identifies a 2D joint location, from a perspective of the capture device, of a particular joint of the body. The volumetric capture system also generates a three-dimensional (3D) reference model of the body that represents the particular joint of the body at a 3D joint location that is determined based on the 2D joint location identified using the machine learning model. Corresponding methods and systems are also disclosed.

Methods and systems for applying machine learning to volumetric capture of a body in a real-world scene

An exemplary three-dimensional (3D) simulation system accesses a two-dimensional (2D) video image captured by a video capture device and that depicts a bounded real-world scene and a real-world object present within the bounded real-world scene. The 3D simulation system accesses respective 3D models of the bounded real-world scene and the real-world object. Based on the 2D video image, the 3D simulation system tracks a spatial characteristic of the real-world object relative to the bounded real-world scene. Based on the tracked spatial characteristic of the real-world object and the 3D models of the bounded real-world scene and the real-world object, the 3D simulation system generates a 3D simulation of the bounded real-world scene within which the real-world object is simulated in accordance with the tracked spatial characteristic of the real-world object. Corresponding methods and systems are also disclosed.

Methods and systems for performing 3D simulation based on a 2D video image

A method includes receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a sporting event. The method further includes determining boundaries of the sporting event from the two-dimensional video streams. The method further includes identifying a location of a sporting object during the sporting event. The method further includes identifying one or more players in the sporting event. The method further includes identifying poses of each of the one or more players during the sporting event. The method further includes generating a three-dimensional model of the sporting event based on the boundaries of the sporting event, the location of the sporting object during the sporting event, and the poses of each of the one or more players during the sporting event. The method further includes generating a simulation of the three-dimensional model.

Generating three-dimensional content from two-dimensional images

Systems and methods relate to receiving a plurality of video streams captured of a subject by a plurality of video cameras, each video stream including video frames time-synchronized according to a shared frame rate, each video camera having a known vantage point in a predetermined coordinate system; obtaining at least one three-dimensional (3D) mesh of the subject at the shared frame rate, the 3D mesh time-synchronized with the video frames of the video streams, the at least one mesh including a plurality of vertices with known locations in the predetermined coordinate system; calculating one or more lists of visible-vertices at the shared frame rate, each list including a subset of the plurality of vertices of the at least one 3D mesh of the subject, the subset being a function of the location of the known vantage point associated with at least one of the plurality of video cameras; generating one or more time-synchronized data streams at the shared frame rate, the one or more time-synchronized data streams including: one or more video streams encoding at least one of the plurality of video streams; and one or more geometric-data streams including the calculated one or more visible-vertices lists; and transmitting the one or more time-synchronized data streams to a receiver for rendering of a viewpoint-adaptive 3D persona of the subject.

Viewpoint-adaptive three-dimensional (3D) personas

Systems and methods relate to encoded video streams including geometric-data streams transmitted to a receiver for rendering of a viewpoint-adaptive 3D persona. A method includes obtaining at least one triangle-based three-dimensional (3D) submesh of a subject, wherein the obtained triangle-based 3D submesh includes a plurality of submesh vertices that define a plurality of submesh triangles, identifying a plurality of strips of the submesh triangles, generating triangle-strip data representing the identified strips of submesh triangles, generating compressed-submesh data that includes the triangle-strip data, and transmitting the compressed-submesh data to a receiver for reconstruction of the triangle-based 3D submesh of the subject.

Systems and methods for capturing, transferring, and rendering viewpoint-adaptive three-dimensional (3D) personas

Systems and methods relate receiving video streams captured of a subject by video cameras, each video stream including video frames that are time-synchronized with the video, each video camera having a known vantage point in a predetermined coordinate system; obtaining at least one three-dimensional (3D) mesh of the subject, the mesh being time-synchronized and including a plurality of mesh vertices with known locations; identifying a user-selected viewpoint, and identifying a viewpoint-specific subset of the mesh vertices visible; generating 3D submeshes of the subject by calculating visible-vertices lists from the vantage point of each video camera from which the viewpoint-specific subset of mesh vertices is visible; projecting mesh vertices from the calculated visible-vertices lists on to video pixels; and rendering viewpoint-adaptive 3D personas of the subject by weighting video pixel colors from different video-camera vantage points according to the geometric relationship of each video-camera vantage point to the user-selected viewpoint.

Systems and methods relate to encoded video streams including geometric-data streams transmitted to a receiver for rendering of a viewpoint-adaptive 3D persona. A method includes obtaining a three-dimensional (3D) mesh of a subject generated from depth-camera-captured information about the subject, obtaining a facial-mesh model, locating a facial portion of the obtained 3D mesh of the subject, computing a geometric transform based on the facial portion and the facial-mesh model, the geometric transform determined in response to one or more aggregated error differences between a plurality of feature points on the facial-mesh model and a plurality of corresponding feature points on the facial portion of the obtained 3D mesh, generating a transformed facial-mesh model using the geometric transform and generating a hybrid mesh of the subject at least in part by combining the transformed facial-mesh model and at least a portion of the obtained 3D mesh.

Systems and methods for reference-model-based modification of a three-dimensional (3D) mesh data model

Disclosed herein are methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data. One embodiment takes the form of a process that includes obtaining video data depicting at least a portion of a user. The process also includes obtaining short-range depth data associated with the video data. The process also includes obtaining long-range depth data associated with the video data. The video data, short-range depth data, and long-range depth data may be obtained via a single 3-D video camera. The process also includes classifying pixels of the video data as foreground based at least in part on both the short-range depth data and the long-range depth data. In some embodiments, classifying pixels of the video data as foreground comprises employing an alpha mask. The alpha mask may comprise binary foreground (hard) indicators. The alpha mask may comprise foreground-likelihood (soft) indicators.

Methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data

Disclosed herein are methods and systems for assigning pixels distance-cost values using a flood fill technique. One embodiment takes the form of a process that includes obtaining video data depicting a head of a user, obtaining depth data associated with the video data, and selecting seed pixels for a flood fill at least in part by using the depth information. The process also includes performing the flood fill from the selected seed pixels. The flood fill assigns respective distance-cost values to pixels of the video data based on position-space cost values and color-space cost values. In some embodiments, the process also includes classifying pixels of the video data as foreground based at least in part on the assigned distance-cost values. In some other embodiments, the process also includes assigning pixels of the video data foreground-likelihood values based at least in part on the assigned distance-cost values.

Methods and systems for assigning pixels distance-cost values using a flood fill technique

Disclosed herein are systems and methods for iterative adjustment of video-capture settings based on identified persona. In an embodiment, a method includes receiving video frames that are being captured by a video camera of an ongoing scene. The method also includes identifying a persona in one or more of the received frames at least in part by identifying, in each such frame, a set of pixels classified as being representative of the persona. The method also includes selecting, based collectively on the brightness values of the pixels in the identified set, a respective adjustment command for one or more adjustable video-capture settings of the camera, as well as outputting the selected commands to the camera for use in continuing to capture video data representative of the ongoing scene.

Growing community of inventors

Chicago, IL, United States of America

Simion Venshtain