Tencent America LLC

Chao Huang

Shan Liu

Xiaozhong Xu

Xiang Zhang

Jun Tian

Wei Fan

Hui Tang

Lianyi Han

Shih-Yao Lin

Kun Wang

Zhen Qian

Zhimin Huo

Yusheng Xie

Min Tu

Tao Yang

Thuong Nguyen Canh

Xiaozhong Chen

Shihyao Lin

A method performed by at least one processor in a decoder includes receiving a coded video bitstream that includes a compressed two dimensional (2D) mesh corresponding to a surface of three dimensional (3D) volumetric object. The method further includes predicting a current vertex included in the compressed 2D mesh based on an interpolation-based hierarchical prediction scheme that uses at least one sampled vertex included in the compressed 2D mesh. The method further includes deriving a prediction residual associated with the predicted current vertex. The method further includes reconstructing a boundary vertex associated with the 3D volumetric object based on the predicted current vertex and the derived prediction residual.

Coding of boundary UV2XYZ index for mesh compression

In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry receives a data structure for a mesh frame with polygons representing a surface of an object. The data structure for the mesh frame includes a UV atlas that associates vertices of the mesh frame to UV coordinates in the UV atlas. The processing circuitry determines respective sampling rates for regions of the mesh frame according to respective characteristics of the regions of the mesh frame and applies, on the UV atlas, the respective sampling rates for the regions of the mesh frame to determine sampling positions on the UV atlas. The processing circuitry generates one or more sampled two dimensional (2D) maps for the mesh frame according to the sampling positions on the UV atlas, and encodes the one or more sampled 2D maps into a bitstream.

Method and apparatus of adaptive sampling for mesh compression by encoders

Method, apparatus, and system for parallel dynamic mesh alignment are provided. The process may include determining that a temporal alignment is present between one or more frames in a received input mesh including a plurality of polygons that describe a surface of a volumetric object, then, determining an intra-frame alignment scheme to spatially align charts within a frame. The process may also include applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for inter-frame alignment; and processing the one or more frames in parallel based on the intra-frame alignment scheme and inter-frame alignment.

Parallel approach to dynamic mesh alignment

A plurality of neighboring vertices of a current vertex in a current frame of a mesh is determined. The current frame corresponds to the mesh at a first time instance. Each of the plurality of neighboring vertices is connected to the current vertex through a respective edge in the mesh. A plurality of neighboring estimation errors of the plurality of neighboring vertices is determined. Each of the plurality of neighboring estimation errors indicates a difference between a reference vertex of a corresponding one of the plurality of neighboring vertices in a reference frame and the corresponding one of the plurality of neighboring vertices in the current frame. The reference frame corresponds to the mesh at a second time instance. A prediction residue of the current vertex is determined based on the plurality of neighboring estimation errors. Prediction information of the current vertex is generated based on the determined prediction residue.

Temporal prediction based vertex position compression

Aspects of the disclosure provide methods and apparatuses for mesh coding. In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry receives a bitstream carrying a 3D mesh frame. The 3D mesh frame represents a surface of an object with polygons, the polygons are defined by vertices and edges connecting the vertices, the 3D mesh frame is parameterized into a plurality of 2D charts in a 2D UV atlas. The processing circuitry determines respective chart based coding modes for the plurality of 2D charts according to one or more indices decoded from the bitstream, and decodes, from the bitstream, at least respective geometry information associated with the plurality of 2D charts according to the respective chart based coding modes. The processing circuitry reconstructs the 3D mesh frame according to the respective geometry information associated with the plurality of 2D charts.

Method and apparatus for chart based mesh compression

Method, apparatus, and system for atlas sampling-based dynamic mesh compression are provided. The process may include receiving a mesh frame with polygons representing a surface of an object; determining that the mesh frame is associated with an irregular UV chart in response to one or more singularity components in the mesh frame; performing boundary detection to the UV chart to identify one or more boundary loops in the UV chart; and compressing information of the identified one or more boundary loops into a bitstream.

Atlas sampling based mesh compression with charts of general topology

A method and apparatus that receives a coded video bitstream from an encoder; retrieves, from the coded video bitstream, a mesh including a plurality of polygons that describe one or more surfaces of a volumetric object; predicts a current value of an index of the UV-to-XYZ index array from at least one previously coded value included in another index in the UV-to-XYZ index array, wherein the UV-to-XYZ index array includes a plurality of indices which correspond each two-dimensional (2D) UV vertex associated with the mesh to a three-dimensional (3D) XYZ vertex associated with the mesh; derives a prediction residual associated with the current value of the index of the UV-to-XYZ index array; and reconstructs a boundary UV coordinate based on the predicted current value of the index of the UV-to-XYZ index array and the derived prediction residual.

Predictive coding of boundary UV2XYZ index for mesh compression

Coding information of a mesh is received. The coding information includes a plurality of first coordinates and a plurality of second coordinates corresponding to a plurality of vertices and a texture map that are associated with the mesh. A respective first coordinate and a respective second coordinate associated with each of the plurality of vertices are normalized by adjusting the respective first coordinate based on a first factor and the respective second coordinate based on a second factor. The first factor and the second factor are associated with at least one of (i) a bit depth value indicating a coded range of the first coordinates and the second coordinates and (ii) a size of the texture map. The normalized respective first coordinate and the normalized respective second coordinate are expanded based on the first factor and the second factor respectively.

UV coordinate ranges and texture map size

A method and apparatus comprising computer code configured to cause a processor or processors to obtain volumetric data of at least one three-dimensional (3D) visual content, derive a first layer of a mesh including a plurality of first vertices from at least one frame of the volumetric data, determine displacement vectors from ones of the plurality of vertices of the first layer to respective ones of a plurality of second vertices of a second layer of the mesh, a total quantity of the plurality of second vertices is greater than a total quantity of the plurality of first vertices, and signal the volumetric data based on the displacement vectors.

Vertex prediction based on decoded neighbors

Method, apparatus, and system for generating temporally correlated UV atlases are provided. The process may include generating plurality of consistent UV charts based on partition information associated with one or more previous frames; removing one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more incident faces associated with the one or more non-manifold vertices to corresponding UV charts; merging more than one of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with the one or more previous frames; and generating temporally correlated UV atlases for the current frame based on aligning the plurality of consistent UV charts with the one or more reference charts associated with the one or more previous frames.

Mesh parameterization with temporally correlated UV atlases

Aspects of the disclosure provide methods and apparatuses for mesh coding (e.g., compression and decompression). In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry decodes, using a decoder supporting a first bitdepth, a plurality of segmental attribute values having the first bitdepth from a bitstream carrying a mesh that represents a surface of an object. The plurality of segmental attribute values is associated with attribute values of the mesh, the attribute values of the mesh have a second bitdepth that is higher than the first bitdepth. The processing circuitry determines the attribute values of the mesh having the second bitdepth according to the plurality of segmental attribute values having the first bitdepth.

Segment based compression methods for mesh compression

A method and apparatus that receives a coded video bitstream from an encoder; retrieves, from the coded video bitstream, a mesh including a plurality of polygons that describe a surface of a volumetric object; triangulates one or more vertices of the mesh, the one or more vertices are located on a boundary of the mesh or interior of the mesh; infers connectivity information between the one or more vertices using the triangulation of the one or more vertices; and reconstructs one or more boundary UV coordinates based on the connectivity information.

Triangulation methods with boundary information for dynamic mesh compression

There is includes a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining volumetric data of at least one three-dimensional (3D) visual content, obtaining a patch by dividing a plurality of 3D meshes from the volumetric data, the patch including vertices of at least one of the 3D meshes, forming a prediction group comprising a subset of the vertices of the patch, and signaling a prediction mode of the prediction group collectively for the subset of the plurality of vertices of the patch.

Dynamic mesh compression using inter and intra prediction

A plurality of vertices corresponding to a mesh of a current frame that is positionally tracked is divided into a plurality of vertex groups based on a predetermined constant integer. The plurality of vertex groups includes a first vertex group. A first position of a current vertex in the first vertex group is estimated based on a second position of a reference vertex in a reference frame that is different from the current frame. An estimation error indicating a difference between the reference vertex in the reference frame and the current vertex in the first vertex group is determined. A prediction mode of vertices in the first vertex group is encoded based at least on the estimation error between the reference vertex and the current vertex. Prediction information of the vertices in the first vertex group is generated based on the encoded prediction mode of the vertices of the first vertex group.

Grouping of mesh vertices in position compression

A plurality of candidate predictions of a current vertex is determined. The current vertex is one of a plurality of vertices of a mesh. Each of the plurality of candidate predictions is determined based on a respective one of a plurality of triangles in the mesh and including two-dimensional (2D) coordinates in a 2D domain. An average prediction of the plurality of candidate predictions is computed. A 2D prediction of the current vertex is selected from a prediction list that includes the plurality of candidate predictions and the average prediction. A three-dimensional (3D) prediction of the current vertex is determined based on one or more neighboring vertices of the current vertex in the mesh. A prediction residue comprising a prediction error of the current vertex is encoded. The prediction residue is obtained based on the selected 2D prediction and the determined 3D prediction of the current vertex.

Growing community of inventors

Palo Alto, CA, United States of America

Chao Huang