Sony Group Corporation

Sony Corporation of America

Sony Corporation

Sony Group Corporatiom

Danillo Graziosi

Ali Jezu Tabatabai

Alexandre Zaghetto

Arash Vosoughi

Ohji Nakagami

Vladyslav Zakharchenko

Pranav Aniruddha Kadam

A new high-level syntax element referred to as a basemesh patch data unit allows the transmission of syntax elements related to the basemesh component of a V3C bitstream. Previously, the V3C standard did not have any additional patches targeting the basemesh only. A basemesh patch data unit works with the previously disclosed sub-patch concept to provide syntax elements to be used with basemesh bitstream. The basemesh patch data unit allows mesh data and other types of data (e.g., point clouds) to be mixed together at the patch level and generate a richer representation of 3D objects. Furthermore, the basemesh patch data unit expands the concept of sub-meshes and patches by providing a flexible way to arrange the data in 3D (at the basemesh level) and in 2D (at the texture map level) allowing different configurations, instance, multiple attribute images and multiple basemeshes.

V3C syntax new basemesh patch data unit

Quantization-related parameters are removed from the lifting transform parameters structure and have an independent structure referred to as quantization parameters that will contain the quantization-related information. The modified implementation has no performance difference from a reference implementation, and the modified implementation now can address displacement quantized values, even if they were not transformed.

Displacement quantization syntax

Depth image generation is improved by more efficient encoding using video codecs. The mapping of the depth to the luma channel is performed by not using all bits available, and with the remaining bits, a depth scaling factor is generated and incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described, where the positions of vertices are adjusted according to the normal estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared to the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the positions of the vertices are adjusted to match the estimated normal from the pixels' surfaces. The adjustment can follow an iterative minimization process.

Mesh geometry coding

New syntax elements are used to extend patch types, and the syntax is added to the V3C standard. The new syntax defines patches that encode meshes by projecting connected triangles to a 2D surface, patches that encode triangles or triangle strips without any projection, or patches that are tracked over time and are encoded by projecting connected triangles to a 2D surface. Furthermore, the syntax allows for different ways of coding the mesh-specific information. For instance, the syntax enables three different encoding methods for the vertex position: explicit (directly added to the atlas stream), embedded on video data (occupancy map data), or encoded using an external mesh encoder.

Mesh patch syntax

An attribute prediction and compensation scheme for geometry-based dynamic point cloud compression is described herein. A combination of multiple reference frames are able to be used as a predictor for current frames. The method described herein improves efficiency and accuracy.

Inter-frame attribute coding in geometry-based dynamic point clouds compression

Methods, systems and device for efficiently compressing task-oriented dynamic meshes using occupancy networks are described herein. A single trained occupancy network model is able to reconstruct a mesh video using a few additional points per input mesh frame. To optimize the compression of the model and points, the estimated rate to compress the occupancy network is able to be included in the loss function. This minimizes the number of bits to encode the model, while it tries to reproduce the meshes as well as possible. An adaptive subsampling per input mesh is added to optimize the mesh reconstruction and the N-point point clouds compression. To optimize the model to perform a particular task, a metric is added to the cost function that takes this task into account.

Task-oriented dynamic mesh compression using occupancy networks

An electronic device and method for multiscale inter-prediction for dynamic point cloud compression is provided. The electronic device receives a set of reference point cloud frames and a current point cloud frame. The electronic device generates reference frame data comprising a feature set for each reference point cloud frame and a first set of features for the current point cloud frame. The electronic device predicts a second set of features for the current point cloud frame, using a first neural network predictor, based on the reference frame data. The electronic device computes a set of residual features based on the first set of features and the second set of features. The electronic device generates a set of quantized residual features based on the set of residual features and a bitstream of encoded point cloud data for the current 3D point cloud frame based on the set of quantized residual features.

Multiscale inter-prediction for dynamic point cloud compression

An architecture includes new blocks that transform mesh connectivity to enable lossy compression. In addition, a method generates surface patches from meshes and preserves the overall aspect of the object, while generating patches for efficient encoding using the V3C standard. Triangles are joined together into connected components, and conditional projection of each triangle to the surface is performed to enforce connectivity at the projected surface. A new depth filtering algorithm is able to be used to improve compression performance. The new blocks added for connectivity compression and the techniques introduced to ensure 2D connectivity to allow for UV texture map transmission and to filter depth values and avoid high-frequency edges in the depth image are described herein.

Patch generation for dynamic mesh coding

Ways to post-process a decoded mesh and modify received triangles per patch to improve the mesh geometry are described herein. Since the transmitted geometry contains the high-resolution surface information, the information is able to be applied to the mesh reconstruction operation to generate triangles that are aligned with the original surface. Methods include generating new triangles by splitting the received triangles' edges according to their size, by inserting new vertices at the triangle's centroids, by splitting the vertices, and by performing marching cubes in surfaces defined by the geometry images.

Mesh patch sub-division

Methods, systems and devices described herein implement a task-driven machine learning-based compression scheme for point cloud geometry implicit representation. The machine learning-based codec is able to be optimized for a task to achieve better compression rates by being conditioned to what the reconstructed signal will be used for. The latent representation of the point cloud or the neural network that implicitly represents the point cloud itself are able to be compressed. The methods described herein perform efficient compression of the implicit representation of a point cloud given a target task.

Task-driven machine learning-based representation and compression of point cloud geometry

An electronic device and method for adaptive mode selection for point cloud compression, is provided. The electronic device receives a 3D point cloud geometry and partitions the 3D point cloud geometry into a set of 3D blocks. For a 3D block of the set of 3D blocks, mode decision information is determined. The mode decision information includes class information of the 3D point cloud geometry, operational conditions associated with an encoding stage of the 3D point cloud geometry, or mode-related information associated with one or more 3D blocks of the set of 3D blocks. Based on the mode decision information, one or more modes are selected for the 3D block from a plurality of modes. Each mode corresponds to a function that is used to encode the 3D block. The 3D block is encoded based on the one or more modes.

Adaptive mode selection for point cloud compression

Ways to simplify connectivity data for patches are described herein. The patches are generated considering the high-resolution mesh information. The connectivity data is simplified at the patch level, while the geometry image is still preserved. For the connectivity simplification, only triangles inside the patch are simplified. If the border is still preserved, the reconstruction in 3D will not suffer from artifacts. The high-resolution geometry image can be used to reverse the simplification and improve the connectivity at the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only.

Mesh patch simplification

A method of compressing meshes using a projection-based approach, leveraging and expanding the tools and syntax generated for projection-based volumetric content compression is described. The mesh is segmented into surface patches, with the difference that the segments follow the connectivity of the mesh. The dense mesh compression utilizes 3D surface patches to represent connected triangles on a mesh surface and groups of vertices to represent triangles not captured by surface projection. Each surface patch (or 3D patch) is projected to a 2D patch, whereby for the mesh, the triangle surface sampling is similar to a common rasterization approach. For each patch, position and connectivity of the projected vertices are kept. The sampled surface resembles a point cloud and is coded with the same approach used for point cloud compression. The list of vertices and connectivity per patch is encoded, and the data is sent with the coded point cloud data.

Dense mesh compression

An artificial intelligence (AI) based method for calibration of a gaming ecosystem and to maximize satisfaction of a player involved in a gaming session such as single-player gaming session, multi-player online gaming session, and a videogame recommendation system is provided. The system detects the gaming session that includes an execution of a video game for a first gameplay and acquires sensor data associated with a player involved in the first gameplay. The sensor data corresponds to a duration of the detected gaming session. The system determines one or more indicators of a dissatisfaction of the player with the first gameplay based on application of one or more Artificial Intelligence (AI) models on the sensor data. The system controls the execution of the video game to modify one or more aspects associated with the first gameplay or a second gameplay of the video game that is different from the first gameplay.

Enhancement of gameplay experience based on analysis of player data

A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content.

Growing community of inventors

San Jose, CA, United States of America

Danillo Graziosi