Sony Group Corporation

Sony Pictures Entertainment, Inc.

Nurulize, Inc.

Tobias Anderberg

Scott Metzger

Malik Williams

David Bailey

Brad Hunt

Aligning and coloring a volumetric image, including: capturing intensity data using at least one scanner; generating an intensity image using the intensity data, wherein the intensity image includes at least one feature in a scene, the at least one feature including a sample feature; capturing image data using at least one camera, wherein the image data includes color information; generating a camera image using the image data, wherein the camera image includes the sample feature; matching the sample feature in the intensity image with the sample feature in the camera image to align the intensity image and the camera image; and generating a color image by applying the color information to the aligned intensity image.

Trans-spectral feature detection for volumetric image alignment and colorization

Reducing data used during capture in a physical capture volume by selectively activating image capture devices from a virtual view, including: setting up a virtual camera to receive information about and visualize the physical capture volume and a plurality of image capture devices in the virtual view; providing, to the virtual camera, the virtual view of the physical capture volume with a capability to move around the physical capture volume and activate or deactivate each of the plurality of image capture devices; calculating a view frustum, wherein the view frustum is a region of 3-D space within the physical capture volume that would appear on a view screen of the virtual camera; and defining the view frustum of the virtual camera which intersects with the plurality of image capture devices defined in the virtual view.

View-independent multi-camera volumetric capture system

Managing cameras and controlling video display, including: identifying a first group of cameras, each camera in the first group having a respective image stream; identifying a second group of cameras, each camera in the second group having a respective image stream, and each camera in the second group is not in the first group; assigning a name to each group; assigning a name to each camera; displaying the name of each group in a user interface on a computer system; displaying the name of each camera in the user interface; receiving a selection of a group through the user interface; displaying the image stream for each camera in the selected group simultaneously; receiving a selection of one camera through the user interface; and displaying the image stream for the selected camera.

Efficient user interface navigation for multiple real-time streaming devices

Capturing and visualizing video, including: capturing video data using a plurality of cameras; sending the captured video data to a first shader; calculating depth information at the first shader using the captured video data; generating a three-dimensional (3-D) point cloud using the depth information; and rendering a visualization image using the 3-D point cloud.

Real-time volumetric visualization of 2-D images

Automatically shifting between virtual lens parameters and optical lens parameters for an optical camera positioned within a physical scene, including: providing a spatial awareness of the physical scene including position and orientation of a display screen to the optical camera; creating a feedback loop between the optical camera and lens to enable the lens to communicate lens settings to a scene registration; determining when focus of the lens is beyond an optical focal limit from the lens settings; and enabling the lens, by the scene registration, to automatically shift from using the optical lens parameters to using the virtual lens parameters when the focus of the lens moves beyond the optical focal limit.

Progressive morphological lens parameter encoding

Enabling colorization and color adjustments on 3D point clouds, which are projected onto a 2D view with an equirectangular projection. A user may color regions on the 2D view and preview the changes immediately in a 3D view of the point cloud. Embodiments render the color of each point in the point cloud by testing whether the 2D projection of the point is inside the colored region. Applications may include generation of a color 3D virtual reality environment using point clouds and color-adjusted imagery.

Point cloud colorization system with real-time 3D visualization

Managing volumetric data, including: defining a view volume in a volume of space, wherein the volumetric data has multiple points in the volume of space and at least one point is in the view volume and at least one point is not in the view volume; defining a grid in the volume of space, the grid having multiple cells and dividing the volume of space into respective cells, wherein each point has a corresponding cell in the grid, and each cell in the grid has zero or more corresponding points; and reducing the number of points for a cell in the grid where that cell is outside the view volume.

Reducing volumetric data while retaining visual fidelity

Managing device settings on multiple devices, including: creating a first group of devices including two or more devices, wherein each device in the first group has at least one first device setting; creating a second group of devices including two or more devices, wherein each device in the second group has at least one second device setting and each device in the second group is not in the first group; sending the at least one first device setting to each device in the first group of devices in parallel, so that each device in the first group changes at least one device setting according to the at least one received first device setting; and sending the at least one second device setting to each device in the second group of devices in parallel, so that each device in the second group changes at least one device setting according to the at least one received second device setting.

Managing multiple image devices

Method that merges two or more point clouds captured from a scene, eliminates redundant points, and retains points that best represent the scene. The method may generally include a detection step, which locates points from different clouds that are close and thus potentially redundant, followed by a selection step that identifies preferred points. Clouds may be represented as range images, which may simplify both steps. Closeness testing may be optimized by dividing range images into tiles and testing tile bounding volumes for intersections between clouds. Selection of preferred points may incorporate user input, or it may be fully or partially automated. User selection may be performed using 2D drawing tools on range images to identify images with preferred views of a scene. Automated selection may assign a quality measure to points based for example on the surface resolution of each point cloud scan at overlapping points.

Method of merging point clouds that identifies and retains preferred points

System that enables colorization and color adjustments on 3D point clouds. Point clouds are projected onto a 2D view, for example with an equirectangular projection. A user may color regions on the 2D view, and preview the changes immediately in a 3D view of the point cloud without a time-consuming update of the point cloud data. Because any 2D projection introduces distortions, 3D preview is important for the user to observe the effects of the color changes on the point cloud. Embodiments may for example render the color of each point in the point cloud by testing whether the 2D projection of the point is inside the colored region. Alternatively, the region may be triangulated, and a GPU may render a texture image and use this texture to render the 3D view. Applications may include generation of a color 3D virtual reality environment using point clouds and color-adjusted imagery.

Growing community of inventors

Los Angeles, CA, United States of America

Tobias Anderberg