Nvidia Corporation

Dawid Stanislaw Pajak

Jan Kautz

Hassane S Azar

Kari Pulli

David Patrick Luebke

Niranjan Avadhanam

Orazio Gallo

Yuzhuo Ren

Stefan Eckart

Felix Heide

Nagilla Dikpal Reddy

Mushfiqur Rouf

Bryan Dudash

Swagat Mohapatra

Morgan McGuire

Michael Stengel

Markus Steinberger

Yun-Ta Tsai

Fu-Chung Huang

Rochelle Pereira

Ben Boudaoud

Scott Saulters

Alexander Majercik

Jongmin Baek

Guangli Dai

In various examples, color harmonization is applied to images of an environment in a reference light space. For example, different cameras on an ego-object may use independent capturing algorithms to generate processed images of the environment representing a common time slice using different capture configuration parameters. The processed images may be transformed into deprocessed images by inverting one or more stages of image processing to transform the processed images into a reference light space of linear light, and color harmonization may be applied to the deprocessed images in the reference light space. After applying color harmonization, corresponding image processing may be reapplied to the harmonized images using corresponding capture configuration parameters, the resulting processed harmonized images may be stitched into a stitched image, and a visualization of the stitched image may be presented (e.g., on a monitor visible to an occupant or operator of the ego-object).

Image stitching with color harmonization of de-processed images for surround view systems and applications

In various examples, color statistic(s) from ground projections are used to harmonize color between reference and target frames representing an environment. The reference and target frames may be projected onto a representation of the ground (e.g., a ground plane) of the environment, an overlapping region between the projections may be identified, and the portion of each projection that lands in the overlapping region may be taken as a corresponding ground projection. Color statistics (e.g., mean, variance, standard deviation, kurtosis, skew, correlation(s) between color channels) may be computed from the ground projections (or a portion thereof, such as a majority cluster) and used to modify the colors of the target frame to have updated color statistics that match those from the ground projection of the reference frame, thereby harmonizing color across the reference and target frames.

Image stitching with color harmonization for surround view systems and applications

A remote device utilizes ray tracing to compute a light field for a scene to be rendered, where the light field includes information about light reflected off surfaces within the scene. This light field is then compressed utilizing lossless or lossy compression and one or more video compression techniques that implement temporal reuse, such that only differences between the light field for the scene and a light field for a previous scene are compressed. The compressed light field data is then sent to a client device that decompresses the light field data and uses such data to obtain the light field for the scene at the client device. This light field is then used by the client device to compute global illumination for the scene. The global illumination may be used to accurately render the scene at the mobile device, resulting in a realistic scene that is presented by the mobile device.

Streaming a light field compressed utilizing lossless or lossy compression

A method, computer readable medium, and system are disclosed for generating mixed-primary data for display. The method includes the steps of receiving a source image that includes a plurality of pixels, dividing the source image into a plurality of blocks, analyzing the source image based on an image decomposition algorithm, encoding chroma information and modulation information to generate a video signal, and transmitting the video signal to a mixed-primary display. The chroma information and modulation information correspond with two or more mixed-primary color components and are generated by the image decomposition algorithm to minimize error between a reproduced image and the source image. The two or more mixed-primary colors selected for each block of the source image are not limited to any particular set of colors and each mixed-primary color component may be selected from any color capable of being reproduced by the mixed-primary display.

Mixed primary display with spatially modulated backlight

A system and method are provided for a 3D modeling system with which an encoded video stream is produced. The system includes a content engine, an encoder, and a fixed function engine. The fixed function engine receives content information from the content engine. The fixed function engine produces encoder information from the content information. The encoder uses the encoder information to produce an encoded video stream having at least one of a higher quality and a lower bandwidth than a video stream encoded without the encoder information.

System and method for controlling video encoding using content information

A system and method are provided for improving video encoding using content information. A three-dimensional (3D) modeling system produces an encoded video stream. The system includes a content engine, a renderer, and a video encoder. The renderer receives 3D model information from the content engine relating and to produces corresponding two-dimensional (2D) images. The video encoder receives the 2D images and produce a corresponding encoded video stream. The video encoder receives content information from the content engine, transforms the content information into encoder control information, and controls the video encoder using the encoder control information.

System and method for improving video encoding using content information

A computer implemented method of determining a latent image from an observed image is disclosed. The method comprises implementing a plurality of image processing operations within a single optimization framework, wherein the single optimization framework comprises solving a linear minimization expression. The method further comprises mapping the linear minimization expression onto at least one non-linear solver. Further, the method comprises using the non-linear solver, iteratively solving the linear minimization expression in order to extract the latent image from the observed image, wherein the linear minimization expression comprises: a data term, and a regularization term, and wherein the regularization term comprises a plurality of non-linear image priors.

Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image

A technique for efficiently compressing rendered three-dimensional images in a remote rendering system adds a novel render-assisted prediction function to an existing video compression framework, such as the standard H.264/5 framework. Auxiliary rendering information is separated from rendering information used to describe a reference image by a server system. A client system may alter the auxiliary data and generate a new image based on the reference image and rendered scene information from the auxiliary data without creating additional network bandwidth or server workload.

Render-assisted compression for remote graphics

A video frame compression system includes a rendering engine that provides a current video frame and current additional rendering information. Additionally, the video frame compression system includes a warping engine that generates a warped video frame, wherein the warped video frame is a transformation of a previous video frame that is based on the current additional rendering information. Further, the video frame compression system includes a video encoder that compresses the current video frame by using the warped video frame as a reference frame and separately compresses the current additional rendering information. Still further, the video frame compression system includes a packetizer that provides main and auxiliary data streams corresponding to the compressed current video frame and the compressed current additional rendering information, respectively. A video frame decompression system and methods of video frame compression and decompression are also provided.

Extending prediction modes and performance of video codecs

A computer implemented method of performing an approximate-nearest-neighbor search is disclosed. The method comprises dividing an image into a plurality of tiles. Further, for each of the plurality of tiles, perform the following in parallel on a processor: (a) dividing image patches into a plurality of clusters, wherein each cluster comprises similar images patches, and wherein the dividing continues recursively until a size of a cluster is below a threshold value; (b) performing a nearest-neighbor query within each of the plurality of clusters; and (c) performing collaborative filtering in parallel for each image patch, wherein the collaborative filtering aggregates and processes nearest neighbor image patches from a same cluster containing a respective image patch to form an output image.

Efficient approximate-nearest-neighbor (ANN) search for high-quality collaborative filtering

An approach is provided for performing back-end operations for camera control. In one example, a method includes the following: receiving a user edit via a user interface device that displays an interpretation of a scene at which a camera lens of the camera is pointing, wherein the user edit is based on user input that is associated with a selection region on the user interface device; generating an edits mask based on one or more matching image patches, which are based on the user edit and a high dynamic range (HDR) image generated by the camera; performing one or more tone mapping operations based on the edits mask and the HDR image in order to generate a tone mapped HDR image; and performing one or more metering operations based on the edits mask and the tone mapped HDR image in order to generate metering parameters for frame capturing operations.

Approach for camera control

A system and method are provided for performing joint color and depth encoding. Color data and depth data for an image is received. Based on the color data, confidence values are computed for the depth data and the depth data is encoded based on the confidence values to represent a correlated portion of the depth data and a decorrelated portion of the depth data. In one embodiment, the depth data comprises per-pixel vergence angles.

Growing community of inventors

Mountain View, CA, United States of America

Dawid Stanislaw Pajak