Nvidia Corporation

Hassane S Azar

Franck Diard

Garry W Amann

Stephen Donald Lew

Stefan Eckart

Vinayak Jayaram Pore

Dawid Stanislaw Pajak

Xun Wang

Amit Parikh

Bipin Todur

Reza Marandian Hagh

Alan Daniel Larson

Keshava Prasad

Prashant Sohani

Bryan Dudash

Haitao Xue

Swagat Mohapatra

Alexander McAuley

Henry Packard Moreton

Duncan Andrew Riach

John F Spitzer

Douglas H Rogers

Michael L Lightstone

Tony Tamasi

Rochelle Pereira

Mark M Vojkovich

Alok Ahuja

Ravi Kumar Boddeti

Mihai Sipitca

Douglas Sim Dietrich, Jr

Ajit Lalwani

Rudragouda Pharale

Diksha Garg

Franck R Diard

In various examples, a deep neural network (DNN) based pre-filter for content streaming applications is used to dynamically adapt scene entropy (e.g., complexity) in response to changing network or system conditions of an end-user device. For example, where network and/or system performance issues or degradation are identified, the DNN may be implemented as a frame pre-filter to reduce the complexity or entropy of the frame prior to streaming—thereby allowing the frame to be streamed at a reduced bit rate without requiring a change in resolution. The DNN-based pre-filter may be tuned to maintain image detail along object, boundary, and/or surface edges such that scene navigation—such as by a user participating in an instance of an application—may be easier and more natural to the user.

Entropy-based pre-filtering using neural networks for streaming applications

In various examples, machine learning of encoding parameter values for a network is performed using a video encoder. Feedback associated with streaming video encoded by a video encoder over a network may be applied to an MLM(s). Using such feedback, the MLM(s) may predict a value(s) of an encoding parameter(s). The video encoder may then use the value to encode subsequent video data for the streaming. By using the video encoder in training, the MLM(s) may learn based on actual encoded parameter values of the video encoder. The MLM(s) may be trained via reinforcement learning based on video encoded by the video encoder. A rewards metric(s) may be used to train the MLM(s) using data generated or applied to the physical network in which the MLM(s) is to be deployed and/or a simulation thereof. Penalty metric(s) (e.g., the quantity of dropped frames) may also be used to train the MLM(s).

Machine learning of encoding parameters for a network using a video encoder

Systems and methods are disclosed for dynamically smoothing images based on network conditions to adjust a bitrate needed to transmit the images. Content in the images is smoothed to reduce the quantity of bits needed to encode each image. Filtering the images modifies regions including content having a high frequency of pixel variation, reducing the frequency, so the pixel colors in the region appear 'smoothed' or homogeneous. In other words, a region of an image showing a grassy lawn has a high frequency of variation from pixel to pixel resulting from the fine detail of separate blades of grass that may be similar in color, but not homogeneous. Encoding the region as a single shade of green (or multi-pixel regions of different shades of green) enables a viewer to recognize it as a grassy lawn while greatly reducing the number of bits needed to represent the region.

Dynamic image smoothing based on network conditions

Apparatuses, systems, and techniques for isolating the performance of a quality-of-service (QoS) policy for improved data streaming systems and applications. In at least one embodiment, a metric is determined for a QoS policy used to provide an application session based on a value of at least one characteristic of the application session that reflects an impact of one or more external conditions beyond the control of the QoS policy.

Isolating QoS policy performance for improved data streaming systems and applications

Embodiments of the present invention provide a low-latency approach for local or remote application streaming that reaches high FPS targets without overloading the available streaming bandwidth, for example, by limiting the bit rate to the same value that is used by traditional 60 FPS streaming solutions. A client device and server device cooperate to actively monitor and control a video stream to maintain an acceptable balance between latency and video quality by adjusting the frequency or resolution when necessary to improve the streaming experience. When the server device captures and transmits frames at a higher rate, the software stack executing on the client device is able to display frames with less delay, even on a display device limited to 60 Hz, thereby achieving additional latency reduction.

Method and system for low latency high frame rate streaming

A computer application streaming system includes an optimization unit coupled to a streaming device to determine streaming optimal playable settings for a remote user device corresponding to a selected computer application and a sending unit coupled to the optimization unit to manage streaming of the streaming optimal playable settings over a network connected to the remote user device. A receiving unit is coupled to the network to recover the streaming optimal playable settings for application to the remote user device when employing the selected computer application. An optional feedback unit is coupled to the remote user device to provide remote information over the network for modifying the streaming optimal playable settings, and an optional update unit is coupled to the streaming device to manage modification of the streaming optimal playable settings as directed by the remote information. A method of streaming a computer application is also provided.

Optimal settings for application streaming

A video encoder, a method of encoding a frame of video data, and a three-dimensional modeling system producing an encoded video stream are disclosed herein. In one embodiment, the method includes: (1) receiving from an application a frame of video data to be encoded, (2) determining a gamer's attention area for the frame of video data and (3) changing an encoding of the frame of video data by allocating bits for the frame based upon the gamer's attention area.

Rate control bit allocation for video streaming based on an attention area of a gamer

A system and method are provided for a 3D modeling system with which an encoded video stream is produced. The system includes a content engine, an encoder, and a fixed function engine. The fixed function engine receives content information from the content engine. The fixed function engine produces encoder information from the content information. The encoder uses the encoder information to produce an encoded video stream having at least one of a higher quality and a lower bandwidth than a video stream encoded without the encoder information.

System and method for controlling video encoding using content information

A system and method are provided for improving video encoding using content information. A three-dimensional (3D) modeling system produces an encoded video stream. The system includes a content engine, a renderer, and a video encoder. The renderer receives 3D model information from the content engine relating and to produces corresponding two-dimensional (2D) images. The video encoder receives the 2D images and produce a corresponding encoded video stream. The video encoder receives content information from the content engine, transforms the content information into encoder control information, and controls the video encoder using the encoder control information.

System and method for improving video encoding using content information

A video frame compression system includes a rendering engine that provides a current video frame and current additional rendering information. Additionally, the video frame compression system includes a warping engine that generates a warped video frame, wherein the warped video frame is a transformation of a previous video frame that is based on the current additional rendering information. Further, the video frame compression system includes a video encoder that compresses the current video frame by using the warped video frame as a reference frame and separately compresses the current additional rendering information. Still further, the video frame compression system includes a packetizer that provides main and auxiliary data streams corresponding to the compressed current video frame and the compressed current additional rendering information, respectively. A video frame decompression system and methods of video frame compression and decompression are also provided.

Extending prediction modes and performance of video codecs

Video filtering using a programmable graphics processor is described. The programmable graphics processor may be programmed to complete a plurality of video filtering operations in a single pass through a fragment-processing pipeline within the programmable graphics processor. Video filtering functions such as deinterlacing, chroma up-sampling, scaling, and deblocking may be performed by the fragment-processing pipeline. The fragment-processing pipeline may be programmed to perform motion adaptive deinterlacing, wherein a spatially variant filter determines, on a pixel basis, whether a 'bob', a 'blend', or a &#x201c;weave&#x201d; operation should be used to process an interlaced image.

Method and apparatus for filtering video data using a programmable graphics processor

A system is presented that is configured to reduce power consumption when performing processing tasks. The system includes a first processing entity capable of performing a set of operations, and a second processing entity configured to consume less power than the first processing entity and capable of performing a subset of operations that is part of the set of operations. During system operation, the second processing entity is configured to perform the subset of operations instead of the first processing entity.

Picture processing using a hybrid system configuration

A system and method for adjusting pictures minimizes the impact on graphics processing performance of a discrete processor. A hybrid system configuration includes the discrete processor and an integrated processor, where the discrete processor typically consumes more power and provides greater processing performance compared with the integrated processor. A picture is produced by a video or graphics engine of a discrete processor within a hybrid system. Each picture is then transferred to a back buffer in the host processing memory. The picture is analyzed to produce picture analysis results that are used to generate adjustment settings. The back buffer is swapped to become the front buffer and the adjustment settings are applied to the picture by an integrated processor to display an adjusted picture. The adjustment may be used in conjunction with power saving techniques to maintain the image quality when display backlighting is reduced.

Split-frame post-processing techniques are used in a programmable video post processing engine. A frame of video data is divided into a processing region and a control region that contain either different portions of the frame or copies of a portion of the frame. Post-processing operations are performed for the processing region but not for the control region. The processing and control regions are then displayed simultaneously on the same display device (e.g., side by side), facilitating visual comparisons of images with and without post-processing.

Growing community of inventors

Santa Clara, CA, United States of America

Hassane S Azar