Sarnoff Corporation

Other

Lg Electronics Inc.

Ravi Krishnamurthy

Sriram Sethuraman

Ya-Qin Zhang

Tihao Chiang

Xudong Song

Xiaobing Lee

Paul Hatrack

Zudong Song

Apparatus and method for classifying regions of an image, based on the relative 'importance' of the various areas and to adaptively use the importance information to allocate processing resources and input image formation.

Method and apparatus for region-based allocation of processing resources and control of input image formation

A PC-type computer has a system bus (e.g., a PCI bus) configured with a main CPU board, a statistical multiplexing (stat-mux) board, and a plurality of video/audio encoder boards, each configured to receive and compress a corresponding video/audio stream. The stat-mux board performs statistical multiplexing on the different compressed bitstreams to transmit multiple bitstreams over individual shared communication channels. Although each of the boards is configured to the system bus, each encoder board has a digital signal processor (DSP) with a synchronized serial interface (SSI) output port that is directly connected to an SSI input port on a DSP on the stat-mux board (which, in one embodiment, has four such DSPs each with six such SSI input ports). As such, (up to 24) compressed video/audio bitstreams generated on the various encoder boards can be transmitted directly to the stat-mux board without having to go through the system bus. In this way, the computer system can provide statistical multiplexing of low-latency video/audio bitstreams without having to suffer the processing delays associated with conventional transmission over PCI system buses.

Computer system for statistical multiplexing of bitstreams

When two or more different video streams a e compressed for concurrent transmission of multiple compressed video bitstreams over a single shared communication channel, control over both (1) the transmission of data over the shared channel and (2) the compression processing that generates the bitstreams is exercised taking into account the differing levels of latency required for the corresponding video applications. For example, interactive video games typically require lower latency than other video applications such as video streaming, web browsing, and electronic mail. A multiplexer and traffic controller takes these differing latency requirements, along with bandwidth and image fidelity requirements, into account when controlling both traffic flow and compression processing. In addition, an off-line profiling tool analyzes typical video applications off-line in order to generate profiles of different types of video applications that are then accessed in real-time by a call admission manager responsible to controlling the admission of new video application sessions as well as the assignment of admitted applications to specific available video encoders, which themselves may differ in video compression processing power as well as in the degree to which they allow external processors (like the multiplexer and traffic controller) to control their internal compression processing.

Latency-based statistical multiplexing

An apparatus and a concomitant method for using M-ary pyramid decomposition in combination with N-scale tiling to reduce the computational complexity in determining motion vectors for block-based motion estimation is disclosed.

Apparatus and method for employing M-ary pyramids with N-scale tiling

For video compression processing, each frame in a video sequence is segmented into one or more different regions, where the macroblocks of each region are to be encoded using the same quantizer value, but the quantizer value can vary between regions in a frame. For example, for the videophone or video-conferencing paradigm of one or more &ldquo;talking heads&rdquo; in front of a relatively static background, each frame is segmented into a foreground region corresponding to the talking head, a background region corresponding to the static background, and an intervening transition region. An encoding complexity measure is generated for each macroblock of the previous frame using a (e.g., first-order) rate distortion model and the resulting macroblock-level encoding complexities are used to generate an average encoding complexity for each region. These region complexities are then used to select quantizer values for each region in the current frame, e.g., iteratively until the target bit rate for the frame is satisfied to within a specified tolerance range. The selected quantizer values may be modified based on spatial and/or temporal constraints to satisfy spatial requirements of the video compression algorithm and/or to provide temporal smoothness in quality, respectively.

Quantizer selection based on region complexities derived using a rate distortion model

A variety of different types of video frame encoders can be configured with, e.g., a multimedia processing subsystem, as long as the video frame encoder conforms to the interface protocol of the subsystem. A video controller in the subsystem performs the higher-level functions of coordinating the encoding of the video stream, thereby allowing the video frame encoder to limit its processing to the lower, frame level. In particular, the video controller provides information needed by the video frame encoder to encode the current frame in the video sequence. In addition to the raw image data, this information includes the type of frame to be encoded (e.g., an I or P frame), the currently available bandwidth for encoding the current frame, the time since the previous encoded frame, the desired frame rate, and a quality measure that may be used to trade off spatial and temporal qualities. The video frame encoder either encodes the frame as requested or indicates to the video controller that the frame should be skipped or otherwise not encoded as requested. The video controller can then respond appropriately, e.g., by requesting the video frame encoder to encode the next frame in the video sequence.

Frame-level rate control for plug-in video codecs

Apparatus and method for classifying regions of an image, based on the relative &ldquo;importance&rdquo; of the various areas and to adaptively use the importance information to allocate processing resources and input image formation.

A method and apparatus for encoding, illustratively, a video information stream to produce an encoded information stream according to a group of frames (GOF) information structure where the GOF structure and, optionally, a bit budget are modified in response to, respectively, information discontinuities and the presence of redundant information in the video information stream (due to, e.g., 3:2 pull-down processing).

Method and apparatus for encoding video information

An apparatus and method for reducing memory resource requirements in, e.g., an image processing system by utilizing a packed data pixel representation and, optionally, M-ary pyramid decomposition, for pixel block or pixel group searching and matching operations.

Method and apparatus for estimating motion using block features obtained from an M-ary pyramid

An apparatus and a concomitant method for using M-ary pyramid decomposition to reduce the computational complexity in determining motion vectors for block-based motion estimation is disclosed.

Apparatus and method for employing M-ary pyramids to enhance feature-based classification and motion estimation

Frames in a video sequence are divided into two or more regions and a specified number of macroblocks are selected in each region for intra-coding. Depending on the particular implementation, for one or more of the regions, the intra-macroblocks are selected randomly, while at least one other region is dividing into a specified number of slices with the least-recently intra-coded macroblock in each slice selected for intra-coding. When an error is detected at the decoder, the decoder discards data in the corresponding packet and applies a concealment strategy that involves using motion-compensated data if the motion vectors were accurately decoded; otherwise, using non-motion-compensated reference data for the macroblocks affected by the discarding of data. The refresh strategy of the present invention can be used to provide the resulting encoded bitstream with resilience to transmission errors, while maintaining an acceptable degree of video compression.

Region-based refresh strategy for video compression

Apparatus and method for classifying regions of an image, based on the relative &ldquo;importance&rdquo; of the various areas and to adaptively use the importance information to allocate processing resources, e.g., bit allocation in an encoding environment.

Method and apparatus for block classification and adaptive bit allocation

During video coding, a transform such as a discrete cosine transform (DCT) is applied to blocks of image data (e.g., motion-compensated interframe pixel differences) and the resulting transform coefficients for each block are quantized at a specified quantization level. Notwithstanding the fact that some coefficients are quantized to non-zero values, at least one non-zero quantized coefficient is treated as if it had a value of zero for purposes of further processing (e.g., run-length encoding (RLE) the quantized data). When segmentation analysis is performed to identify two or more different regions of interest in each frame, the number of coefficients that are treated as having a value of zero for RLE is different for different regions of interest (e.g., more coefficients for less-important regions). In this way, the number of bits used to encode image data are reduced to satisfy bit rate requirements without (1) having to drop frames adaptively, while (2) conforming to constraints that may be imposed on the magnitude of change in quantization level from frame to frame.

Treating non-zero quantized transform coefficients as zeros during video compression processing

An image is divided into one or more (e.g., foreground) regions of interest with transition regions defined between each region of interest and the relatively least-important (e.g., background) region. Each region is encoded using a single selected quantization level, where quantizer values can differ between different regions. In general, in order to optimize video quality while still meeting target bit allocations, the quantizer assigned to a region of interest is preferably lower than the quantizer assigned to the corresponding transition region, which is itself preferably lower than the quantizer assigned to the background region. The present invention can be implemented iteratively to adjust the quantizer values as needed to meet the frame's specified bit target. The present invention can also be implemented using a non-iterative scheme that can be more easily implemented in real time. The present invention enables a video compression algorithm to meet a frame-level bit target, while ensuring spatial and temporal smoothness in frame quality, thus resulting in improved visual perception during playback.

Intra-frame quantizer selection for video compression

An apparatus and a concomitant method for performing hierarchial block-based motion estimation with a high degree of scalability is disclosed. The present invention decomposes each of the image frames within an image sequence into an M-ary pyramid. Different dynamic ranges for representing the pixel values are used for different levels of the M-ary pyramid, thereby generating a plurality of different &ldquo;P-bit&rdquo; levels, i.e., a plurality of different M-ary pyramid architectures. The present scalable hierarchical motion estimation provides the flexibility of switching from one M-ary pyramid architecture to another M-ary pyramid architecture according to the available platform resources and/or user's choice.

Growing community of inventors

Princeton, NJ, United States of America

Ravi Krishnamurthy