At&t Bell Laboratories

Lucent Technologies Inc.

At+t Corp.

Robert J Safranek

Arun N Netravali

Eric D Petajan

Kim N Matthews

Scott C Knauer

Peter H Westerink

James David Johnston

Jelena Kovacevic

Jianying Hu

Aleksandra Mojsilovic

S Kicha Ganapathy

David L Neuhoff

Thrasyvoulos N Pappas

Nambirajan Seshadri

Nuggehally Sampath Jayant

Arnaud Eric Jacquin

John Hartung

Yi-Jen Chiu

Scott C Knaur

Edmund Meng Yeh

The invention provides a perceptually-based system for pattern retrieval and matching, suitable for use in a wide variety of information processing applications. An illustrative embodiment of the system uses a predetermined vocabulary comprising one or more dimensions to extract color and texture information from an information signal, e.g., an image, selected by a user. The system then generates a distance measure characterizing the relationship of the selected image to another image stored in a database, by applying a grammar, comprising a set of predetermined rules, to the color and texture information extracted from the selected image and corresponding color and texture information associated with the stored image. The vocabulary may include dimensions such as overall color, directionality and orientation, regularity and placement, color purity, and pattern complexity and heaviness. The rules in the grammar may include equal pattern, overall appearance, similar pattern, and dominant color and general impression, with each of the rules expressed as a logical combination of values generated for one or more of the dimensions. The distance measure may include separate color and texture metrics characterizing the similarity of the respective color and texture of the two images being compared. The invention is also applicable to other types of information signals, such as sequences of video frames.

Retrieval and matching of color patterns based on a predetermined vocabulary and grammar

A method of encoding a video sequence including a sequence of video images includes first comparing elements of a portion of a first video image (e.g., pixels of a macroblock of a current frame) with elements of a portion of a second video image (e.g., corresponding pixels of a macroblock of a previous frame) to generate respective intensity difference values for the element comparisons. Then, a first value (e.g., one) is assigned to the intensity difference values that are above a visually perceptible threshold value and a second value (e.g., zero) is assigned to the intensity difference values that are at or below the visually perceptible threshold value. Next, the method includes summing the first and second values to generate a sum. If the sum is greater than a predetermined decision value, the portion of the first video image is encoded (e.g., motion compensated). The method is fully compatible and, thus, may be implemented with video standards such as, for example, H.261, H.263, Motion-JPEG, MPEG-1, and MPEG-2.

Perceptual-based spatio-temporal segmentation for motion estimation

An image-processing system for perceptual coding of an image is disclosed. Coding is accomplished through an analysis of human visual sensitivity to noise in halftone images and an analysis of one or more signals representing the image to be coded. These analyses determine levels of noise. A first image is encoded so as to produce encoded values without introducing noise which exceeds a determined level of noise. Analysis of human visual sensitivity is carried out under a set of user determined conditions comprising viewing distance and lighting. The encoded image is communicated and decoded to produce representation of the first image for display. As part of displaying the representation of the first image, a halftoning process is carried out. The halftoning process may comprise the interpolation of data values of the representation of the first image as well as the addition of micro-dither to the representation. Illustrative halftoning techniques include classical halftoning, least-squares halftoning, and error diffusion halftoning. Printer model-based halftoning techniques for both least-squares halftoning and error diffusion halftoning are presented. Applications of the model-based halftoning techniques for systems employing conventional coding of images are also discussed.

Image processing system

Method and apparatus are disclosed for deinterlacing of an interlaced video frame sequence using interpolation estimations, such as spatial and temporal interpolations. Interpolations requiring a less accurate estimation of missing pixel values in the frames being deinterlaced, such that an interpolation may be performed with a minimum of error, are performed before interpolations which require a more accurate estimation of missing pixel values for performing an interpolation, such that estimates of missing pixel values are obtained with a minimum of error. Interpolation estimations are weighted in combination for computing approximations of missing pixel values in accordance with the errors associated with the respective interpolations.

Method and apparatus for converting an interlaced video frame sequence

Compression of signals is achieved through a simple decision of whether or not to encode certain frequency bands; not how well to encode all of the frequency bands. Based on the input signal and on a preselected perceptual model, a 'just noticeable difference' (jnd) noise spectrum is computed. This spectrum is applied to a selector where it is used in the decision to select a chosen number of frequency bands of the input signal. The bands selected are the bands with the greatest energy relative to the jnd energy in the same band. Each of the selected bands is encoded and transmitted to the receiver. Both analog and digital realizations are presented.

Compression of signals for perceptual quality by selecting frequency

An image-coding system reduces image data redundancies and perceptual irrelevancies through progressive sub-band coding. The image is separated into a plurality of sub-bands. From this sub-band information, a perceptual metric, based on the properties of the sub-band filters, quantizer error distribution, and properties of the human visual system, is determined which provides the maximum amount of coding noise that may be introduced to each pixel in every sub-band without causing perceptible degradation of the coded image. This perceptual metric is used to adjust the quantizer used in encoding each sub-band signal. In addition, redundancy in the output of the quantizer is reduced using a multidimensional Huffman compression scheme.

Perceptually-adapted image coding system

An improved predictive encoder where the leak signal is a function of the buffer fullness of the encoder. More specifically, the signals stored in the encoder output buffer are further encoded based on the fullness of the buffer, and information about this further encoding is used in determining the leak factor level. In accordance with another improvement, this leak factor level is not constrained to granularity that is imposed by the decoder hardware. Removal of the constraint is accomplished by cycling through a sequence of permissible leak levels that averages at the desired level.

Adaptive leak HDTV encoder

An image-processing system for perceptual coding of images by analyzing one or more images to determine levels of imperceptible noise and encoding a first image to produce encoded values without introducing noise exceeding a determined level of imperceptible noise is disclosed. Analyzing of images is carried out under a set of user determined conditions comprising viewing distance and lighting. The encoded image is communicated and decoded to produce representation of the first image for display. As part of displaying the representation of the first image, a halftoning process is carried out. The halftoning process may comprise the interpolation of data values of the representation of the first image as well as the addition of micro-dither to the representation. Illustrative halftoning techniques include classical halftoning, least-squares halftoning, and error diffusion halftoning. Printer model-based halftoning techniques for both least-squares halftoning and error diffusion halftoning are presented. Applications of the model-based halftoning techniques for systems employing conventional coding of images are also discussed.

The effects of perturbations in a decoder responsive to a signal encoded in a predictive coding schema are reduced by shortening the time needed to converge the decoder's prediction signal to that of the encoder. This is accomplished by temporarily altering the leak factor when it is known that a perturbation is about to manifest itself and by, in some situations, altering the incoming encoded signal itself.

HDTV receiver

A motion compensated encoder where motion vectors are selected based on the prediction error generated in localized areas of the encoded image and based on an available bit budget. The motion vectors are created by dividing the image into blocks of two sizes and by considering the best mix of large and small size blocks, and their associated motion vectors, that minimize the overall prediction error, within the constraints of the bit budget. For convenience, the image division is arranged so that a given number of small sized blocks forms one large sized block (e.g. 16:1). Also, the block sizes are arranged so that employing only large sized blocks does not exceed the given bit budget, while employing only the small sized blocks does exceed the given bit budget.

HDTV encoder with forward estimation and constant rate motion vectors

A quantizer, with quantization control that is sensitive to input signal characteristics and to output buffer fullness responds to an input signal that is divided into blocks and DCT transformed. The transformed signal is analyzed to develop a brightness correction and to evaluate the texture of the image and the change in texture in the image. Based on these, and in concert with the human visual perception model, perception threshold signals are created for each subband of the transformed signal. Concurrently, scale factors for each subband of the transformed signal are computed, and a measure of variability in the transformed input signal is calculated. A measure of the fullness of the buffer to which the quantizer sends its encoded results is obtained, and that measure is combined with the calculated signal variability to develop a correction signal. The correction signal modifies the perception threshold signals to develop threshold control signals that are applied to the quantizer. The scale factors are also applied to the quantizer, as well as a global target distortion measure.

Adaptive non-linear quantizer

In a differential PCM encoder, the problem of error perpetuation is solved by leaking a changing, rather than a fixed, fraction of the input signal to the differential PCM. The fraction leaked is sensitive to the characteristics of the signal. In one embodiment the fraction leaked is fixed for a frame in accordance with a chosen characteristic of the frame signal. In another embodiment, the fraction leaked is set in accordance with one function when a chosen characteristic of the frame signal exceeds a given level, and follows another function when the chosen characteristic does not exceed the chosen level. In a still another embodiment, the fraction leaked is set to one of two levels, based on a chosen characteristic of the frame signal.

An HDTV receiver design includes a reconstruction section and an inner loop section. The reconstruction section comprises a receiving section for accepting the television receiver's antenna signals, for separating out the component signals from the received signals and for decoding the separated signals. The decoded signals are applied to a quantization decoder that is responsive to vector codebook and to applied quantized vector signals, and the output signals of the quantization decoder are applied to an inverse DCT circuit. The inner loop comprises an adder for adding an estimate signal to the output of the DCT circuit, a frame buffer, a motion compensation circuit that is capable of translating large blocks as well as small blocks and a leak circuit that modulates the output of the motion compensator circuit in accordance with a received leak control signal. The input to the frame buffer is also applied to conventional amplification and display circuits to form the familiar output of a television receiver.

Growing community of inventors

New Providence, NJ, United States of America

Robert J Safranek