British Telecommunications Public Limited Company

British Telecommunications Corporation

Li-Qun Xu

Bangjun Lei

David G Morrison

Jason Morphett

Charles Nightingale

Yongmin Li

David Geoffrey Morrison

Ebroul Izquierdo

Jose-Luis Landabaso

Jianmin Jiang

Alan Hanjalic

Sergio Benini

Pere Puig Folch

Bin Luo

Arasanathan Anjulan

Andrea Cavallaro

Banjun Lei

Global motion between frames of a motion-compensated inter-frame encoded video sequence is estimated directly from the motion vectors encoded within the frames. For any particular frame, the motion vectors are first decoded, and a finite number of sets of vectors are selected. An affine or other geometrical transform is then used to generate a motion estimation for each set, and then the least median squared error present in each motion estimation is calculated for each estimation. The motion estimation with the smallest least median squared error is then selected as being representative of the global motion in the image of the frame. A panoramic image generating method and system makes uses of the global motion estimations thus obtained.

Method and system for estimating global motion in video sequences

A method of segmenting a sequence of video images according to scene activity, the method comprising: defining a first series of nodes in a first multi-dimensional space, each node corresponding to an image of the sequence of video images; defining a transformation function that maps each of the first series of nodes to a corresponding node in a second multi-dimensional space having a lower dimensionality than the first multi-dimensional space; applying said transformation function to each of the first series of nodes to define a second series of respective nodes in the second multi-dimensional space; applying a data clustering algorithm to the second series of nodes to identify clusters of nodes within the second multi-dimensional space, the data clustering algorithm being constrained by a measure of feature distance between a pair of clusters of nodes and a measure of temporal distance between the pair of clusters of nodes; determining a representative image from each cluster of nodes and plotting each representative image with respect to a measure of the elapsed time of the sequence of video images to form an scene density curve indicating the underlying scene change activities; and segmenting the sequence of video images in accordance with local minima and/or maxima of the scene density curve.

Video segmentation

Video data defining a series of images is processed to define a first series of nodes in a first multidimensional space, each of the first series of nodes corresponding to an image of the series of images and its location in the first space defined in dependence on features of the respective image. A transformation function maps each of the nodes in the first multidimensional space onto a respective node in a second lower dimensionality multidimensional space while maintaining neighborhood relationships between nodes. A second series of nodes in the second multidimensional space is defined in accordance with the transformation function and a clustering analysis is performed in dependence on the nodes of the second multidimensional space.

Video abstraction

The invention relates to estimating the global motion between frames of a motion-compensated inter-frame encoded video sequence, directly from the motion vectors encoded within the frames. For any particular frame, a motion estimation is determined from motion vectors direct from the frame's anchor frame to the frame in question. This motion estimation is then checked against pre-determined criteria, and where the criteria are not met, re-estimation along a different route is performed, using the bi-directional motion vectors contained within B-frames. A panoramic image generating method and system which makes uses of the global motion estimations thus obtained is also described.

In a digital video surveillance system, a number of processing stages are employed to identify foreground regions representing moving objects in a video sequence. An object tracking stage () is also provided in order to identify a correspondence between candidate objects in a current frame and those that have already been identified in one or more previous frames. In this way, it is possible to calculate the path taken by the or each foreground object and to record this path information in a trajectory database. In order to improve tracking performance, the object tracking stage () employs a state transitional object management scheme which determines whether or not a particular object is tracked. The path information generated by the object tracking stage () can be displayed on a video monitor () of the surveillance system and/or made available to high level applications ().

Classifying an object in a video frame

A video surveillance system () comprises a camera (), a personal computer (PC) () and a video monitor (). Video processing software is provided on the hard disk drive of the PC (). The software is arranged to perform a number of processing operations on video data received from the camera, the video data representing individual frames of captured video. In particular, the software is arranged to identify one or more foreground blobs in a current frame, to match the or each blob with an object identified in one or more previous frames, and to track the motion of the or each object as more frames are received. In order to maintain the identity of objects during an occlusion event, an appearance model is generated for blobs that are close to one another in terms of image position. Once occlusion takes place, the respective appearance models are used, in combination with a depth factor representing the depth order of the occluded objects, to segment the resulting group blob into regions which are classified as representing one or other of the merged objects.

Method of tracking objects in a video sequence

In a digital video surveillance system, a number of processing stages are employed to identify foreground regions representing moving objects in a video sequence. An object tracking stageis also provided in order to identify a correspondence between candidate objects in a current frame and those that have already been identified in one or more previous frames. In this way, it is possible to calculate the path taken by the or each foreground object and to record this path information in a trajectory database. In order to improve tracking performance, the object tracking stageemploys a state transitional object management scheme which determines whether or not a particular object is to be tracked. As part of the object management scheme, spurious objects, which are identified on the basis of their motion characteristics, are deleted from the system. This ensures that valuable processing resources are not wasted tracking unwanted artifacts which may represent, for example, noise or random motion.

Identifying spurious regions in a video frame

The invention provides a method and apparatus for performing correspondence estimation between pixels of a stereo image pair to obtain matching information for corresponding pixels in each image. To perform a match for a particular pixel in a first image firstly an adaptive curve is constructed about the pixel, being a sequence of connected pixels with similar intensity values to the pixel being matched. The adaptive curve thus constructed is then used as a matching element within the second image to find a matching pixel representative of the same 3D scene point in the second image to the particular pixel. By performing matching in this manner for every pixel in an image, accurate disparity maps can be obtained which are then used in a known image synthesis algorithm to produce novel images of a scene of improved quality.

Method and apparatus for image matching

In an intelligent video surveillance system, video processing software performs a number of operations on video data received from a camera, including foreground extraction, shadow removal and object tracking. The foreground extraction stage classifies each pixel of a received frame as representing either foreground or background. Since shadow regions can be wrongly classified as foreground, a two-branch shadow removal operation is employed, comprising weak shadow removal () and strong shadow removal (). The purpose of weak shadow removal () is to remove only the most severe shadows in each frame. Conversely, the purpose of strong shadow removal () is to remove substantially every shadow present in each frame. By comparing the overlap of foreground regions in the two resulting images, it is possible to identify foreground regions that have fragmented due to strong shadow removal, and thereafter classify them as representing a common foreground object. Foreground objects are better defined in terms of their shape and subsequent processing stages will not treat fragmented regions as representing separate objects.

Method and system for processing video data including foreground extraction

A shot-based video content analysis method and system is described for providing automatic recognition of logical story units (LSUs). The method employs vector quantization (VQ) to represent the visual content of a shot, following which a shot clustering algorithm is employed together with automatic determination of merging and splitting events. The method provides an automated way of performing the time-consuming and laborious process of organising and indexing increasingly large video databases such that they can be easily browsed and searched using natural query structures.

Method and system for semantically segmenting scenes of a video sequence

A segmentation operation is applied to an input image to identify foreground objects of interest, and then a shadow removal operation is applied to remove any detected shadows from the foreground segmentation. The shadow removal algorithms can leave holes and bisections in the segmentation map, however, which will then subsequently impact on an object detection step performed using connected component analysis. To get around this problem, a conditional morphological dilation operation is applied to the segmentation map to 'grow' the segmented blobs to fill in any holes and bisections, without re-growing shadow pixels in the segmentation. The result is an object detection method and system which is robust to illumination changes causing shadows and/or highlights.

Object detection in images

A sequence of intensity measures is determined corresponding to each of a sequence of video frames. Such can be used for selection of videos according to user preferences and for providing highlights from a video sequence. Low level video characteristics are used which may be related to arousal and valence affects conveyed to a viewer while watching the video recording.

Image processing for analyzing video content

The luminance and chrominance values of pixels encoded according to MPEG or JPEG encoding are extracted using a forward discrete cosine transform (DCT). The forward DCT used in MPEG encoding is analyzed to derive a set of equations which directly relate a pixel value in an encoded image to one or more of the DCT coefficients obtained via the forward DCT transform in the usual image encoding process. These predetermined equations are used to allow for extremely fast and computationally efficient extraction of pixel values directly from the DCT coefficients of an encoded pixel block, without having to undergo an inverse DCT transform. Original images from MPEG and JPEG encoded versions may be extracted in a fast and efficient manner.

Method and apparatus for extracting pixel values

A method and system for processing an image recorded by an imaging device may include: a) providing an image having an array of adjacent elements each corresponding to a respective part of the image and having a respective intensity value associated therewith; b) processing the intensity values to determine an intensity contrast value for each respective element according to differences in intensity values between respective adjacent elements; c) determining a depth value for each element corresponding to the distance between the imaging device and at least part of an image forming object represented in the image by the respective element; d) processing the depth values to determine a depth contrast value for each respective element according to differences in depth values between respective adjacent elements; and, processing the intensity contrast values and depth contrast values to identify at least one area of the image corresponding to one or more respective objects in the image being processed. The last noted process step can be a non-linear diffusion process whereby variations in grey level values are diffused in regions corresponding to objects or background and enhanced in regions corresponding to object boundaries.

Growing community of inventors

Ipswich, United Kingdom

Li-Qun Xu