Sarnoff Corporation

David Sarnoff Research Center, Inc.

Michael Wade Hansen

Peter Jeffrey Burt

Stephen Charles Hsu

Michal Irani

Padmanabhan Anandan

Gooitzen Siemen Van Der Wal

Lambert Ernest Wixson

Michael Raymond Piacentino

Robert G Bassman

Jayakrishnan Kumar Eledath

Clay Douglas Spence

Bogdan Calin Mihai Matei

Bhavesh B Bhatt

Gooitzen Sieman Van Der Wal

Robert Mandelbaum

Barbara Hanna

Deepam Mishra

Frederic William Brehm

Bill J Call

Arthur Robert Pope

Lucas Kelsey McDowell

A method and system for improving the accuracy and timeliness of video metadata by incorporating information related to the motion of the camera as derived from the video imagery itself. Frame-to-frame correspondences are used to accurately estimate changes in camera pose. While the method and system do not require geo-registration, geo-registration results, if available, may be considered in processing the video images and generating improved camera pose estimates.

Method and system for improving video metadata through the use of frame-to-frame correspondences

Apparatus and a concomitant method of identifying the direction of motion within a scene that is represented by a sequence of images, e.g., a series of frames in a video sequence. The apparatus contains a flow field generator, a flow field segmentor, and alarm detector and an alarm processor. The flow field generator processes the sequence of images and generates a substantially distortionless flow field representing the motion within a scene. The flow field is a vector representation of the motion of the scene that represents both the magnitude and the direction of the motion. The flow field is generated by correlating at least two frames in the sequence of images. This flow field is analyzed by a flow field segmentor to determine the magnitude and direction of motion with the scene and segment the motion information from the static portions of the scene. An alarm detector then processes the motion information to determine if an alarm should be generated based upon the motion information. The alarm processor activates an appropriate alarm response, e.g., illuminate warning lamps, sound a warning alarm, lock doors, alert security personnel, and the like.

Apparatus and a method for detecting motion within an image sequence

A method for fixating a camera, mounted on a motorized mount, on a target point. The method includes receiving a plurality of images representative of a scene; selecting, within the plurality of images, a first display reference image containing the target point at a first coordinate location; estimating, in a current image within the plurality of images, a transformation that maps the current image to the first display reference image; estimating a second coordinate location of the target point in the current image using the transformation; computing an image slip between the target point in the current image and the target point in the first display reference image; converting the image slip into an angular correction for fixating the camera; modifying the angular correction by using closed-loop control with the motorized mount; and warping the current image using the transformation to align the current image with the first display reference image. This warping step provides a stable view of the scene even in the presence of camera movements that cannot be eliminated using mechanical stabilization, and in the presence of motor control lags and oscillations due to measurement and actuator latencies.

Method and apparatus for fixating a camera on a target point using image alignment

A system for automatically generating a mosaic from a plurality of input images. The system sequentially executes an image alignment process, an editing process, and a combining process such that, from a sequence of images, the system automatically produces a seamless mosaic for various applications. Illustrative applications for the mosaic include: (1) a mosaic based display system including an image printing system, (2) a surveillance system and (3) a mosaic based compression system. The mosaic based display system permits a system user to display, manipulate and alter a mosaic. The mosaic based compression system exploits the temporal and spatial redundancy in image sequences and efficiently compresses the image information. The compression system can be used for compressing image information for storage in a storage device or can be used for compressing image information for transmission through a band-limited transmission channel. The surveillance system combines the mosaic construction system, the compression system and the display system into a comprehensive system for remote motion detection.

Mosaic based image processing system

An image stabilizer selectively adds image data from a background image to the current image to compensate for data in the current image that is missing due to a sudden shift in the current image relative to the previous images. The current image is warped into the coordinate system of the background image and then the warped current image is merged with the background image to replace any blank areas in the current image with corresponding pixel values from the background image. The image data from the background image which is to be substituted into the warped current image is subject to a low-pass filtering operation before it is merged with the warped current image. The warped current image is merged with the background image to form a modified background image which is then merged with the warped current image. The background image is, itself, warped to track camera motion in obtaining the current image before the background image is merged with the warped current image.

Apparatus and method for removing blank areas from real-time stabilized images by inserting background information

A real-time modular video processing system (VPS) which can be scaled smoothly from relatively small systems with modest amounts of hardware to very large, very powerful systems with significantly more hardware. The modular video processing system includes a processing module containing at least one general purpose microprocessor which controls hardware and software operation of the video processing system using control data and which also facilitates communications with external devices. One or more video processing modules are also provided, each containing parallel pipelined video hardware which is programmable by the control data to provide different video processing operations on an input stream of video data. Each video processing module also contains one or more connections for accepting one or more daughterboards which each perform a particular image processing task. A global video bus routes video data between the processing module and each video processing module and between respective processing modules, while a global control bus provides the control data to/from the processing module from/to the video processing modules separate from the video data on the global video bus. A hardware control library loaded on the processing module provides an application programming interface including high level C-callable functions which allow programming of the video hardware as components are added and subtracted from the video processing system for different applications.

Modular parallel-pipelined vision system for real-time video processing

Digital signal processing circuitry implemented in ASICs or FPGAs is built by combining multi-component constructs (e.g. macrocells). These circuits may be modified to include a timing channel by augmenting selected ones of the constructs to include a path which propagates a timing signal with a delay that compensates for the signal processing delay through the construct. The selected constructs are those that are used in a critical processing path in the digital signal processing circuitry. A timing compensation circuit may also be defined as a construct. This block receives two digital data signals having accompanying timing signals and delays the first signal that provides valid data until the second signal also provides valid data, as determined by their timing signals. A configurable arithmetic and logic unit (ALU) made using these techniques includes a timing compensation circuit, a look-up table and an accumulator. The configurable ALU may also include a timing signal selection circuit which selects between the each of two input timing signals, the logical AND of the input timing signals and the logical OR of the two input timing signals to produce an output timing signal. A programmable multiply-accumulator includes a matrix of multipliers, each of which may receive one of a plurality of input signals. The input signals are delayed through a pipeline and a portion of this pipeline is reserved for delaying one or more timing signals associated with the plurality of input signals. The delayed timing signals form the timing signals that are associated with the output signal of the programmable multiply accumulator.

Digital signal processing circuitry having integrated timing information

Apparatus and a concomitant method of identifying the direction of motion within a scene that is represented by a sequence of images, e.g., a series of frames in a video sequence. The apparatus contains a flow field generator, a flow field segmentor, and alarm detector and an alarm processor. The flow field generator processes the sequence of images and generates a substantially distortionless flow field representing the motion within a scene. The flow field is a vector representation of the motion of the scene that represents both the magnitude and the direction of the motion. The flow field is generated by correlating at least two frames in the sequence of images. This flow field is analyzed by a flow field segmentor to determine the magnitude and direction of motion with the scene and segment the motion information from the static portions of the scene. An alarm detector then processes the motion information to determine if an alarm should be generated based upon the motion information. The alarm processor activates an appropriate alarm response, e.g., illuminate warning lamps, sound a warning alarm, lock doors, and alert security personnel.

Apparatus for image processing a sequence of images containing a parallel-pipelined image processor comprised of image memories, a pyramid processing circuit, an arithmetic logic unit, a crossbar switch for video routing through the various components of the processor, signal processors to provide hardware programming through a global bus and also perform image processing operations. Images can be passed directly from the crossbar switch to internal static RAM of the signal processors through a first-in, first-out interface at full video rates.

Parallel-pipelined image processing system

System for automatically aligning images to form a mosaic image

Method and apparatus for performing mosaic based image compression

A pipelined parallel processor (PPP) integrated onto a single integrated circuit. The PPP includes a filtering unit, internal routing circuitry such as a crosspoint switch, an internal frame store controller, and one or more function circuits. The function circuits may be, for example, arithmetic units, lookup tables, timing compensators, adders/subtractors, statistics modules, image shifting circuitry, and other useful processing devices. The components of the PPP are interconnected with the crosspoint switch which routes data between the frame store controller, the filtering unit, function circuits, external input channels, and external output channels.

Growing community of inventors

New Hope, PA, United States of America

Michael Wade Hansen