Massachusetts Institute of Technology

Gentech Corporation

Thinking Machines Corporation

Istituto Trentino Di Cultura

Instituto Trentino Di Cultura

Technopolis Csata Novus Ortus

Tomaso A Poggio

Pawan Sinha

Amnon Shashua

David J Beymer

John Gregory Harris

Pamela R Lipson

Nicola Ancona

Roberto Brunelli

Antoine F Ezzat

Michael Jones

Thomas Vetter

Howard L Resnikoff

Karl P Sims

Bin Zhang

W Eric Grimson

Maximilian Riesenhuber

W Eric L Grimson

Andrew Lo

Thomas R Serre

Lior Wolf

Stephen E Librande

Luigi Stringa

Stanley M Bileschi

Daniele Falavigna

Christian R Shelton

Chiejin Cheng

Michael Oren

Kah-Kay Sung

Tung Chan

Constatine P Papageorgiou

An artificial visual recognition system and method employ a digital processor and a model executed by the digital processor. The model has a loose hierarchy of layers. Each layer, from a lowest hierarchy level to a top level, provides relatively increasing selectivity and invariance of the input image. The hierarchy allows bypass routes between layers. On output, the model produces feature recognition and classification of an object in the input image. In some embodiments, windowing means provide windows of the input image to the model, and the model responds to shape-based objects in the input image. In another feature, segmenting means segment the input image and enables the model to determine texture-based objects in the input image.

High-performance vision system exploiting key features of visual cortex

Disclosed herein is an automated Electronic Market-Maker, comprising a Market Quality Control Module for generating trading recommendations to maintain market quality; an Inventory Control Module for generating trading recommendations to maintain inventory; a Speculation Module for generating trading recommendations to generate profits; and an Arbitration Module to receive trading recommendations from said Market Quality Control Module, Inventory Control Module, and Speculation Control Module and resolve any conflicts between the received recommendations.

Electronic market-maker

A method and apparatus for videorealistic, speech animation is disclosed. A human subject is recorded using a video camera as he/she utters a predetermined speech corpus. After processing the corpus automatically, a visual speech module is learned from the data that is capable of synthesizing the human subject's mouth uttering entirely novel utterances that were not recorded in the original video. The synthesized utterance is re-composited onto a background sequence which contains natural head and eye movement. The final output is videorealistic in the sense that it looks like a video camera recording of the subject. The two key components of this invention are 1) a multidimensional morphable model (MMM) to synthesize new, previously unseen mouth configurations from a small set of mouth image prototypes; and 2) a trajectory synthesis technique based on regularization, which is automatically trained from the recorded video corpus, and which is capable of synthesizing trajectories in MMM space corresponding to any desired utterance.

Trainable videorealistic speech animation

An image processing system for generating a class model from an image by identifying relative relationships between different properties of different image regions, includes a region partitioner a relationship processor, a template generator and an image detector. The image processing system utilizes a class model defined by one or more relative relationships between a plurality of image patches. The relative relationships describe the overall organization of images within an image class. The relative relationships are encoded in a global deformable template which can be used to classify or detect images. The class model may be pre-defined or generated by the image processing system. In one embodiment, the class model is generated from a low resolution image.

Method and apparatus for classifying and identifying images

Techniques for finding a correspondence between a source object and a target object, the objects having n>2 dimensions, such that a vector field based on the correspondence generates objects which tend to preserve the characteristics of both the source object and the target object. The techniques result in vector fields which maximize the generated object's similarity to the target object while minimizing structural changes to the source object. Minimization of structural changes is done by associating directional spring functions with points in the source object and finding a vector which generates an object for which changes in the directional spring functions are minimized. Further included are techniques for reducing the computational cost of finding a correspondence by making a set of pairs of increasingly-simplified representations of the source object and the target object and then finding pair vector fields for the pairs, beginning with the most simplified pair and ending with the original representations of the source object and the target object. The increasingly-simplified representations have the property that they preserve the large-scale structure of the original representations of the source object and the target object. The pair vector field for a given pair is found by extending the pair vector field found for the next simpler pair so that it applies to the pair's source object and then using the extended pair vector field as the initial vector field in finding the pair vector field. The techniques may be applied to images and to surfaces in which color is represented by means of additional dimensions. Vector fields found using the techniques may be used to generate objects belonging to a class of objects defined by the source object and at least the target object.

Correspondence between n-dimensional surface: vector fields that are defined by surfaces and that generate surfaces which preserve characteristics of the defining surfaces

A trainable object detection system and technique for detecting objects such as people in static or video images of cluttered scenes is described. The described system and technique can be used to detect highly non-rigid objects with a high degree of variability in size, shape, color, and texture. The system learns from examples and does not rely on any a priori (hand-crafted) models or on motion. The technique utilizes a wavelet template that defines the shape of an object in terms of a subset of the wavelet coefficients of the image. It is invariant to changes in color and texture and can be used to robustly define a rich and complex class of objects such as people. The invariant properties and computational efficiency of the wavelet template make it an effective tool for object detection.

Trainable system to search for objects in images

A method and apparatus of converting input text into an audio-visual speech stream resulting in a talking face image enunciating the text. This method of converting input text into an audio-visual speech stream comprises the steps of: recording a visual corpus of a human-subject, building a viseme interpolation database, and synchronizing the talking face image with the text stream. In a preferred embodiment, viseme transitions are automatically calculated using optical flow methods, and morphing techniques are employed to result in smooth viseme transitions. The viseme transitions are concatenated together and synchronized with the phonemes according to the timing information. The audio-visual speech stream is then displayed in real time, thereby displaying a photo-realistic talking face.

Talking facial display method and apparatus

An image processing system and methods of operation within such a system are disclosed. The image processing system utilizes a class model defined by one or more relative relationships between a plurality of image patches. The relative relationships describe the overall organization of images within an image class. The relative relationships are encoded in a global deformable template which can be used to classify or detect images. The class model may be predefined or generated by an image processing system.

An image processing system (100) analyzes novel images in one of three modes. In a first mode, the system learns a model based on vectorized prototype images, and uses the model to characterize the shape of a novel image as a pixel-by-pixel linear combination of the prototype shapes. In a second mode, both shape and texture vectors are used to characterize the novel image as a linear combination of the prototypes. In a third mode, shape information is modeled by a linear combination of prototypes and texture information is modeled by an optical flow, and processing of shape and texture information is interleaved. Novel images are synthesized by analysis of input images using transformed prototypes. Image compression is achieved by storing or transmitting parameter vectors relating the novel or applied images to the prototypes.

Image compression by pointwise prototype correspondence using shape and

An image processing system (100) analyzes novel images in one of three modes. In a first mode, the system learns a model based on prototype images, and uses the model to characterize the shape of a novel image as a pixel-by-pixel linear combination of the prototypes. In a second mode, both shape and texture are used to characterize the novel image as a linear combination of the prototypes. In a third mode, shape information is modeled by a linear combination of prototypes and texture information is modeled by an optical flow, and processing of shape and texture information is interleaved. Novel images are synthesized by analysis of input images using transformed prototypes.

Image analysis and synthesis networks using shape and texture information

Synthesis of novel images from example images is achieved by determining a pixelwise optical flow among example images, computing a parameter set for a new image, and synthesizing the new image based on the parameter vector and the example images. The parameter set may describe characteristics of the image, in which case the characteristics are applied to a neural network trained with the example images in order to synthesize the novel image. The parameter set may also be an estimate of relative contributions of each of the example images to the new image, in which case the new image may be synthesized by taking a linear combination of the example images, weighted by the parameter set. In one embodiment, both the example and the new images are of the same object. In another embodiment, the example image represents a first object while the new image represents a second object. In yet another embodiment, a set of target images is synthesized from a single target image and a set of source images. Analysis of existing images to determine image parameters is achieved by determining parameters for the existing image based on comparison of the existing image with an image set for which parameters are known.

Example-based image analysis and synthesis using pixelwise correspondence

An image processing system comprising and image recording system and an image rendering system. The image recording system records images of a scene, and comprises a recording device for recording images of the scene on a series of frames, each frame including image information reflecting the scene as illuminated at the time the frame was recorded, a plurality of individually-energizable light sources each for illuminating the scene; and a synchronizer connected to the recording device and the light sources for synchronizing the separate energization of the light sources and the recording by the recording device of the separate frames in the series. The image rendering system generates a rendered image which reflects a desired light source position. The image rendering system specifically comprises a frame store for storing the image information for each of the series of frames, a rendered image store for receiving rendered image information a rendered image a rendered image information generator for generating rendered image information for storage in response to image information from the frame store related to frames of the series as recorded in synchrony with diverse ones of the light sources, the rendered image information generator generating the rendered image information in relation to the image information of each of the frames, as weighted in relation to the relationship between the position of the light sources in synchrony with which the frame was recorded and the desired position of the light source for the rendered image.

System and method for rendering images

An object movement estimator (400) predicts optical flow by estimating changes in sensed images using one-dimensional correlations. Sensed image portions are stored at a first instant of time and compared with sensed image portions at a second instant of time. An optimal flow is estimated by performing a one-dimensional correlation of the sensed image portions at the first and second instants of time, and a time-to-contact is estimated based on the predicted optical flow thus obtained.

Object movement estimator using one-dimensional optical flow

Computer method and apparatus provides remote communication of images with low communication bandwidth requirements. Workstations are remotely coupled to each other across a communication channel (e.g., telephone line or Internet line) and employ the invention method and apparatus. Parameter values of desired images are input at one workstation and transmitted across the communication channel to a remote workstation. In response to the input parameter values, the remote workstation generates and displays the desired images. Video e-mail messages is one application of such remote generation of desired images. Further parameter values may be input at the remote workstation for generation and display of corresponding further images at the initial workstation. Interactive video conferencing results as the workstations remotely communicate images in real-time to each other.

Computer method and apparatus for video conferencing

A face detection system (100) includes an imaging device, a computer having a pattern prototype synthesizer and an image classifier, and an output display device. Face and non-face pattern prototypes are synthesized by a network training process using a number of example images. A new applied image is analyzed by taking a distance from the applied image to each of the prototypes, and a decision is made as to whether the applied image contains a face based on the distances thus obtained. The system and method may be applied to various other detection tasks.

Growing community of inventors

Wellesley, MA, United States of America

Tomaso A Poggio