Method and apparatus for inducing classifiers for multimedia based on unified representation of features reflecting disparate modalities

Abstract

This invention is a system and method to perform categorization (classification) of multimedia items. These items are comprised of a multitude of disparate information sources, in particular, visual information and textual information. Classifiers are induced based on combining textual and visual feature vectors. Textual features are the traditional ones, such as, word count vectors. Visual features include, but are not limited to, color properties of key intervals and motion properties of key intervals. The visual feature vectors are determined in such a fashion that the vectors are sparse. The vector components are features such as the absence or presence of the color green in spatial regions and the absence or the amount of visual flow in spatial regions of the media items. The text and the visual representation vectors are combined in a systematic and coherent fashion. This vector representation of a media item lends itself to well-established learning techniques. The resulting system, subject of this invention, categorizes (or classifies) media items based both on textual features and visual features.