International Business Machines Corporation

Massachusetts Institute of Technology

University of Illinois

Mitsubishi Electric Research Laboratories, Inc.

Indian Institute of Technology

Rogerio Schmidt Feris

Yun Zhai

Sharathchandra UmapathiRao Pankanti

Arun Hampapur

Ankur Datta

Lisa Maire Brown

Russell Patrick Bobbitt

Ying-Li Tian

Clifford A Pickover

Daniel Andre Vaquero

Lisa Marie Brown

Behjat Siddiquie

Minkyong Kim

Zouxuan Lu

Maja Vukovic

Longbin Chen

Ira L Allen

Zuoxuan Lu

Hui Wu

Xiaoyu Wang

Frederik C Kjeldsen

Xiaoxiao Guo

Rameswar Panda

James R Kozloski

Thomas David Erickson

Chiao-Fe Shu

Charles A Otto

Yu Cheng

Dhiraj Joshi

Hao-Wei Liu

Duan D Tran

Itzhack Goldberg

John Richard Smith

Neil Sondhi

Quanfu Fan

Stephen Carl Hammer

Chung-Ching Lin

Michele Merler

Quoc-Bao Nguyen

John Joseph Kent

Kristina Scherbaum

Aude Jeanne Oliva

Valentina Salapura

Ramesh Raskar

Jinjun Xiong

Ying Li

Richard Chen

Christopher Scott Milite

Minh Do

Jing Wang

Leonid Karlinsky

Mandis S Beigi

Curtis Howard Brobst

Stephen R Russo

Frederik C M Kjeldsen

Eliyahu Schwartz

Raymond A Cooke

Raymond Alexander Yeh

Matous Havlena

Yue Meng

Honghui Shi

James Petterson

Daniel André Vaquero

Khoi-Nguyen C Mac

Bowen Pan

Abir Das

One embodiment of the invention provides a method for video recognition. The method comprises receiving an input video comprising a sequence of video segments over a plurality of data modalities. The method further comprises, for a video segment of the sequence, selecting one or more data modalities based on data representing the video segment. Each data modality selected is optimal for video recognition of the video segment. The method further comprises, for each data modality selected, providing at least one data input representing the video segment over the data modality selected to a machine learning model corresponding to the data modality selected, and generating a first type of prediction representative of the video segment via the machine learning model. The method further comprises determining a second type of prediction representative of the entire input video by aggregating all first type of predictions generated.

Adaptive selection of data modalities for efficient video recognition

A system and a method for tracking a plurality of objects, including obtaining input data, estimating a number of skipping frames of the input data based on information from the input data, predicting results based on the estimating of the number of skipping frames, and correcting the predicted results.

System and method for multiple object tracking

A sequence of patch tokens representing an image can be received. A network can be trained to learn informative patch tokens and uninformative patch tokens in the sequence of patch tokens, in learning to recognize an object in the image. The sequence of patch tokens can be reduced by removing the uninformative patch tokens from the sequence of patch tokens. The reduced sequence of patch tokens can be input to an attention-based deep learning neural network. The attention-based deep learning neural network can be fine-tuned to recognize the object in the image using the reduced sequence of patch tokens.

Interpretability-aware redundancy reduction for vision transformers

A base pathway of a computerized two-pathway video action recognition model is trained using a plurality of labeled video samples. The base pathway is trained using a plurality of unlabeled video samples at a first framerate. An auxiliary pathway of the computerized two-pathway video action recognition model is trained using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate, wherein the training of the base pathway and the training of the auxiliary pathway result in a trained computerized two-pathway video action recognition model. A candidate video is categorized using the trained computerized two-pathway video action recognition model and the categorized candidate video is stored in a computer-accessible video database system for information retrieval.

Temporal contrastive learning for semi-supervised video action recognition

Methods, apparatus, and systems for multi-resolution processing for video classification. A plurality of video frames of a video are obtained and a resolution for classifying each video frame of the plurality of video frames is determined by analyzing each video frame using a policy network. Based on the determined resolution, each video frame having a determined resolution is rescaled and each rescaled video frame is routed to a classifier of a backbone network that corresponds to the determined resolution. Each rescaled video frame is classified using the corresponding classifier of the backbone network to obtain a plurality of classifications and the classifications are averaged to determine an action classification of the video.

Dynamic multi-resolution processing for video classification

A method includes receiving input from a client at least partially specifying one or more characteristics, wherein the initial input includes a seed image and a natural language statement describing a desired change to the seed image; predicting one or more attributes of the seed image by operation of a neural network on the seed image; and parsing the natural language statement to identify desired changes to the one or more attributes of the seed image. The method also includes generating an interim target image by changing the one or more attributes of the seed image, according to the parsed natural language statement; selecting a set of images from an image database for output to the client, each of said set of images being determined to at least partially satisfy the one or more changed attributes of the seed image; and displaying the set of images to the client.

Dialog-based image retrieval with contextual information

A method and system for auto-curating a media are provided. Media content is received over the network interface. A set of markers is identified for the media content, each marker corresponding to one of a plurality of visible and audible cues in the media content. Segments in the media content are identified based on the identified set of markers. An excitement score is computed for each segment based on the identified markers that fall within the segment. A highlight clip is generated by identifying segments having excitement scores greater than a threshold.

Auto-curation and personalization of sports highlights

Techniques regarding action detection based on motion in receptive fields of a neural network model are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory, and that can execute the computer executable components stored in the memory. The computer executable components can comprise a motion component that can extract a motion vector from a plurality of adaptive receptive fields in a deformable convolution layer of a neural network model. The computer executable components can also comprise an action detection component that can generate a spatio-temporal feature by concatenating the motion vector with a spatial feature extracted from the deformable convolution layer.

Action detection by exploiting motion in receptive fields

A search engine is modified to perform increasingly precise image searching using iterative Natural Language (NL) interactions. From an NL search input, the modification extracts a set of input features, which includes a set of response features corresponding to an NL statement in the NL search input and a set of image features from a seed image in the NL search input. The modification performs image analysis on an image result in a result set of a query including at least some of the input features. In a next iteration of NL interactions, at least some of the result set is provided. An NL response in the iteration is added to a cumulative NL basis, and a revised result set is provided, which includes a new image result corresponding to a new response feature extracted from the cumulative NL basis.

Image retrieval using interactive natural language dialog

Embodiments may provide learning and recognition of classifications using only one or a few examples of items. For example, in an embodiment, a method of computer vision processing may be implemented in a computer comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method may comprise training a neural network system implemented in the computer system to classify images into a plurality of classes using one or a few training images for each class and a plurality of associated semantic information, wherein the plurality of associated semantic information is from a plurality of sources and comprises at least some of class/object labels, textual description, or attributes, and wherein the neural network is trained by modulating the training images by sequentially applying the plurality of associated semantic information and classifying query images using the trained neural network system.

System and method for augmenting few-shot object classification with semantic information from multiple sources

An information processing system, a computer readable storage medium, and a method for providing a recommendation for a plaything as a recommended item can include analyzing information received from a person monitoring system to provide an analysis for providing the recommendation for the plaything, and based on the analysis, sending a representation of the recommended item in a signal to a shopping cart such as an online shopping cart. The system can include an analysis module that receives information from a person monitoring system, and at least one processor configured to analyze information received from the person monitoring system (for one or more persons) to provide an analysis. The analysis provides a recommendation for a plaything. The processor can further send a representation of the recommended item in a signal to a shopping cart to upload into the shopping cart based on the analysis. Other embodiments are disclosed.

Method, a system, and a computer readable storage medium for automated recommendations for an online shopping cart

A method includes: receiving initial input from a client at least partially specifying one or more characteristics sought by the client; selecting a set of images from an image database for output to the client; and determining after each set of images whether an end condition has occurred. The method also includes, until the end condition has occurred: responsive to each set of images output to the client, receiving additional input from the client further specifying the one or more characteristics sought by the client; and responsive to each input received from the client, selecting another set of images for presentation to the client, said set of images being determined to at least partially satisfy the one or more characteristics specified by all input received from the client, said determination being based at least in part on side information for respective images for at least the set of images.

Information relating to at least one of a user and a user environment is acquired. A user cognitive state is determined based on the acquired information. A graphical control element is automatically configured based on the user cognitive state. The graphical control element is automatically presented on a display interface of a user device to control viewing of content displayed on the user device.

Cognitive graphical control element

An object recognition system includes a parameter generator for generating a parameter that indicates whether an object depicted in an image pair is the same or different object, and a pose difference for the image pair and a parameter refiner for refining the parameter, and an applicator for applying the refined parameter to object recognition.

System and method for pose-aware feature learning

Growing community of inventors

White Plains, NY, United States of America

Rogerio Schmidt Feris