Adobe, Inc.

University System of Maryland

Uttaran Bhattacharya

Dinesh Manocha

Trisha Mittal

Rohan Chandra

Aniket Bera

Viswanathan Swaminathan

Gang Wu

Stefano Petrangeli

Eunyee Koh

Saayan Mitra

Xiang Chen

Tong Yu

Victor Soares Bursztyn

Manas Garg

Pooja Guhan

Alexandru Ionut Hodorogea

Saurabh Tripathy

A method, apparatus, non-transitory computer readable medium, and system for document retrieval include obtaining a query and a document. A prompt generator generates a prompt for a reasoning model based on the query and the document. The reasoning model generates a reasoning result based on the prompt. In some cases, the reasoning result indicates that the document answers the query. A machine learning model provides the document in response to the query based on the reasoning result.

In-context and semantic-aware ensemble model for document retrieval

A video is classified as real or fake by extracting facial features, including facial modalities and facial emotions, and speech features, including speech modalities and speech emotions, from the video. The facial and speech modalities are passed through first and second neural networks, respectively, to generate facial and speech modality embeddings. The facial and speech emotions are passed through third and fourth neural networks, respectively, to generate facial and speech emotion embeddings. A first distance, d, between the facial modality embedding and the speech modality embedding is generated, together with a second distance, d, between the facial emotion embedding and the speech emotion embedding. The video is classified as fake if a sum of the first distance and the second distance exceeds a threshold distance. The networks may be trained using real and fake video pairs for multiple subjects.

System and method for detecting fabricated videos

Embodiments are disclosed for predicting, using neural networks, editing operations for application to a video sequence based on processing conversational messages by a video editing system. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input including a video sequence and text sentences, the text sentences describing a modification to the video sequence, mapping, by a first neural network content of the text sentences describing the modification to the video sequence to a candidate editing operation, processing, by a second neural network, the video sequence to predict parameter values for the candidate editing operation, and generating a modified video sequence by applying the candidate editing operation with the predicted parameter values to the video sequence.

Predicting video edits from text-based conversations using neural networks

Systems, methods, apparatuses, and computer program products for recognizing human emotion in images or video. A method for recognizing perceived human emotion may include receiving a raw input. The raw input may be processed to generate input data corresponding to at least one context. Features from the raw input data may be extracted to obtain a plurality of feature vectors and inputs. The plurality of feature vectors and the inputs may be transmitted to a respective neural network. At least some of the plurality of feature vectors may be fused to obtain a feature encoding. Additional feature encodings may be computed from the plurality of feature vectors via the respective neural network. A multi-label emotion classification of a primary agent may be performed in the raw input based on the feature encoding and the additional feature encodings.

Human emotion recognition in images or video

Systems, methods, apparatuses, and computer program products for providing multimodal emotion recognition. The method may include receiving raw input from an input source. The method may also include extracting one or more feature vectors from the raw input. The method may further include determining an effectiveness of the one or more feature vectors. Further, the method may include performing, based on the determination, multiplicative fusion processing on the one or more feature vectors. The method may also include predicting, based on results of the multiplicative fusion processing, one or more emotions of the input source.

System and method for multimodal emotion recognition

In implementations for highlight video generated with adaptable multimodal customization, a multimodal detection system tracks activities based on poses and faces of persons depicted in video clips of video content. The system determines a pose highlight score and a face highlight score for each of the video clips that depict at least one person, the highlight scores representing a relative level of the interest in an activity depicted in a video clip. The system also determines pose-based emotion features for each of the video clips. The system can detect actions based on the activities of the persons depicted in the video clips, and detect emotions exhibited by the persons depicted in the video clips. The system can receive input selections of actions and emotions, and filter the video clips based on the selected actions and emotions. The system can then generate a highlight video of ranked and filtered video clips.

Highlight video generated with adaptable multimodal customization

Growing community of inventors

College Park, MD, United States of America

Uttaran Bhattacharya