Sri International

Ajay Divakaran

Harpreet Singh Sawhney

Hui Ting Cheng

Amir Tamrakar

Karan Sikka

Qian Yu

Omar Javed

Jayakrishnan Kumar Eledath

Yi Yao

Jingen Liu

Saad Masood Khan

Elizabeth E Shriberg

Sang-Hack Jung

Behjat Siddiquie

Mayank Bansal

Michael A Cogswell

Dennis Lee Matthies

William Scott Mark

Jeffrey Lubin

Kenneth C Nitz

Girish Acharya

Bogdan Calin Mihai Matei

Edgar Kalns

Michael Frandsen

Min Yin

Saad Ali

Jihua Huang

Cory Albright

Michael Wessel

Jiejie Zhu

Xiao Lin

Arijit Ray

Giedrius T Burachas

Anirban Roy

Hyong-Gyun Kim

Andreas Tsiartas

Yunye Gong

Ankan Bansal

Chris Broaddus

Weiyu Zhang

Brent Mombourquette

Eric Law

Greg Ho

Supun Samarasekera

James R Carpenter

Zhiwei Zhu

Mohamed R Amer

Timothy J Shields

Steven S Weiner

Raia Hadsell

Subhodev Das

Ishani Chakraborty

Colleen Richey

Susmit Jha

Jesse Hostetler

Karuna Ahuja

Manika Puri

David A Salter

James F Carpenter

Ali Chaudhry

Thomas Germano

Pallabi Ghosh

Kamran Alipour

Sujeong Kim

David Chisolm

Pritish Sahu

Samyak Datta

Meng Ye

Nikoletta Basiou

David Chisholm

Sara Rutherford-quach

Nonye M Alozie

Elizabeth McBride

Stephanie Nunn

Anirudh Som

Bladimir Lopez-Prado

David Salter

Julia Kruk

Bo Dong

Yangyi Chen

Svati Dhamija

Hapreet S Sawhney

Tom Pschar

Christopher P Broaddus

Parneet Kaur

Nicholas Vandervalk

Shunguang Wu

In general, the disclosure describes techniques for joint spatiotemporal Artificial Intelligence (AI) models that can encompass multiple space and time resolutions through self-supervised learning. In an example, a method includes for each of a plurality of multimodal data, generating, by a computing system, using a first machine learning model, a respective modality feature vector representative of content of the multimodal data, wherein each of the generated modality feature vectors has a different modality; processing, by the computing system, each of generated modality feature vectors with a second machine learning model comprising an encoder model to generate event data comprising a plurality of events and/or activities of interest; and analyzing, by the computing system, the event data to generate anomaly data indicative of detected anomalies in the multimodal data.

Spatial-temporal anomaly and event detection using night vision sensors

A method, apparatus and system for adapting a pre-trained network for application to a different dataset includes arranging at least two different types of active adaptation modules in a pipeline configuration, wherein an output of a previous active adaptation module produces an input for a next active adaptation module in the pipeline in the form of adapted network data until a last active adaptation module, and wherein each of the at least two different types of adaptation modules can be switched on or off, determining at least one respective hyperparameter for each of the at least two different types of active adaptation modules, and applying the at least one respective determined hyperparameter to each of the at least two different types of active adaptation modules for processing received data from the pretrained network to determine an adapted network.

Modular adaptation for cross-domain few-shot learning

In an example, a method for fine-tuning a Large Visual Language Model (LVLM) includes providing visual queries, each of the visual queries comprises at least an image and a textual query related to the image; processing, by the LVLM, the visual queries to extract visual embeddings from the visual queries, wherein the LVLM comprises a Visual Language Model (VLM), a first Large Language Model (LLM), and a linear projection layer interconnecting the VLM and the LLM; for visual queries: i) generating, by the LVLM, a response to the corresponding visual query based on the corresponding visual embedding; ii) evaluating, by a second LLM, the generated response to verify that the generated response satisfies predefined criteria; and iii) providing, by the second LLM, a feedback to the LVLM, in response to the evaluating the generated response; and fine-tuning the LVLM using aggregated feedback provided by the second LLM for the visual queries.

Instruction-guided visual embeddings and feedback-based learning in large vision-language models

A computing system comprising a memory configured to store an artificial intelligence (AI) model and an image, and a computation engine executing one or more processors may be configured to perform the techniques for error-based explanations for AI behavior. The computation engine may execute the AI model to analyze the image to output a result. The AI model may, when analyzing the image to output the result, process, based on data indicative of the result, the image to assign an error score to each image feature extracted from the image, and obtain, based on the error scores, an error map. The AI model may next update, based on the error map and to obtain a first updated image, the image to visually indicate the error score assigned to each of the image features, and output one or more of the error scores, the error map, and the first updated image.

Error-based explanations for artificial intelligence behavior

A method, apparatus and system for determining user-content associations for determining and providing user-preferred content using multimodal embeddings include creating an embedding space for multimodal content by creating a first modality vector representation of the multimodal content having a first modality, creating a second modality vector representation of the multimodal content having a second modality, creating a user vector representation, as a third modality, for each user associated with at least a portion of the multimodal content, and embedding the first and the second modality vector representations and the user vector representations in the common embedding space using at least a mixture of loss functions for each modality pair of the first, the at least second and the third modalities that pushes closer co-occurring pairs of multimodal content. Embodiments can further include generating content using determined attributes of a message to be conveyed and features of the user-preferred content.

User targeted content generation using multimodal embeddings

In some examples, a computer-implemented collaboration assessment model identifies actions of each of two or more individuals depicted in video data, identify, based at least on the identified actions of each of the two or more individuals depicted in the video data, first behaviors at a first collaboration assessment level, identify, based at least on the identified actions of each of the two or more individuals depicted in the video data, second behaviors at a second collaboration assessment level different from the first collaboration assessment level, and generate and output, based at least on the first behaviors at the first collaboration assessment level and the second behaviors at the second collaboration assessment level, an indication of at least one of an assessment of a collaboration effort of the two or more individuals or respective assessments of individual contributions of the two or more individuals to the collaboration effort.

Automated collaboration skills assessment

Methods, computing devices, and computer-program products are provided for implementing a virtual personal assistant. In various implementations, a virtual personal assistant can be configured to receive sensory input, including at least two different types of information. The virtual personal assistant can further be configured to determine semantic information from the sensory input, and to identify a context-specific framework. The virtual personal assistant can further be configured to determine a current intent. Determining the current intent can include using the semantic information and the context-specific framework. The virtual personal assistant can further be configured to determine a current input state. Determining the current input state can include using the semantic information and one or more behavioral models. The behavioral models can include one or more interpretations of previously-provided semantic information. The virtual personal assistant can further be configured to determine an action using the current intent and the current input state.

VPA with integrated object recognition and facial expression recognition

In general, the disclosure describes techniques for characterizing a dynamical system and a neural ordinary differential equation (NODE)-based controller for the dynamical system. An example analysis system is configured to: obtain a set of parameters of a NODE model used to implement the NODE-based controller, the NODE model trained to control the dynamical system; determine, based on the set of parameters, a system property of a combined system comprising the dynamical system and the NODE-based controller, the system property comprising one or more of an accuracy, safety, reliability, reachability, or controllability of the combined system; and output the system property to modify one or more of the dynamical system or the NODE-based controller to meet a required specification for the combined system.

Analysis and design of dynamical system controllers using neural differential equations

A method, apparatus and system for training an embedding space for content comprehension and response includes, for each layer of a hierarchical taxonomy having at least two layers including respective words resulting in layers of varying complexity, determining a set of words associated with a layer of the hierarchical taxonomy, determining a question answer pair based on a question generated using at least one word of the set of words and at least one content domain, determining a vector representation for the generated question and for content related to the at least one content domain of the question answer pair, and embedding the question vector representation and the content vector representations into a common embedding space where vector representations that are related, are closer in the embedding space than unrelated embedded vector representations. Requests for content can then be fulfilled using the trained, common embedding space.

System and method for content comprehension and response

Techniques are disclosed for identifying multimodal subevents within an event having spatially-related and temporally-related features. In one example, a system receives a Spatio-Temporal Graph (STG) comprising (1) a plurality of nodes, each node having a feature descriptor that describes a feature present in the event, (2) a plurality of spatial edges, each spatial edge describing a spatial relationship between two of the plurality of nodes, and (3) a plurality of temporal edges, each temporal edge describing a temporal relationship between two of the plurality of nodes. Furthermore, the STG comprises at least one of: (1) variable-length descriptors for the feature descriptors or (2) temporal edges that span multiple time steps for the event. A machine learning system processes the STG to identify the multimodal subevents for the event. In some examples, the machine learning system comprises stacked Spatio-Temporal Graph Convolutional Networks (STGCNs), each comprising a plurality of STGCN layers.

Identifying complex events from hierarchical representation of data set features

A method, apparatus and system for zero shot object detection includes, in a semantic embedding space having embedded object class labels, training the space by embedding extracted features of bounding boxes and object class labels of labeled bounding boxes of known object classes into the space, determining regions in an image having unknown object classes on which to perform object detection as proposed bounding boxes, extracting features of the proposed bounding boxes, projecting the extracted features of the proposed bounding boxes into the space, computing a similarity measure between the projected features of the proposed bounding boxes and the embedded, extracted features of the bounding boxes of the known object classes in the space, and predicting an object class label for proposed bounding boxes by determining a nearest embedded object class label to the projected features of the proposed bounding boxes in the space based on the similarity measures.

Zero-shot object detection

A method, apparatus and system for visual grounding of a caption in an image include projecting at least two parsed phrases of the caption into a trained semantic embedding space, projecting extracted region proposals of the image into the trained semantic embedding space, aligning the extracted region proposals and the at least two parsed phrases, aggregating the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation and projecting the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption. The method, apparatus and system can further include a parser for parsing the caption into the at least two parsed phrases and a region proposal module for extracting the region proposals from the image.

Align-to-ground, weakly supervised phrase grounding guided by image-caption alignment

A method, apparatus and system for understanding visual content includes determining at least one region proposal for an image, attending at least one symbol of the proposed image region, attending a portion of the proposed image region using information regarding the attended symbol, extracting appearance features of the attended portion of the proposed image region, fusing the appearance features of the attended image region and features of the attended symbol, projecting the fused features into a semantic embedding space having been trained using fused attended appearance features and attended symbol features of images having known descriptive messages, computing a similarity measure between the projected, fused features and fused attended appearance features and attended symbol features embedded in the semantic embedding space having at least one associated descriptive message and predicting a descriptive message for an image associated with the projected, fused features.

Aligning symbols and objects using co-attention for understanding visual content

Growing community of inventors

Monmouth Junction, NJ, United States of America

Ajay Divakaran