International Business Machines Corporation

Rensselaer Polytechnic Institute

Bo Wu

Chuang Gan

Dakuo Wang

Zhenfang Chen

Yang Zhang

Pin-Yu Chen

Rameswar Panda

Qi Cheng Li

Sijia Liu

Mo Yu

Tengfei Ma

Antonio Jose Jimeno Yepes

Xiaoxiao Guo

Kaizhi Qian

Lirong Xia

Ao Liu

Chunheng Jiang

Jianxi Gao

An output layer is removed from a pre-trained neural network model and a neural capacitance probe unit with multiple layers is incorporated on top of one or more bottom layers of the pre-trained neural network model. The neural capacitance probe unit is randomly initialized and a modified neural network model is trained by fine-tuning the one or more bottom layers on a target dataset for a maximum number of epochs, the modified neural network model comprising the neural capacitance probe unit incorporated with multiple layers on top of the one or more bottom layers of the pre-trained neural network model. An adjacency matrix is obtained from the initialized neural capacitance probe unit and a neural capacitance metric is computed using the adjacency matrix. An active model is selected using the neural capacitance metric and a machine learning system is configured using the active model.

Neural capacitance: neural network selection via edge dynamics

A system may include a memory and a processor in communication with the memory. The processor may be configured to perform operations. The operations may include receiving an input, extracting features from the input, and mining object relations using the features. The operations may include determining feature vectors using the object relations and generating, using the feature vectors, an output indicating a target region, wherein the target region corresponds to the input.

Image grounding with modularized graph attentive networks

According to one embodiment, a method, computer system, and computer program product for identifying one or more intrinsic physical properties of one or more objects is provided. The present invention may include identifying one or more objects in a video set, extracting observable physical properties of the identified one or more objects from the video set, including one or more trajectories, and inferring, by a property-based graph neural network, intrinsic properties of the one or more objects based on the trajectories.

Object-centric and relation-centric graph neural networks for physical property discovery

Mechanisms are provided for performing artificial intelligence-based video question answering. A video parser parses an input video data sequence to generate situation data structure(s), each situation data structure comprising data elements corresponding to entities, and first relationships between entities, identified by the video parser as present in images of the input video data sequence. First machine learning computer model(s) operate on the situation data structure(s) to predict second relationship(s) between the situation data structure(s). Second machine learning computer model(s) execute on a received input question to predict an executable program to execute to answer the received question. The program is executed on the situation data structure(s) and predicted second relationship(s). An answer to the question is output based on results of executing the program.

Neural-symbolic action transformers for video question answering

A vehicle light signal detection and recognition method, system, and computer program product include bounding, using a coarse attention module, one or more regions of an image of an automobile including at least one of a brake light and a signal light generated by automobile signals which include illuminated sections to generate one or more bounded region, removing, using a fine attention module, noise from the one or more bounded regions to generate one or more noise-free bounded regions, and identifying the at least one of the brake light and the signal light from the one or more noise-free bounded regions.

Coarse-to-fine attention networks for light signal detection and recognition

A processor may receive a video including a plurality of video frames in sequence and a question regarding the video. For a video frame in the plurality of video frames, a processor may parse the video frame into objects and relationships between the objects, and create a subgraph of nodes representing objects and edges representing the relationships, where parsing and creating are performed for each video frame in the plurality of video frames, where a plurality of subgraphs can be created. A processor may create a hypergraph connecting subgraphs by learning relationships between the nodes of the subgraphs, where a hyper-edge is created to represent a relationship between at least one node of one subgraph and at least one node of another subgraph in the plurality of subgraphs. A processor may generate an answer to the question based on the hypergraph.

Transformers for real world video question answering

One or more computer processors improve action recognition by removing inference introduced by visual appearances of objects within a received video segment. The one or more computer processors extract appearance information and structure information from a received video segment. The one or more computer processors calculate a factual inference (TE) for the received video segment utilizing the extracted appearance information and structure information. The one or more computer processors calculate a counterfactual debiasing inference (NDE) for the received video segment. The one or more computer processors calculate a total indirect effect (TIE) by subtracting the calculated counterfactual debiased inference from the calculated factual inference. The one or more computer processors action recognize the received video segment by selecting a classification result associated with a highest calculated TIE.

Counterfactual debiasing inference for compositional action recognition

A bi-directional spatial-temporal transformer neural network (BDSTT) is trained to predict original coordinates of a skeletal joint in a specific frame through relative relationships of the skeletal joint to other joints and to the state of the skeletal joint in other frames. Obtain a plurality of frames comprising coordinates of the skeletal joint and coordinates of other joints. Produce a spatially masked frame by masking the original coordinates of the skeletal joint. Provide the specific frame, the spatially masked frame, and at least one more frame to a coordinate prediction head of the BDSTT. Obtain, from the coordinate prediction head, a prediction of coordinates for the skeletal joint. Adjust parameters of the BDSTT until a mean-squared error, between the prediction of coordinates for the skeletal joint and the original coordinates of the skeletal joint, converges.

Skeleton-based action recognition using bi-directional spatial-temporal transformer

Unsupervised learning for video classification. One or more features from one or more video clips are extracted using a spatial-temporal encoder. The one or more extracted features are processed, using a video instance discrimination task, to generate a classification label, the classification label indicating whether two of the video clips are from a same video. The one or more extracted features are processed, using a pair-wise speed discrimination task, to generate a comparison label, the comparison label indicating a relative playback speed between two given video clips. A search is performed in a video database for a video that is similar to a given video based on the comparison label.

Unsupervised video representation learning

Systems, computer-implemented methods, and/or computer program products facilitating a process to identify and respond to a primary electronic message are provided. According to an embodiment, a system can comprise a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can include a determination component can determine that a primary electronic message has not received a response electronic message. An analysis component can generate a generated electronic message addressing the informational or emotional content of the primary electronic message. In one or more embodiments, an updating component can update the analytical model based on one or more feedbacks to the generated electronic message, where the analytical model can remain active while being updated. The one or more feedbacks can comprise a feedback from an entity-in-the-loop monitoring outputs of the analytical model including the generated electronic message.

Automated conversational response generation

Interpretation maps of convolutional neural networks having certifiable robustness using R&#xe9;nyi differential privacy are provided. In one aspect, a method for generating an interpretation map includes: adding generalized Gaussian noise to an image x to obtain T noisy images, wherein the generalized Gaussian noise constitutes perturbations to the image x; providing the T noisy images as input to a convolutional neural network; calculating T noisy interpretations of output from the convolutional neural network corresponding to the T noisy images; re-scaling the T noisy interpretations using a scoring vector &#x3c5; to obtain T re-scaled noisy interpretations; and generating the interpretation map using the T re-scaled noisy interpretations, wherein the interpretation map is robust against the perturbations.

Certifiably robust interpretation

Dual-modality relation networks for audio-visual event localization can be provided. A video feed for audio-visual event localization can be received. Based on a combination of extracted audio features and video features of the video feed, informative features and regions in the video feed can be determined by running a first neural network. Based on the informative features and regions in the video feed determined by the first neural network, relation-aware video features can be determined by running a second neural network. Based on the informative features and regions in the video feed, relation-aware audio features can be determined by running a third neural network. A dual-modality representation can be obtained based on the relation-aware video features and the relation-aware audio features by running a fourth neural network. The dual-modality representation can be input to a classifier to identity an audio-visual event in the video feed.

Dual-modality relation networks for audio-visual event localization

A method for implementing video frame synthesis using a tensor neural network includes receiving input video data including one or more missing frames, converting the input video data into an input tensor, generating, through tensor completion based on the input tensor, output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including a plurality of phases implementing a tensor iterative shrinkage thresholding algorithm (ISTA), and obtaining a loss function based on the output video data.

Video frame synthesis using tensor neural networks

Machine learning-based techniques for summarizing collections of data such as image and video data leveraging side information obtained from related (e.g., video) data are provided. In one aspect, a method for video summarization includes: obtaining related videos having content related to a target video; and creating a summary of the target video using information provided by the target video and side information provided by the related videos to select portions of the target video to include in the summary. The side information can include video data, still image data, text, comments, natural language descriptions, and combinations thereof.

Growing community of inventors

Cambridge, MA, United States of America

Bo Wu