Mitsubishi Electric Research Laboratories, Inc.

University of Minnesota

The Australian National University

Mitsubishi Electric Corporation

Anoop Cherian

Tim Marks

Jonathan Le Roux

Chiori Hori

Alan Sullivan

Takaaki Hori

Anthony Vetro

Ye Wang

Vassilios Morellas

Bret Harsham

Nikolaos Papanikolopoulos

Siheng Chen

Shijie Geng

Alexander Truskinovsky

Goncalo José Dias Pais

Safa Medin

Peng Gao

Aeron Shuchin

Moitreya Chatterjee

Jue Wang

Sudipta Paul

Xiulong Liu

Panagiotis Stanitsas

An anomaly detector for controlling a system is provided. The system comprises one or multiple tools to perform one or multiple tasks. The anomaly detector collects a feedforward signal indicative of a sequence of control inputs to the plurality of actuators and a feedback signal indicative of a sequence of outputs of the system caused by the plurality of actuators operated based on the sequence of control inputs. The anomaly detector further combines input state variables extracted from the feedforward signal and output state variables extracted from the feedback signal to form a sequence of extended states of the system. The attention model further encodes the sequence of extended states to produce an encoding of each extended state of the sequence of extended states in a latent space. The anomaly detector further detects an anomaly in a current operation of the system based on the encoded sequence of extended states.

System and method for anomaly detection using an attention model

A controller for controlling a robot is provided. The controller comprises a hierarchical multimodal reinforcement learning (RL) neural network including a first level controller and three second level controllers. The second level controllers comprise a first sub level controller to receive input data based on predefined questions, a second sub level controller to receive the input data by generating a validation question based on state of the RL neural network and a third sub level controller to determine the input data based on state of the RL neural network. The controller is configured to select one of the second level controllers using the first level controller to perform a first interaction relating to a task based on the state of the RL neural network; generate a control command using the selected second level controller based on the corresponding input data; and control operation of the robot by executing control command.

System and method for controlling a robot

An imaging controller is provided for segmenting instances from depth images including objects to be manipulated by a robot. The imaging controller includes an input interface configured to receive a depth image that includes objects, a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image, and a processor, coupled with the memory, configured to perform the instructions to segment a pickable instance using the trained neural network. The instructions include steps of selecting a tallest point in the depth image, defining a region using a shape such that the region surrounds the tallest point, sampling points in the region of the depth image, computing depth-geodesics between the tallest point and the sampled points, submitting the depth-geodesics to the neural network to segment the pickable instance among instances of the objects in the depth image, and an output interface configured to output a geometrical feature of the pickable instance to a manipulator controller of the robot.

Discriminative 3D shape modeling for few-shot instance segmentation

Embodiments disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio video representation corresponds to a graph of nodes connected by edges. A node in the graph is indicative of the video features of an object in the scene. An edge in the graph connecting two nodes indicates an interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with audio features of a sound generated by the interaction of the corresponding two objects. The graph of the audio-video representation of the scene may be used to perform a variety of different tasks. Examples of the tasks include one or a combination of an action recognition, an anomaly detection, a sound localization and enhancement, a noisy-background sound removal, and a system control.

Method and system for scene-aware audio-video representation

An artificial intelligence (AI) system that includes a processor configured to execute modules of the AI system. The modules comprise a feature extractor, an adversarial noise generator, a compressor and a classifier. The feature extractor is trained to process input data to extract features of the input data for classification of the input data. The adversarial noise generator is trained to generate noise data for distribution of features of the input data such that a misclassification rate of corrupted features that include the extracted features corrupted with the generated noise data is greater than a misclassification rate of the extracted features. The compressor is configured to compress the extracted features. The compressed features are closer to the extracted features than to the corrupted features. The classifier is trained to classify the compressed features.

Artificial intelligence system for classification of data based on contrastive learning

Present disclosure discloses an image processing system and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class (e.g., human faces). A 2D input image of a 3D object of the predetermined class is manipulated by manipulating physical properties of the 3D object, such as a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The physical properties are extracted from the 2D input image using a neural network that is trained to reconstruct the 2D input image. The 2D input image is reconstructed by disentangling the physical properties from pixels of the 2D input image using multiple subnetworks. The disentangled physical properties produced by the multiple subnetworks are combined into a 2D output image using a differentiable renderer.

System and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects

System and method for generating verisimilar images from real depth images. Train a generative adversarial neural network (GAN) by accessing test depth images having identical instances as instances of a real depth image. Input the test depth images in the generator to generate estimated depth images representing an implicit three-dimensional model of the object. Input, each estimated depth image into a discriminator to obtain a loss and into a pose encoder to obtain a matching loss. Iteratively repeat processes until the losses are minimized to a threshold, to end training. Identify the instances in the real image using the trained GAN pose encoder, to produce a pose transformation matrix for each instance in the real image. Identify pixels in the depth images corresponding to the instances of the real image and merge the pixels for the depth images to form an instance segmentation map for the real depth image.

InSeGAN: a generative approach to instance segmentation in depth images

A navigation system for providing driving instructions to a driver of a vehicle traveling on a route is provided. The driving instructions are generated by executing a multimodal fusion method that comprises extracting features from sensor measurements, annotating the features with directions for the vehicle to follow the route with respect to objects sensed by the sensors, and encoding the annotated features with a multimodal attention neural network to produce encodings. The encodings are transformed into a common latent space, and the transformed encodings are fused using an attention mechanism producing an encoded representation of the scene. The method further comprises decoding the encoded representation with a sentence generation neural network to generate a driving instruction and submitting the driving instruction to an output device.

Method and system for scene-aware interaction

Embodiments of the present disclosure discloses a scene-aware video encoder system. The scene-aware encoder system transforms a sequence of video frames of a video of a scene into a spatio-temporal scene graph. The spatio-temporal scene graph includes nodes representing one or multiple static and dynamic objects in the scene. Each node of the spatio-temporal scene graph describes an appearance, a location, and/or a motion of each of the objects (static and dynamic objects) at different time instances. The nodes of the spatio-temporal scene graph are embedded into a latent space using a spatio-temporal transformer encoding different combinations of different nodes of the spatio-temporal scene graph corresponding to different spatio-temporal volumes of the scene. Each node of the different nodes encoded in each of the combinations is weighted with an attention score determined as a function of similarities of spatio-temporal locations of the different nodes in the combination.

Scene-aware video encoder system and method

A system includes an imager, a processor, and an output module. The imager is configured to provide a plurality of tissue images. The processor is coupled to the imager and is configured to receive the plurality of images. The processor is coupled to a memory. The memory has instructions for determining classification of a region of tissue associated with the plurality of tissue images. Determining classification includes fusing discriminator outputs from a region covariance descriptor and from a normalized color histogram discriminator. The output module is coupled to the processor. The output module is configured to provide a three dimensional representation of the tissue.

Computer vision for cancerous tissue recognition

A scene captioning system is provided. The scene captioning system includes an interface configured to acquire a stream of scene data signals including frames and sound data, a memory to store a computer-executable scene captioning model including a scene encoder, a timing decoder, a timing detector, and a caption decoder, wherein the audio-visual encoder is shared by the timing decoder and the timing detector and the caption decoder, and a processor, in connection with the memory. The processor is configured to perform steps of extracting scene features from the scene data signals by use of the audio-visual encoder, determining a timing of generating a caption by use of the timing detector, wherein the timing is arranged an early stage of the stream of scene data signals, and generating the caption based on the scene features by using the caption decoder according to the timing.

Low-latency captioning system

Embodiments of the present disclosure disclose an anomaly detector for detecting an anomaly in a sequence of poses of a human performing an activity. The anomaly detector includes an input interface configured to accept input data indicative of a distribution of the sequence of poses, a memory configured to store a discriminative one-class classifier having a pair of complementary classifiers bounding normal distribution of pose sequences in a reproducing kernel Hilbert space (RKHS), a processor configured to embed the input data into an element of the RKHS and classify the embedded data using the discriminative one-class classifier, and an output interface configured to render a classification result.

Anomaly detector for detecting anomaly using complementary classifiers

A computer-implemented method for training a dialogue response generation system and the dialogue response generation system are provided. The method includes arranging a first multimodal encoder-decoder for the dialogue response generation or video description having a first input and a first output, wherein the first multimodal encoder-decoder has been pretrained by training audio-video datasets with training video description sentences, arranging a second multimodal encoder-decoder for dialog response generation having a second input and a second output, providing first audio-visual datasets with first corresponding video description sentences to the first input of the first multimodal encoder-decoder, wherein the first encoder-decoder generates first output values based on the first audio-visual datasets with the first corresponding description sentences, providing the first audio-visual datasets excluding the first corresponding video description sentences to the second multimodal encoder-decoder. In this case, the second multimodal encoder-decoder generates second output values based on the first audio-visual datasets without the first corresponding video description sentences.

System and method for a dialogue response generation system

A scene aware dialog system includes an input interface to receive a sequence of video frames, contextual information, and a query and a memory configured to store neural networks trained to generate a response to the input query by analyzing one or combination of input sequence of video frames and the input contextual information. The system further includes a processor configured to detect and classify objects in each video frame of the sequence of video frames; determine relationships among the classified objects in each of the video frame; extract features representing the classified objects and the determined relationships for each of the video frame to produce a sequence of feature vectors; and submit the sequence of feature vectors, the input query and the input contextual information to the neural network to generate a response to the input query.

Growing community of inventors

Belmont, MA, United States of America

Anoop Cherian