Mitsubishi Electric Research Laboratories, Inc.

Shijie Geng

Jonathan Le Roux

Anoop Cherian

Chiori Hori

Peng Gao

A scene aware dialog system includes an input interface to receive a sequence of video frames, contextual information, and a query and a memory configured to store neural networks trained to generate a response to the input query by analyzing one or combination of input sequence of video frames and the input contextual information. The system further includes a processor configured to detect and classify objects in each video frame of the sequence of video frames; determine relationships among the classified objects in each of the video frame; extract features representing the classified objects and the determined relationships for each of the video frame to produce a sequence of feature vectors; and submit the sequence of feature vectors, the input query and the input contextual information to the neural network to generate a response to the input query.

Growing community of inventors

Piscataway, NY, United States of America

Shijie Geng