Fmr Corp.

Sijing Lv

Hua Hao

Byung Chun

Tieyi Guo

Yong Zou

Bing Cui

Methods and apparatuses for multimodal enhancement of interactions in conversation service applications include a server that establishes a chat-based communication session between a conversation service application and a client computing device. Query input is captured from a user during the session. The server converts an audio segment corresponding to a video into a video transcript using speech recognition, and determines a transcript portion for the video that is responsive to the query input using a dense passage retriever model. The server selects video frames from the video that correspond to the transcript portion and determines, for the selected frames, a predicted relevancy between the selected frame and the query input. The server generates a response to the query input comprising the selected frames based upon the relevancy. The server transmits the response to the client device for display.

Growing community of inventors

Dalian, China

Sijing Lv