Fmr Corp.

Byung Chun

Tieyi Guo

Hua Hao

Allen Detmer

Pengrui Sun

Chao Yu

Hao Fu

Chen Bi

Yong Zou

Bing Cui

Veronica Grigoras

Keith Pepin

Brian Christoper Mansfield

Wenbin Ren

Christopher Lackey

Philipp Hanes

Amanda Bedard

Sijing Lv

Manikanta Panati

Jia You

Ou Li

David Favicchio

Chad Iverson

Yachao He

Methods and apparatuses for conversation dialogue orchestration in virtual assistant communication sessions include a server that establishes a chat session between a virtual assistant (VA) application and a client device. The VA application captures an utterance generated by a user and processes the utterance to instantiate a dialogue behavior tree comprising workflow agents each associated with executable code for completing a corresponding workflow action. The VA application traverses the behavior tree to generate a response to the utterance, including evaluating one or more conditions associated with a workflow agent to determine whether to execute the code in the workflow agent, and when the conditions associated with the workflow agent are met, executing the code to complete the workflow action and storing a sub-response in a dialogue memory. The VA application coalesces the sub-responses to generate a final response and transmits the final response to the client device.

Conversation dialogue orchestration in virtual assistant communication sessions

Methods and apparatuses for multimodal enhancement of interactions in conversation service applications include a server that establishes a chat-based communication session between a conversation service application and a client computing device. Query input is captured from a user during the session. The server converts an audio segment corresponding to a video into a video transcript using speech recognition, and determines a transcript portion for the video that is responsive to the query input using a dense passage retriever model. The server selects video frames from the video that correspond to the transcript portion and determines, for the selected frames, a predicted relevancy between the selected frame and the query input. The server generates a response to the query input comprising the selected frames based upon the relevancy. The server transmits the response to the client device for display.

Multimodal enhancement of interactions in conversation service applications

Methods and apparatuses are described for enhancing chatbot recognition of user intent through graph analysis. A computing device classifies user messages associated with a chat session as either comprehensible user messages that are mapped to a user intent or incomprehensible utterances that are not mapped to a user intent. The computing device arranges the comprehensible messages into first clusters and arranges the incomprehensible messages into second clusters. The computing device generates a first sub-graph for each first cluster and generates a second sub-graph for each second cluster. The computing device calculates a graph kernel matrix using the sub-graphs and trains a classification model using the graph kernel matrix. The computing device executes the trained classification model using the graph kernel matrix as input to generate a predicted intent identifier for the incomprehensible messages.

Enhancing chatbot recognition of user intent through graph analysis

Methods and apparatuses are described for automatic refinement of intent classification for virtual assistant applications. A computing device creates utterance groups comprising messages. The device determines, using a first machine learning (ML) classifier, a first predicted intent associated with each utterance group and determines, using a second ML classifier, a second predicted intent associated with each utterance group. The device combines the first and second predicted intents to generate a final predicted intent, determines, for each incomprehensible utterance group, whether the final predicted intent overlaps with the final predicted intent for other utterance groups, and selects messages having no overlapping intent to create new intents. The device determines, for each comprehensible utterance group, whether the final predicted intent overlaps with a preassigned intent for other utterance groups and combines the messages that have an overlapping intent to create and label a new utterance group with the overlapping intent.

Automatic refinement of intent classification for virtual assistant applications

Methods and apparatuses are described for capturing and utilizing context data in virtual assistant communication sessions. A chat-based communication session is established between a virtual assistant application and a remote device. An event manager captures a first data event of the virtual assistant application. A parent software application receives the first data event and selects one or more session context data elements using a callback function of the first data event. The event manager captures a second data event of the parent software application comprising the session context data elements. The virtual assistant receives the second data event and inserts one or more messages comprising the session context data elements from the second data event into the chat-based communication session.

Capturing and utilizing context data in virtual assistant communication sessions

Methods and systems are described for generating integrated intelligent content overlays for media content streams. A server computing device receives a video content stream from a video data source. The server extracts a corpus of machine-recognizable text from the video content stream, the corpus of machine-recognizable text corresponding to at least one of audio or closed captioning text associated with the video content stream. The server identifies one or more entity names contained in the corpus of machine-recognizable text. The server determines a set of content keywords associated with each of the identified entity names. The server generates a content overlay for the video content stream comprising one or more layers that include graphical content relating to at least one of the sets of content keywords. The server integrates the content overlay into the video content stream to generate a customized video content stream.

Integrated intelligent overlay for media content streams

Growing community of inventors

Boston, MA, United States of America

Byung Chun