Adobe, Inc.

Amazon Technologies, Inc.

Seokhwan Kim

Walter Chang

Franck Dernoncourt

Hailin Jin

Yang Liu

Ning Xu

Wentian Zhao

Robinson Piramuthu

Dilek Zeynep Hakkani-Tur

Gokhan Tur

Trung Huu Bui

Chen Fang

Nedim Lipka

Walter Wei-Tuh Chang

Doo Soon Kim

Frederic Thevenet

Carl Dockhorn

Yinglan Ma

Laurie Marie Byrum

Sean Fitzgerald

Ragunandan Rao Malangully

Chan Young Park

Behnam Hedayatnia

Sanqiang Zhao

Eshan Bhatnagar

Di Jin

Alexandros Papangelis

Spandana Gella

Sean MacAvaney

Devamanyu Hazarika

Zahra Rahimi

Mahdi Namazifar

Taehwan Kim

Chao Zhao

Techniques for generating opinion-based content responsive to a user input are described. The system may receive a user input, and determine dialog context data corresponding to a dialog between a user and the system, and including the user input. The system may determine generation of content responsive to the user input requires opinion-based knowledge, and may extract entities from the dialog context data, and determine natural language data of a knowledge base that includes entities similar to the extracted entities. The system may processes the natural language data and the dialog context data to determine a subset of the natural language data that is responsive to the user input. The system may generate output data responsive to the user input using the responsive natural language data and the dialog context.

Opinion-based natural language response generation

Techniques for rendering visual content, in response to one or more utterances, are described. A device receives one or more utterances that define a parameter(s) for desired output content. A system (or the device) identifies natural language data corresponding to the desired content, and uses natural language generation processes to update the natural language data based on the parameter(s). The system (or the device) then generates an image based on the updated natural language data. The system (or the device) also generates video data of an avatar. The device displays the image and the avatar, and synchronizes movements of the avatar with output of synthesized speech of the updated natural language data. The device may also display subtitles of the updated natural language data, and cause a word of the subtitles to be emphasized when synthesized speech of the word is being output.

Virtual conversational companion

The present disclosure relates to generating digital responses based on digital dialog states generated by a neural network having a dynamic memory network architecture. For example, in one or more embodiments, the disclosed system provides a digital dialog having one or more segments to a dialog state tracking neural network having a dynamic memory network architecture that includes a set of multiple memory slots. In some embodiments, the dialog state tracking neural network further includes update gates and reset gates used in modifying the values stored in the memory slots. For instance, the disclosed system can utilize cross-slot interaction update/reset gates to accurately generate a digital dialog state for each of the segments of digital dialog. Subsequently, the system generates a digital response for each segment of digital dialog based on the digital dialog state.

Utilizing a dynamic memory network for state tracking

This disclosure relates to methods, non-transitory computer readable media, and systems that can classify term sequences within a source text based on textual features analyzed by both an implicit-class-recognition model and an explicit-class-recognition model. For example, by applying machine-learning models for both implicit and explicit class recognition, the disclosed systems can determine a class corresponding to a particular term sequence within a source text and identify the particular term sequence reflecting the class. The dual-model architecture can equip the disclosed systems to apply (i) the implicit-class-recognition model to recognize implicit references to a class in source texts and (ii) the explicit-class-recognition model to recognize explicit references to the same class in source texts.

Classifying terms from source texts using implicit and explicit class-recognition-machine-learning models

The present disclosure relates to systems, methods, and non-transitory computer-readable media for generating a response to a question received from a user during display or playback of a video segment by utilizing a query-response-neural network. The disclosed systems can extract a query vector from a question corresponding to the video segment using the query-response-neural network. The disclosed systems further generate context vectors representing both visual cues and transcript cues corresponding to the video segment using context encoders or other layers from the query-response-neural network. By utilizing additional layers from the query-response-neural network, the disclosed systems generate (i) a query-context vector based on the query vector and the context vectors, and (ii) candidate-response vectors representing candidate responses to the question from a domain-knowledge base or other source. To respond to a user's question, the disclosed systems further select a response from the candidate responses based on a comparison of the query-context vector and the candidate-response vectors.

Generating responses to queries about videos utilizing a multi-modal neural network with attention

In implementations of answering questions during video playback, a video system can receive a question related to a video at a timepoint of the video during playback of the video, and determine audio sentences of the video that occur within a segment of the video that includes the timepoint. The video system can generate a classification vector from words of the question and the audio sentences, and determine an answer to the question utilizing the classification vector. The video system can obtain answer candidates, and the answer to the question can be selected as one of the answer candidates based on matching the classification vector to one of the answer vectors.

Answering questions during video playback

The present disclosure relates to utilizing a deep recurrent neural network for accurately performing punctuation restoration. For example, the disclosed systems can provide a sequence of words to a punctuation restoration neural network having multiple bi-directional recurrent layers and one or more neural attention mechanisms. In one or more embodiments, the punctuation restoration neural network incorporates layer-wise attentions and/or multi-head attention. The disclosed systems can utilize the punctuation restoration neural network to generate probabilities for each word, indicating the likelihood that each possible punctuation mark is associated with that word. Based on these probabilities, the disclosed systems can generate a punctuated transcript that includes punctuation before the appropriate words.

Utilizing deep recurrent neural networks with layer-wise attention for punctuation restoration

Generating a response to a user query utilizing visual features of a video segment and a query-response-neural network

Methods and systems are provided for identifying subparts of a text. A neural network system can receive a set of sentences that includes context sentences and target sentences that indicate a decision point in a text. The neural network system can generate context vector sentences and target sentence vectors by encoding context from the set of sentences. These context sentence vectors can be weighted to focus on relevant information. The weighted context sentence vectors and the target sentence vectors can then be used to output a label for the decision point in the text.

Automatic text segmentation based on relevant context

The present disclosure relates to systems, non-transitory computer-readable media, and methods that generate ground truth annotations of target utterances in digital image editing dialogues in order to create a state-driven training data set. In particular, in one or more embodiments, the disclosed systems utilize machine and user defined tags, machine learning model predictions, and user input to generate a ground truth annotation that includes frame information in addition to intent, attribute, object, and/or location information. In at least one embodiment, the disclosed systems generate ground truth annotations in conformance with an annotation ontology that results in fast and accurate digital image editing dialogue annotation.

Generating ground truth annotations corresponding to digital image editing dialogues for training state tracking models

Utilizing a dynamic memory network to track digital dialog states and generate responses

Techniques are disclosed for generating a structured transcription from a speech file. In an example embodiment, a structured transcription system receives a speech file comprising speech from one or more people and generates a navigable structured transcription object. The navigable structured transcription object may comprise one or more data structures representing multimedia content with which a user may navigate and interact via a user interface. Text and/or speech relating to the speech file can be selectively presented to the user (e.g., the text can be presented via a display, and the speech can be aurally presented via a speaker).

Emphasizing key points in a speech file and structuring an associated transcription

Growing community of inventors

San Jose, CA, United States of America

Seokhwan Kim