Adobe, Inc.

University of North Carolina at Chapel Hill

Trung Huu Bui

Franck Dernoncourt

Walter Chang

Doo Soon Kim

Hung Hai Bui

Zhe Lin

Quan Anh Tran

Tuan Manh Lai

Seunghyun Yoon

Ajinkya Gorakhnath Kale

Quan Hung Tran

Hailin Jin

Zhaowen Wang

Carl Dockhorn

Chen Fang

Walter Wei-Tuh Chang

Michael F Kraley

Lidan Wang

Sasha Spala

Yu Gong

Subhadeep Dey

Pranjal Daga

Ramesh Radhakrishna Manuvinakurike

Sachin Soni

Baldo Antonio Faieta

Ning Xu

Shawn Alan Gaither

Tong Sun

Kushal Kafle

Jing Shi

Nicholas Miller

Kate Sousa

Benjamin Leviant

Fengbin Chen

Venkat Barakam

Yipin Zhou

Nham Le

Anirban Roychowdhury

Francois Guerin

Tushar Dublish

Joon Kim

Scott David Cohen

Jonathan Worthen Brandt

Kevin Gary Smith

Jennifer Anne Healey

Chen Chen

Mihir Naware

Nedim Lipka

Mingyang Ling

Natwar Modani

Cuong D Nguyen

Seokhwan Kim

Varun Manjunatha

Moumita Sinha

Xiang Chen

Sheng Li

Jane Elizabeth Hoffswell

Yinglan Ma

Seung-hyun Yoon

John Thomas Kucera

Timothy Jeewun Ganter

Sean Fitzgerald

Sangwoo Cho

Mohit Bansal

Hao Tan

Nham Van Le

Kan Chen

Jinfeng Xiao

Arman Cohan

Adyasha Maharana

Yeon Seonwoo

Jielin Qiu

Meryem M'hamdi

Hyounghun Kim

Cesa Salaam

Zahra Rahimi

Jaemin Cho

Jianguo Zhang

Qiuyu Chen

Xuanli He

Chenyun Wu

Nikita Soni

Roger K Brooks

Robert S N Dates

Jacqueline Brixey

Kishore Aradhya

Tzu-Hsiang Lin

Khalil Mrini

Anthony M Colas

Systems and methods for natural language processing are described. Embodiments of the present disclosure identify a task set including a plurality of pseudo tasks, wherein each of the plurality of pseudo tasks includes a support set corresponding to a first natural language processing (NLP) task and a query set corresponding to a second NLP task; update a machine learning model in an inner loop based on the support set; update the machine learning model in an outer loop based on the query set; and perform the second NLP task using the machine learning model.

Cross-lingual meta-transfer learning adaptation to natural language understanding

A computer-implemented method is disclosed for determining one or more characteristics of a dialog between a computer system and user. The method may comprise receiving a system utterance comprising one or more tokens defining one or more words generated by the computer system; receiving a user utterance comprising one or more tokens defining one or more words uttered by a user in response to the system utterance, the system utterance and the user utterance forming a dialog context; receiving one or more utterance candidates comprising one or more tokens; for each utterance candidate, generating an input sequence combining the one or more tokens of each of the system utterance, the user utterance, and the utterance candidate; and for each utterance candidate, evaluating the generated input sequence with a model to determine a probability that the utterance candidate is relevant to the dialog context.

Methods and systems for determining characteristics of a dialog between a computer and a user

In some examples, an augmented reality (AR) server receives instructional data to be rendered in AR. The AR rendering server extracts multiple instruction steps from the instructional data and determines multiple spatial identifiers associated with the multiple instruction steps respectively. The multiple spatial identifiers correspond to multiple spatial objects in a real-world environment. The AR rendering server then generates AR rendering data for displaying the multiple instruction steps on an AR device at selected locations associated with the multiple spatial objects in the real-world environment. The AR rendering data is then transmitted to the AR device.

Rendering and anchoring instructional data in augmented reality with context awareness

Systems and methods for color prediction are described. Embodiments of the present disclosure receive an image that includes an object including a color, generate a color vector based on the image using a color classification network, where the color vector includes a color value corresponding to each of a set of colors, generate a bias vector by comparing the color vector to teach of a set of center vectors, where each of the set of center vectors corresponds to a color of the set of colors, and generate an unbiased color vector based on the color vector and the bias vector, where the unbiased color vector indicates the color of the object.

Systems and methods for predicting long-tail colors in images

Systems and methods for natural language processing are described. Embodiments are configured to receive a structured representation of a search query, wherein the structured representation comprises a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, wherein the modification expression comprises a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

Scene graph modification based on natural language commands

Systems and methods for text processing are described. Embodiments of the present disclosure receive a query comprising a natural language expression; extract a plurality of mentions from the query; generate a relation vector between a pair of the plurality of mentions using a relation encoder network, wherein the relation encoder network is trained using a contrastive learning process where mention pairs from a same document are labeled as positive samples and mention pairs from different documents are labeled as negative samples; combine the plurality of mentions with the relation vector to obtain a virtual knowledge graph of the query; identify a document corresponding to the query by comparing the virtual knowledge graph of the query to a virtual knowledge graph of the document; and transmit a response to the query, wherein the response includes a reference to the document.

Virtual knowledge graph construction for zero-shot domain-specific document retrieval

Systems and methods for coreference resolution are provided. One aspect of the systems and methods includes inserting a speaker tag into a transcript, wherein the speaker tag indicates that a name in the transcript corresponds to a speaker of a portion of the transcript; encoding a plurality of candidate spans from the transcript based at least in part on the speaker tag to obtain a plurality of span vectors; extracting a plurality of entity mentions from the transcript based on the plurality of span vectors, wherein each of the plurality of entity mentions corresponds to one of the plurality of candidate spans; and generating coreference information for the transcript based on the plurality of entity mentions, wherein the coreference information indicates that a pair of candidate spans of the plurality of candidate spans corresponds to a pair of entity mentions that refer to a same entity.

Systems and methods for coreference resolution

Embodiments of the disclosure provide a machine learning model for generating a predicted executable command for an image. The learning model includes an interface configured to obtain an utterance indicating a request associated with the image, an utterance sub-model, a visual sub-model, an attention network, and a selection gate. The machine learning model generates a segment of the predicted executable command from weighted probabilities of each candidate token in a predetermined vocabulary determined based on the visual features, the concept features, current command features, and the utterance features extracted from the utterance or the image.

Systems and methods for image processing using natural language

Systems and methods for video segmentation and summarization are described. Embodiments of the present disclosure receive a video and a transcript of the video; generate visual features representing frames of the video using an image encoder; generate language features representing the transcript using a text encoder, wherein the image encoder and the text encoder are trained based on a correlation between training visual features and training language features; and segment the video into a plurality of video segments based on the visual features and the language features.

Multimodal unsupervised video temporal segmentation for summarization

Techniques for training a language model for code switching content are disclosed. Such techniques include, in some embodiments, generating a dataset, which includes identifying one or more portions within textual content in a first language, the identified one or more portions each including one or more of offensive content or non-offensive content; translating the identified one or more salient portions to a second language; and reintegrating the translated one or more portions into the textual content to generate code-switched textual content. In some cases, the textual content in the first language includes offensive content and non-offensive content, the identified one or more portions include the offensive content, and the translated one or more portions include a translated version of the offensive content. In some embodiments, the code-switched textual content is at least part of a synthetic dataset usable to train a language model, such as a multilingual classification model.

Generating synthetic code-switched data for training language models

The present disclosure relates to systems, methods, and non-transitory computer readable media for determining speech emotion. In particular, a speech emotion recognition system generates an audio feature vector and a textual feature vector for a sequence of words. Further, the speech emotion recognition system utilizes a neural attention mechanism that intelligently blends together the audio feature vector and the textual feature vector to generate attention output. Using the attention output, which includes consideration of both audio and text modalities for speech corresponding to the sequence of words, the speech emotion recognition system can apply attention methods to one of the feature vectors to generate a hidden feature vector. Based on the hidden feature vector, the speech emotion recognition system can generate a speech emotion probability distribution of emotions among a group of candidate emotions, and then select one of the candidate emotions as corresponding to the sequence of words.

Bi-directional recurrent encoders with multi-hop attention for speech emotion recognition

Systems and methods for image captioning are described. One or more aspects of the systems and methods include generating a training caption for a training image using an image captioning network; encoding the training caption using a multi-modal encoder to obtain an encoded training caption; encoding the training image using the multi-modal encoder to obtain an encoded training image; computing a reward function based on the encoded training caption and the encoded training image; and updating parameters of the image captioning network based on the reward function.

Image captioning

The present disclosure relates to systems, methods, and non-transitory computer-readable media that edit digital images using combinations of speech input and gesture interactions. For instance, in some embodiments, the disclosed systems receive speech input from a client device displaying a digital image within a graphical user interface, the digital image portraying an object. Additionally, the disclosed systems detect, via the graphical user interface, one or more gesture interactions with respect to the object of the digital image. Based on the speech input, the disclosed systems determine an edit for the object of the digital image indicated by the one or more gesture interactions. Further, the disclosed systems modify the object within the digital image using the edit indicated by the one or more gesture interactions.

Modifying digital images using combinations of direct interactions with the digital images and context-informing speech input

Systems and methods for natural language processing are described. One or more aspects of a method, apparatus, and non-transitory computer readable medium include receiving a text phrase; encoding the text phrase using an encoder to obtain a hidden representation of the text phrase, wherein the encoder is trained during a first training phrase using self-supervised learning based on a first contrastive loss and during a second training phrase using supervised learning based on a second contrastive learning loss; identifying an intent of the text phrase from a predetermined set of intent labels using a classification network, wherein the classification network is jointly trained with the encoder in the second training phase; and generating a response to the text phrase based on the intent.

Intent detection

System and methods for a text summarization system are described. In one example, a text summarization system receives an input utterance and determines whether the utterance should be included in a summary of the text. The text summarization system includes an embedding network, a convolution network, an encoding component, and a summary component. The embedding network generates a semantic embedding of an utterance. The convolution network generates a plurality of feature vectors based on the semantic embedding. The encoding component identifies a plurality of latent codes respectively corresponding to the plurality of feature vectors. The summary component identifies a prominent code among the latent codes and to select the utterance as a summary utterance based on the prominent code.

Topical vector-quantized variational autoencoders for extractive summarization of video transcripts

Growing community of inventors

San Jose, CA, United States of America

Trung Huu Bui