Adobe, Inc.

Handong Zhao

Zhe Lin

Sungchul Kim

Vlad Ion Morariu

Tong Sun

Sheng Li

Zhaowen Wang

Ryan A Rossi

Ajinkya Gorakhnath Kale

Tak Yeon Lee

Varun Manjunatha

Nikolaos Barmpalios

Curtis Michael Wigington

Christopher Alan Tensmeyer

Jiuxiang Gu

Kai Li

Zhankui He

Yikun Xian

Jianming Zhang

Quan Anh Tran

Rajiv Jain

Mingyang Ling

Sana Malik Lee

Subrata Mitra

Jason Wen Yong Kuen

Tong Yu

Ruiyi Zhang

Xiaowei Jia

Xihui Liu

Rui Wang

Eunyee Koh

William Brandon George

Chen Fang

Saayan Mitra

Fan Du

Georgios Theocharous

Gang Wu

Sumit Shekhar

Branislav Kveton

Trevor Paulsen

Somdeb Sarkhel

Ritwik Sinha

Zheng Wen

Kanak Vivek Mahadik

Vasanthi Holtcamp

Nikhil Belsare

Thomas M Jacobs

Ani Nenkova

Zening Qu

Duy-Trung Trong Dinh

Jason Kuen

Camille Harris

Caroline Jiwon Kim

Youngsuk Park

Zhiqiang Tao

Peizhao Li

Methods and systems disclosed herein relate generally to systems and methods for generating visual relationship graphs that identify relationships between objects depicted in an image. A vision-language application uses transformer encoders to generate a graph structure, in which the graph structure represents a dependency between a first region and a second region of an image. The dependency indicates that a contextual representation of the first region was derived, at least in part, by processing the second region. The contextual representation identifies a predicted identity of an image object depicted in the first region. The predicted identity is determined at least in part by identifying a relationship between the first region and other data objects associated with various modalities.

Self-supervised visual-relationship probing

Embodiments of the technology described herein describe a machine classifier capable of continually learning new classes through a continual few-shot learning approach. A natural language processing (NLP) machine classifier may initially be trained to identify a plurality of other classes through a conventional training process. In order to learn a new class, natural-language training data for a new class is generated. The training data for the new class may be few-shot training data. The training also uses synthetic training data that represents each of the plurality of other classes. The synthetic training data may be generated through a model inversion of the original classifier. The synthetic training data and the natural-language training data are used to retrain the NLP classifier to identify text in the plurality of other classes and the new class using.

Teaching a machine classifier to recognize a new class

Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

Generating scene graphs from digital images using external knowledge and image reconstruction

The technology described includes methods for pretraining a document encoder model based on multimodal self cross-attention. One method includes receiving image data that encodes a set of pretraining documents. A set of sentences is extracted from the image data. A bounding box for each sentence is generated. For each sentence, a set of predicted features is generated by using an encoder machine-learning model. The encoder model performs cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features is based on a masking function and the sentence. The set of masked-visual features is based on the masking function and the corresponding bounding box. A document-encoder model is pretrained based on the set of predicted features for each sentence and pretraining tasks. The pretraining tasks includes masked sentence modeling, visual contrastive learning, or visual-language alignment.

Unified pretraining framework for document understanding

The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating automatic suggestions to effectively modify the organization of an ingested data collection without destruction of the underlying raw data. In particular, in one or more embodiments, the disclosed systems utilize multiple machine learning models in sequence to determine likelihoods that the organizational structure of an ingested data collection should be modified in various ways. In response to generating these likelihoods, the disclosed systems generate corresponding automatic suggestions to modify the organization of the ingested data collection. In response to a detected selection of one or more of the automatic suggestions, the disclosed systems read data out of the ingested data collection in accordance with the selected automatic suggestions to effectively modify the organization of the ingested data collection.

Generating and executing automatic suggestions to modify data of ingested data collections without additional data ingestion

Systems and techniques for multi-task equidistant embedding are described that process categorical feature data to explore feature interactions. A digital analytics system enforces an equidistant relationship among features within a category while extracting high-order feature interactions by punishing both positive correlations and negative correlations among low-dimensional representations of different features. By enforcing an equidistant embedding, information is retained and accuracy is increased while higher order feature interactions are determined. Further, the digital analytics system shares knowledge among different tasks by connecting a shared network representation common to multiple tasks with exclusive network representations specific to particular tasks.

Multi-task equidistant embedding

Digital content search techniques are described that overcome the challenges found in conventional sequence-based techniques through use of a query-aware sequential search. In one example, a search query is received and sequence input data is obtained based on the search query. The sequence input data describes a sequence of digital content and respective search queries. Embedding data is generated based on the sequence input data using an embedding module of a machine-learning model. The embedding module includes a query-aware embedding layer that generates embeddings of the sequence of digital content and respective search queries. A search result is generated referencing at least one item of digital content by processing the embedding data using at least one layer of the machine-learning model.

Digital content query-aware sequential search

Digital content search techniques are described. In one example, the techniques are incorporated as part of a multi-head self-attention module of a transformer using machine learning. A localized self-attention module, for instance, is incorporated as part of the multi-head self-attention module that applies local constraints to the sequence. This is performable in a variety of ways. In a first instance, a model-based local encoder is used, examples of which include a fixed-depth recurrent neural network (RNN) and a convolutional network. In a second instance, a masking-based local encoder is used, examples of which include use of a fixed window, Gaussian initialization, and an adaptive predictor.

Locally constrained self-attentive sequential recommendation

Systems and methods for lifelong schema matching are described. The systems and methods include receiving data comprising a plurality of information categories, classifying each information category according to a schema comprising a plurality of classes, wherein the classification is performed by a neural network classifier trained based on a lifelong learning technique using a plurality of exemplar training sets, wherein each of the exemplar training sets includes a plurality of examples corresponding to one of the classes, and wherein the examples are selected based on a metric indicating how well each of the examples represents the corresponding class, and adding the data to a database based on the classification, wherein the database is organized according to the schema.

Lifelong schema matching

Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Domain adaptation for machine learning models

One example method involves operations for a processing device that include receiving, by a machine learning model trained to generate a search result, a search query for a text input. The machine learning model is trained by receiving pre-training data that includes multiple documents. Pre-training the machine learning model by generating, using an encoder, feature embeddings for each of the documents included in the pre-training data. The feature embeddings are generated by applying a masking function to visual and textual features in the documents. Training the machine learning model also includes generating, using the feature embeddings, output features for the documents by concatenating the feature embeddings and applying a non-linear mapping to the feature embeddings. Training the machine learning model further includes applying a linear classifier to the output features. Additionally, operations include generating, for display, a search result using the machine learning model based on the input.

Self-supervised document representation learning

Certain embodiments involve a method for generating a search result. The method includes processing devices performing operations including receiving a query having a text input by a joint embedding model trained to generate an image result. Training the joint embedding model includes accessing a set of images and textual information. Training further includes encoding the images into image feature vectors based on spatial features. Further, training includes encoding the textual information into textual feature vectors based on semantic information. Training further includes generating a set of image-text pairs based on matches between image feature vectors and textual feature vectors. Further, training includes generating a visual grounding dataset based on spatial information. Training further includes generating a set of visual-semantic joint embeddings by grounding the image-text pairs with the visual grounding dataset. Additionally, operations include generating an image result for display by the joint embedding model based on the text input.

Joint visual-semantic embedding and grounding via multi-task training for image searching

In some embodiments, a data visualization system detects insights from a dataset and computes insight scores for respective insights. The data visualization system further computes insight type scores, from the insight scores, for insight types in the detected insights. The data visualization system determines a selected insight type for the dataset having a higher insight type score than unselected insight types and determines, for the selected insight type, a set of selected insights that have higher insight scores than unselected insights. The data visualization system determines insight visualizations for the set of selected insights and generates, for inclusion in a user interface of the data visualization system, selectable interface elements configured for invoking an editing tool for updating the determined insight visualizations from the dataset. The selectable interface elements are arranged in the user interface according to the insight scores of the set of selected insights.

Configuration of user interface for intuitive selection of insight visualizations

A multimodal recommendation identification system analyzes data describing a sequence of past content item interactions to generate a recommendation for a content item for a user. An indication of the recommended content item is provided to a website hosting system or recommendation system so that the recommended content item is displayed or otherwise presented to the user. The multimodal recommendation identification system identifies a content item to recommend to the user by generating an encoding that encodes identifiers of the sequence of content items the user has interacted with and generating encodings that encode multimodal information for content items in the sequence of content items the user has interacted with. An aggregated information encoding for a user based on these encodings and a system analyzes the content item sequence encoding and interaction between the content item sequence encoding and the multiple modality encodings to generate the aggregated information encoding.

Multimodal sequential recommendation with window co-attention

Growing community of inventors

San Jose, CA, United States of America

Handong Zhao