Adobe, Inc.

University of California

Seiko Epson Corporation

Czech Technical University in Prague

Zhaowen Wang

Hailin Jin

Chen Fang

Zhe Lin

Jimei Yang

Zhifei Zhang

Ning Xu

Yilin Wang

Yang Liu

Luoqi Liu

Matthew David Fisher

Trung Huu Bui

Xin Lu

Handong Zhao

Nirmal Kumawat

I-Ming Pao

Tianlang Chen

Wangcheng Kang

Julian McAuley

Yulun Zhang

Ajinkya Gorakhnath Kale

Gavin Stuart Peter Miller

Jose Ignacio Echevarria Vallespi

Yijun Li

Alan Lee Erickson

Chih-Yao Hsieh

Zhankui He

Elya Shechtman

Xiaohui Shen

Jianchao Yang

Jonathan Worthen Brandt

Zhili Chen

Joon-Young Lee

John Philip Collomosse

Quanzeng You

Rundong Wu

Yipin Zhou

Brian Lynn Price

Jing Xiao

Nathan Aaron Carr

Kalyan Krishna Sunkavalli

Franck Dernoncourt

Baldo Antonio Faieta

Paul John Asente

Aaron Phillip Hertzmann

Jinjun Wang

Sarah Kong

Vladimir Kim

Saeid Motiian

Niloy Jyoti Mitra

Yuyan Song

Alex Charles Filipkowski

Peter Evan O'Donovan

Stephen Diverdi

Duygu Ceylan

Daniel Sýkora

Michal Lukác

Sheng Li

Mai Long

Hao Hu

Seth Shaw

Bayram Safa Cicek

Kan Chen

Samaneh Azadi

Xuaner Zhang

Spyridon Ampanavos

Xingqian Xu

Chinthala Pradyumna Reddy

Jielin Qiu

Zhiqiang Tao

Yue Bai

David Futschik

Shuai Tang

Zegi Gu

Shuhui Jiang

Xuan Li

Haotian Zhang

Xianming Liu

I-Chao Shen

Systems and methods for video segmentation and summarization are described. Embodiments of the present disclosure receive a video and a transcript of the video; generate visual features representing frames of the video using an image encoder; generate language features representing the transcript using a text encoder, wherein the image encoder and the text encoder are trained based on a correlation between training visual features and training language features; and segment the video into a plurality of video segments based on the visual features and the language features.

Multimodal unsupervised video temporal segmentation for summarization

Digital content search techniques are described that overcome the challenges found in conventional sequence-based techniques through use of a query-aware sequential search. In one example, a search query is received and sequence input data is obtained based on the search query. The sequence input data describes a sequence of digital content and respective search queries. Embedding data is generated based on the sequence input data using an embedding module of a machine-learning model. The embedding module includes a query-aware embedding layer that generates embeddings of the sequence of digital content and respective search queries. A search result is generated referencing at least one item of digital content by processing the embedding data using at least one layer of the machine-learning model.

Digital content query-aware sequential search

Digital content layout encoding techniques for search are described. In these techniques, a layout representation is generated (using machine learning automatically and without user intervention) that describes a layout of elements included within the digital content. In an implementation, the layout representation includes a description of both spatial and structural aspects of the elements in relation to each other. To do so, a two-pathway pipeline that is configured to model layout from both spatial and structural aspects using a spatial pathway, and a structural pathway, respectively. In one example, this is also performed through use of multi-level encoding and fusion to generate a layout representation.

Digital content layout encoding for search

Automatic font synthesis for modifying a local font to have an appearance that is visually similar to a source font is described. A font modification system receives an electronic document including the source font together with an indication of a font descriptor for the source font. The font descriptor includes information describing various font attributes for the source font, which define a visual appearance of the source font. Using the source font descriptor, the font modification system identifies a local font that is visually similar in appearance to the source font by comparing local font descriptors to the source font descriptor. A visually similar font is then synthesized by modifying glyph outlines of the local font to achieve the visual appearance defined by the source font descriptor. The synthesized font is then used to replace the source font and output in the electronic document at the computing device.

Preserving document design using font synthesis

Embodiments are disclosed for translating an image from a source visual domain to a target visual domain. In particular, in one or more embodiments, the disclosed systems and methods comprise a training process that includes receiving a training input including a pair of keyframes and an unpaired image. The pair of keyframes represent a visual translation from a first version of an image in a source visual domain to a second version of the image in a target visual domain. The one or more embodiments further include sending the pair of keyframes and the unpaired image to an image translation network to generate a first training image and a second training image. The one or more embodiments further include training the image translation network to translate images from the source visual domain to the target visual domain based on a calculated loss using the first and second training images.

Neural network for image style translation

Digital content search techniques are described. In one example, the techniques are incorporated as part of a multi-head self-attention module of a transformer using machine learning. A localized self-attention module, for instance, is incorporated as part of the multi-head self-attention module that applies local constraints to the sequence. This is performable in a variety of ways. In a first instance, a model-based local encoder is used, examples of which include a fixed-depth recurrent neural network (RNN) and a convolutional network. In a second instance, a masking-based local encoder is used, examples of which include use of a fixed window, Gaussian initialization, and an adaptive predictor.

Locally constrained self-attentive sequential recommendation

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating scalable and semantically editable font representations utilizing a machine learning approach. For example, the disclosed systems generate a font representation code from a glyph utilizing a particular neural network architecture. For example, the disclosed systems utilize a glyph appearance propagation model and perform an iterative process to generate a font representation code from an initial glyph. Additionally, using a glyph appearance propagation model, the disclosed systems automatically propagate the appearance of the initial glyph from the font representation code to generate additional glyphs corresponding to respective glyph labels. In some embodiments, the disclosed systems propagate edits or other changes in appearance of a glyph to other glyphs within a glyph set (e.g., to match the appearance of the edited glyph).

Generating scalable and semantically editable font representations

Embodiments of the technology described herein, are an intelligent system that aims to expedite a text design process by providing text design predictions interactively. The system works with a typical text design scenario comprising a background image and one or more text strings as input. In the design scenario, the text string is to be placed on top of the background. The textual design agent may include a location recommendation model that recommends a location on the background image to place the text. The textual design agent may also include a font recommendation model, a size recommendation model, and a color recommendation model. The output of these four models may be combined to generate draft designs that are evaluated as a whole (combination of color, font, and size) for the best designs. The top designs may be output to the user.

Textual design agent

The present disclosure relates to systems, methods, and non-transitory computer-readable media for accurately and flexibly generating scalable fonts utilizing multi-implicit neural font representations. For instance, the disclosed systems combine deep learning with differentiable rasterization to generate a multi-implicit neural font representation of a glyph. For example, the disclosed systems utilize an implicit differentiable font neural network to determine a font style code for an input glyph as well as distance values for locations of the glyph to be rendered based on a glyph label and the font style code. Further, the disclosed systems rasterize the distance values utilizing a differentiable rasterization model and combines the rasterized distance values to generate a permutation-invariant version of the glyph corresponding glyph set.

Generating scalable fonts utilizing multi-implicit neural font representations

The present disclosure relates to a fashion recommendation system that employs a task-guided learning framework to jointly train a visually-aware personalized preference ranking network. In addition, the fashion recommendation system employs implicit feedback and generated user-based triplets to learn variances in the user's fashion preferences for items with which the user has not yet interacted. In particular, the fashion recommendation system uses triplets generated from implicit user data to jointly train a Siamese convolutional neural network and a personalized ranking model, which together produce a user preference predictor that determines personalized fashion recommendations for a user.

Generating a personalized preference ranking network for providing visually-aware item recommendations

In implementations of recognizing text in images, text recognition systems are trained using noisy images that have nuisance factors applied, and corresponding clean images (e.g., without nuisance factors). Clean images serve as supervision at both feature and pixel levels, so that text recognition systems are trained to be feature invariant (e.g., by requiring features extracted from a noisy image to match features extracted from a clean image), and feature complete (e.g., by requiring that features extracted from a noisy image be sufficient to generate a clean image). Accordingly, text recognition systems generalize to text not included in training images, and are robust to nuisance factors. Furthermore, since clean images are provided as supervision at feature and pixel levels, training requires fewer training images than text recognition systems that are not trained with a supervisory clean image, thus saving time and resources.

Training text recognition systems

Embodiments of the present disclosure are directed towards improved models trained using unsupervised domain adaptation. In particular, a style-content adaptation system provides improved translation during unsupervised domain adaptation by controlling the alignment of conditional distributions of a model during training such that content (e.g., a class) from a target domain is correctly mapped to content (e.g., the same class) in a source domain. The style-content adaptation system improves unsupervised domain adaptation using independent control over content (e.g., related to a class) as well as style (e.g., related to a domain) to control alignment when translating between the source and target domain. This independent control over content and style can also allow for images to be generated using the style-content adaptation system that contain desired content and/or style.

Controlled style-content image generation based on disentangling content and style

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that extract a texture from embedded text within a digital image utilizing kerning-adjusted glyphs. For example, the disclosed systems utilize text recognition and text segmentation to identify and segment glyphs from embedded text depicted in a digital image. Subsequently, in some implementations, the disclosed systems determine optimistic kerning values between consecutive glyphs and utilize the kerning values to reduce gaps between the consecutive glyphs. Furthermore, in one or more implementations, the disclosed systems generate a synthesized texture utilizing the kerning-value-adjusted glyphs by utilizing image inpainting on the textures corresponding to the kerning-value-adjusted glyphs. Moreover, in certain instances, the disclosed systems apply a target texture to a target digital text based on the generated synthesized texture.

Extracting textures from text based images

A multimodal recommendation identification system analyzes data describing a sequence of past content item interactions to generate a recommendation for a content item for a user. An indication of the recommended content item is provided to a website hosting system or recommendation system so that the recommended content item is displayed or otherwise presented to the user. The multimodal recommendation identification system identifies a content item to recommend to the user by generating an encoding that encodes identifiers of the sequence of content items the user has interacted with and generating encodings that encode multimodal information for content items in the sequence of content items the user has interacted with. An aggregated information encoding for a user based on these encodings and a system analyzes the content item sequence encoding and interaction between the content item sequence encoding and the multiple modality encodings to generate the aggregated information encoding.

Multimodal sequential recommendation with window co-attention

Growing community of inventors

San Jose, CA, United States of America

Zhaowen Wang