Adobe Systems Inc.

Adobe, Inc.

Presenter.com, Inc.

Silicon Graphics, Incorporated

Web Ex Communications, Inc.

Jonathan Worthen Brandt

Zhe Lin

Xiaohui Shen

Hailin Jin

Jianchao Yang

Scott David Cohen

Jianming Zhang

Haoxiang Li

Nathan Aaron Carr

Chen Fang

Radomir Mech

Gavin Stuart Peter Miller

Zeke Koch

Walter Wei-Tuh Chang

Elya Shechtman

Lubomir D Bourdev

Zhihong Ding

Vuong Le

Walter Chang

Aseem Agarwala

Shenchang Eric Chen

Guang Chen

Zhaowen Wang

Franck Dernoncourt

Linjie Luo

Xin Lu

Byungmoon Kim

Wilmot Wei-Mau Li

Michael Kaplan

Michael A Marth

Mai Long

Stefan Guggisberg

Jae-Pil Heo

Xiang Yu

Biao Jia

Brian Lynn Price

Trung Huu Bui

Ning Xu

Nedim Lipka

Doo Soon Kim

Mark S Peercy

Sven Duwenhorst

Zhifei Zhang

Nico Becherer

Krishna Kumar Singh

Feng Tang

John Milligan Airey

Varun Manjunatha

Qing Liu

Khoi Pham

Chenxi Zhang

Timothy Jeewun Ganter

Sven Winter

Brandon M Smith

Sangwoo Cho

Klaas Stoeckmann

Thomas Muehrke

Matthew Joss

Celso Gomes

Michael Zimmermann

Olena Soroka

Darshan Prasad

Zhangyang Wang

Thomas Le Paine

Hui Zhang

Whei-Tsu Helen Tahn

Daniil Pakhomov

Mariette Souppe

Ohiremen Dibua

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via multi-layered scene completion techniques facilitated by artificial intelligence. For instance, in some embodiments, the disclosed systems receive a digital image portraying a first object and a second object against a background, where the first object occludes a portion of the second object. Additionally, the disclosed systems pre-process the digital image to generate a first content fill for the portion of the second object occluded by the first object and a second content fill for a portion of the background occluded by the second object. After pre-processing, the disclosed systems detect one or more user interactions to move or delete the first object from the digital image. The disclosed systems further modify the digital image by moving or deleting the first object and exposing the first content fill for the portion of the second object.

Modifying digital images via multi-layered scene completion facilitated by artificial intelligence

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems generate utilizing a segmentation neural network, an object mask for each object of a plurality of objects of a digital image. The disclosed systems detect a first user interaction with an object in the digital image displayed via a graphical user interface. The disclosed systems surface, via the graphical user interface, the object mask for the object in response to the first user interaction. The disclosed systems perform an object-aware modification of the digital image in response to a second user interaction with the object mask for the object.

Modifying digital images via scene-based editing using image understanding facilitated by artificial intelligence

System and methods for a text summarization system are described. In one example, a text summarization system receives an input utterance and determines whether the utterance should be included in a summary of the text. The text summarization system includes an embedding network, a convolution network, an encoding component, and a summary component. The embedding network generates a semantic embedding of an utterance. The convolution network generates a plurality of feature vectors based on the semantic embedding. The encoding component identifies a plurality of latent codes respectively corresponding to the plurality of feature vectors. The summary component identifies a prominent code among the latent codes and to select the utterance as a summary utterance based on the prominent code.

Topical vector-quantized variational autoencoders for extractive summarization of video transcripts

Approaches are described for determining facial landmarks in images. An input image is provided to at least one trained neural network that determines a face region (e.g., bounding box of a face) of the input image and initial facial landmark locations corresponding to the face region. The initial facial landmark locations are provided to a 3D face mapper that maps the initial facial landmark locations to a 3D face model. A set of facial landmark locations are determined from the 3D face model. The set of facial landmark locations are provided to a landmark location adjuster that adjusts positions of the set of facial landmark locations based on the input image. The input image is presented on a user device using the adjusted set of facial landmark locations.

Neural network based face detection and landmark localization

Embodiments of the present invention provide an automated image tagging system that can predict a set of tags, along with relevance scores, that can be used for keyword-based image retrieval, image tag proposal, and image tag auto-completion based on user input. Initially, during training, a clustering technique is utilized to reduce cluster imbalance in the data that is input into a convolutional neural network (CNN) for training feature data. In embodiments, the clustering technique can also be utilized to compute data point similarity that can be utilized for tag propagation (to tag untagged images). During testing, a diversity based voting framework is utilized to overcome user tagging biases. In some embodiments, bigram re-weighting can down-weight a keyword that is likely to be part of a bigram based on a predicted tag set.

Accurate tag relevance prediction for image search

Some embodiments involve a reinforcement learning based framework for training a natural media agent to learn a rendering policy without human supervision or labeled datasets. The reinforcement learning based framework feeds the natural media agent a training dataset to implicitly learn the rendering policy by exploring a canvas and minimizing a loss function. Once trained, the natural media agent can be applied to any reference image to generate a series (or sequence) of continuous-valued primitive graphic actions, e.g., sequence of painting strokes, that when rendered by a synthetic rendering environment on a canvas, reproduce an identical or transformed version of the reference image subject to limitations of an action space and the learned rendering policy.

Reinforcement learning-based techniques for training a natural media agent

Embodiments of the present invention provide systems, methods, and non-transitory computer storage media for parsing a given input referring expression into a parse structure and generating a semantic computation graph to identify semantic relationships among and between objects. At a high level, when embodiments of the preset invention receive a referring expression, a parse tree is created and mapped into a hierarchical subject, predicate, object graph structure that labeled noun objects in the referring expression, the attributes of the labeled noun objects, and predicate relationships (e.g., verb actions or spatial propositions) between the labeled objects. Embodiments of the present invention then transform the subject, predicate, object graph structure into a semantic computation graph that may be recursively traversed and interpreted to determine how noun objects, their attributes and modifiers, and interrelationships are provided to downstream image editing, searching, or caption indexing tasks.

Methods and systems for generating a semantic computation graph for understanding and grounding referring expressions

The present disclosure is directed toward systems and methods for tracking objects in videos. For example, one or more embodiments described herein utilize various tracking methods in combination with an image search index made up of still video frames indexed from a video. One or more embodiments described herein utilize a backward and forward tracking method that is anchored by one or more key frames in order to accurately track an object through the frames of a video, even when the video is long and may include challenging conditions.

Robust tracking of objects in videos

Techniques of directing a user to content based on a semantic interpretation of a query input by the user involves generating links to specific content in a collection of documents in response to user string query, the links being generated based on an answer suggestion lookahead index. The answer suggestion lookahead index references a mapping between a plurality of groups of semantically equivalent terms and a respective link to specific content of the collection of documents. These techniques are useful for the generalized task of natural language question answering.

Conversational query answering system

A non-photorealistic image rendering system and related techniques are described herein that train and implement machine learning models to reproduce digital images in accordance with various painting styles and constraints. The image rendering system can include a machine learning system that utilizes actor-critic based reinforcement learning techniques to train painting agents (e.g., models that include one or more neural networks) how to transform images into various artistic styles with minimal loss between the original images and the transformed images. The image rendering system can generate constrained painting agents, which correspond to painting agents that are further trained to reproduce images in accordance with one or more constraints. The constraints may include limitations of the color, width, size, and/or position of brushstrokes within reproduced images. These constrained painting agents may provide users with robust, flexible, and customizable non-photorealistic painting systems.

Learned model-based image rendering

The present disclosure includes methods and systems for searching for digital visual media based on semantic and spatial information. In particular, one or more embodiments of the disclosed systems and methods identify digital visual media displaying targeted visual content in a targeted region based on a query term and a query area provide via a digital canvas. Specifically, the disclosed systems and methods can receive user input of a query term and a query area and provide the query term and query area to a query neural network to generate a query feature set. Moreover, the disclosed systems and methods can compare the query feature set to digital visual media feature sets. Further, based on the comparison, the disclosed systems and methods can identify digital visual media portraying targeted visual content corresponding to the query term within a targeted region corresponding to the query area.

Utilizing a digital canvas to conduct a spatial-semantic search for digital visual media

Image search techniques and systems involving emotions are described. In one or more implementations, a digital medium environment of a content sharing service is described for image search result configuration and control based on a search request that indicates an emotion. The search request is received that includes one or more keywords and specifies an emotion. Images are located that are available for licensing by matching one or more tags associated with the image with the one or more keywords and as corresponding to the emotion. The emotion of the images is identified using one or more models that are trained using machine learning based at least in part on training images having tagged emotions. Output is controlled of a search result having one or more representations of the images that are selectable to license respective images from the content sharing service.

Image search using emotions

Techniques are disclosed for image feature representation. The techniques exhibit discriminative power that can be used in any number of classification tasks, and are particularly effective with respect to fine-grained image classification tasks. In an embodiment, a given image to be classified is divided into image patches. A vector is generated for each image patch. Each image patch vector is compared to the Gaussian mixture components (each mixture component is also a vector) of a Gaussian Mixture Model (GMM). Each such comparison generates a similarity score for each image patch vector. For each Gaussian mixture component, the image patch vectors associated with a similarity score that is too low are eliminated. The selectively pooled vectors from all the Gaussian mixture components are then concatenated to form the final image feature vector, which can be provided to a classifier so the given input image can be properly categorized.

Local feature representation for image recognition

Methods and systems for recognizing people in images with increased accuracy are disclosed. In particular, the methods and systems divide images into a plurality of clusters based on common characteristics of the images. The methods and systems also determine an image cluster to which an image with an unknown person instance most corresponds. One or more embodiments determine a probability that the unknown person instance is each known person instance in the image cluster using a trained cluster classifier of the image cluster. Optionally, the methods and systems determine context weights for each combination of an unknown person instance and each known person instance using a conditional random field algorithm based on a plurality of context cues associated with the unknown person instance and the known person instances. The methods and systems calculate a contextual probability based on the cluster-based probabilities and context weights to identify the unknown person instance.

Identifying unknown person instances in images

Growing community of inventors

Santa Cruz, CA, United States of America

Jonathan Worthen Brandt