Adobe, Inc.

Midhun Harikumar

Ajinkya Gorakhnath Kale

Pranav Vineet Aggarwal

Zhe Lin

Baldo Antonio Faieta

Saeid Motiian

Pramod Srinivasan

Alex Charles Filipkowski

Daniel Miranda

Venkata Naveen Kumar Yadav Marri

Jianming Zhang

Shabnam Ghadar

Jayant Kumar

Taesung Park

Peter Evan O'Donovan

Samarth Gulati

Ritiz Tambi

Kannan Iyer

Rohith Mohan Dodle

Vipul Dalal

Vinh Ngoc Khuc

Gaurav Kukal

Mina Doroudi

Archit Kalra

Vera Lychagina

Charles R McKinney

Prethebha Chandrasegaran

Sukriti Verma

Hareesh Ravi

Systems and methods for image processing, and specifically for generating object-agnostic image representations, are described. Embodiments of the present disclosure receive a training image including a foreground object and a background, remove the foreground object from the training image to obtain a modified training image, inpaint a portion of the modified training image corresponding to the foreground object to obtain an inpainted training image, encode the training image and the inpainted training image using a machine learning model to obtain an encoded training image and an encoded inpainted training image, and update parameters of the machine learning model based on the encoded training image and the encoded inpainted training image.

Object-agnostic image representation

Systems and methods for image processing are described. Embodiments of the present disclosure obtain a text prompt for text guided image generation. A multi-modal encoder of an image processing apparatus encodes the text prompt to obtain a text embedding. A diffusion prior model of the image processing apparatus converts the text embedding to an image embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, wherein the image includes an element described by the text prompt.

Prior guided latent diffusion

Systems and methods for image processing are described. Embodiments of the present disclosure include an image generation network configured to encode a plurality of abstract images using a style encoder to obtain a plurality of abstract style encodings, wherein the style encoder is trained to represent image style separately from image content. A clustering component clusters the plurality of abstract style encodings to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings. A preset component generates an abstract style transfer preset representing the abstract style cluster.

Preset style transfer

A method and system for color optimization in generated images are described. The method and system include receiving an image generation prompt that includes a text description of target image content and color information describing a target color palette; encoding the image generation prompt to obtain image features that represent the target image content and the target color palette; and generating an image representing the target image content with the target color palette based on the image features.

Systems and methods for color palette optimization

Techniques for generating a novel image using tokenized image representations are disclosed. In some embodiments, a method of generating the novel image includes generating, via a first machine learning model, a first sequence of coded representations of a first image having one or more features; generating, via a second machine learning model, a second sequence of coded representations of a sketch image having one or more edge features associated with the one or more features; predicting, via a third machine learning model, one or more subsequent coded representations based on the first sequence of coded representations and the second sequence of coded representations; and based on the subsequent coded representations, generating, via the third machine learning model, a first portion of a reconstructed image having one or more image attributes of the first image, and a second portion of the reconstructed image associated with the one or more edge features.

Generating novel images using sketch image representations

Systems and methods for image processing are configured. Embodiments of the present disclosure identify target style attributes and target structure attributes for a composite image; generate a matrix of composite feature tokens based on the target style attributes and the target structure attributes, wherein subsequent feature tokens of the matrix of composite feature tokens are sequentially generated based on previous feature tokens of the matrix of composite feature tokens according to a linear ordering of the matrix of composite feature tokens; and generate the composite image based on the matrix of composite feature tokens, wherein the composite image includes the target style attributes and the target structure attributes.

Unsupervised style and color cues for transformer-based image generation

Embodiments are disclosed for machine learning-based generation of recommended layouts. The method includes receiving a set of design elements for performing generative layout recommendation. A number of each type of design element from the set of design elements is determined. A set of recommended layouts are generated using a trained generative layout model and the number and type of design elements. The set of recommended layouts are output.

Machine learning-based layout generation

A non-transitory computer-readable medium includes program code that is stored thereon. The program code is executable by one or more processing devices for performing operations including generating, using a model, a learned image representation of a target image. The operations further include generating, using a text embedding model, a text embedding of a text query. The text embedding and the learned image representation of the target image are in a same embedding space. Additionally, the operations include convolving the learned image representation of the target image with the text embedding of the text query. Moreover, the operations include generating an object-segmented image based on the convolving of the learned image representation of the target image with the text embedding.

Image segmentation using text embedding

Keyword localization digital image search techniques are described. These techniques support an ability to indicate 'where' a corresponding keyword is to be expressed with respect to a layout in a respective digital image resulting from a search query. The search query may also include an indication of a size of the keyword as expressed in the digital image, a number of instances of the keyword, and so forth. Additionally, the techniques and systems as described herein support real time search through use of keyword signatures.

Keyword localization digital image search

A non-transitory computer-readable medium includes program code that is stored thereon. The program code is executable by one or more processing devices for performing operations including generating, by a model that includes trainable components, a learned image representation of a target image. The operations further include generating, by a text embedding model, a text embedding of a text query. The text embedding and the learned image representation of the target image are in a same embedding space. Additionally, the operations include generating a class activation map of the target image by, at least, convolving the learned image representation of the target image with the text embedding of the text query. Moreover, the operations include generating an object-segmented image using the class activation map of the target image.

The present disclosure relates to an image merging system that automatically and seamlessly detects and merges missing people for a set of digital images into a composite group photo. For instance, the image merging system utilizes a number of models and operations to automatically analyze multiple digital images to identify a missing person from a base image, segment the missing person from the second image, and generate a composite group photo by merging the segmented image of the missing person into the base image. In this manner, the image merging system automatically creates merged group photos that appear natural and realistic.

Automatically merging people and objects from multiple digital images to generate a composite digital image

Object search techniques for digital images are described. In the techniques described herein, semantic features are extracted on a per-object basis form a digital image. This supports location of objects within digital images and is not limited to semantic features of an entirety of the digital image as involved in conventional image similarity search techniques. This may be combined with indications a location of the object globally with respect to the digital image through use of a global segmentation mask, use of a local segmentation mask to capture post and characteristics of the object itself, and so on.

Growing community of inventors

Santa Clara, CA, United States of America

Midhun Harikumar