Adobe, Inc.

Pranav Vineet Aggarwal

Ajinkya Gorakhnath Kale

Baldo Antonio Faieta

Zhe Lin

Saeid Motiian

Midhun Harikumar

Ali Aminian

Ritiz Tambi

Aashish Misraa

Venkata Naveen Kumar Yadav Marri

Shabnam Ghadar

Tracy Holloway King

Alex Charles Filipkowski

Peter Evan O'Donovan

Alvin Ghouas

Sukriti Verma

Daniel ReMine

Di Pu

Naveen Marri

A method and system for color optimization in generated images are described. The method and system include receiving an image generation prompt that includes a text description of target image content and color information describing a target color palette; encoding the image generation prompt to obtain image features that represent the target image content and the target color palette; and generating an image representing the target image content with the target color palette based on the image features.

Systems and methods for color palette optimization

Techniques for generating a novel image using tokenized image representations are disclosed. In some embodiments, a method of generating the novel image includes generating, via a first machine learning model, a first sequence of coded representations of a first image having one or more features; generating, via a second machine learning model, a second sequence of coded representations of a sketch image having one or more edge features associated with the one or more features; predicting, via a third machine learning model, one or more subsequent coded representations based on the first sequence of coded representations and the second sequence of coded representations; and based on the subsequent coded representations, generating, via the third machine learning model, a first portion of a reconstructed image having one or more image attributes of the first image, and a second portion of the reconstructed image associated with the one or more edge features.

Generating novel images using sketch image representations

Technology is disclosed herein for enhanced similarity search. In an implementation, a search environment includes one or more computing hardware, software, and/or firmware components in support of enhanced similarity search. The one or more components identify a modality for a similarity search with respect to a query object. The components generate an embedding for the query object based on the modality and based on connections between the query object and neighboring nodes in a graph. The embedding for the query object provides the basis for the search for similar objects.

Unified framework for multi-modal similarity search

Systems and methods for image processing are configured. Embodiments of the present disclosure identify target style attributes and target structure attributes for a composite image; generate a matrix of composite feature tokens based on the target style attributes and the target structure attributes, wherein subsequent feature tokens of the matrix of composite feature tokens are sequentially generated based on previous feature tokens of the matrix of composite feature tokens according to a linear ordering of the matrix of composite feature tokens; and generate the composite image based on the matrix of composite feature tokens, wherein the composite image includes the target style attributes and the target structure attributes.

Unsupervised style and color cues for transformer-based image generation

Embodiments are disclosed for machine learning-based generation of recommended layouts. The method includes receiving a set of design elements for performing generative layout recommendation. A number of each type of design element from the set of design elements is determined. A set of recommended layouts are generated using a trained generative layout model and the number and type of design elements. The set of recommended layouts are output.

Machine learning-based layout generation

Systems and methods for image processing are described. Embodiments of the present disclosure receive a first image depicting a scene and a second image that includes a style; segment the first image to obtain a first segment and a second segment, wherein the first segment has a shape of an object in the scene; apply a style transfer network to the first segment and the second image to obtain a first image part, wherein the first image part has the shape of the object and the style from the second image; combine the first image part with a second image part corresponding to the second segment to obtain a combined image; and apply a lenticular effect to the combined image to obtain an output image.

Stylized motion effects

Systems and methods for image retrieval are described. Embodiments of the present disclosure receive a search query from a user; extract an entity and a color phrase describing the entity from the search query; generate an entity color embedding in a color embedding space from the color phrase using a multi-modal color encoder; identify an image in a database based on metadata for the image including an object label corresponding to the extracted entity and an object color embedding in the color embedding space corresponding to the object label; and provide image information for the image to the user based on the metadata.

Embedding-based color-object retrieval

Visually guided machine-learning language model and embedding techniques are described that overcome the challenges of conventional techniques in a variety of ways. In one example, a model is trained to support a visually guided machine-learning embedding space that supports visual intuition as to 'what' is represented by text. The visually guided language embedding space supported by the model, once trained, may then be used to support visual intuition as part of a variety of functionality. In one such example, the visually guided language embedding space as implemented by the model may be leveraged as part of a multi-modal differential search to support search of digital images and other digital content with real-time focus adaptation which overcomes the challenges of conventional techniques.

Visually guided machine-learning language model

A non-transitory computer-readable medium includes program code that is stored thereon. The program code is executable by one or more processing devices for performing operations including generating, using a model, a learned image representation of a target image. The operations further include generating, using a text embedding model, a text embedding of a text query. The text embedding and the learned image representation of the target image are in a same embedding space. Additionally, the operations include convolving the learned image representation of the target image with the text embedding of the text query. Moreover, the operations include generating an object-segmented image based on the convolving of the learned image representation of the target image with the text embedding.

Image segmentation using text embedding

The present disclosure describes systems and methods for information retrieval. Embodiments of the disclosure provide a color embedding network trained using machine learning techniques to generate embedded color representations for color terms included in a text search query. For example, techniques described herein are used to represent color text in a same space as color embeddings (e.g., an embedding space created by determining a histogram of LAB based colors in a three-dimensional (3D) space). Further, techniques are described for indexing color palettes for all the searchable images in the search space. Accordingly, color terms in a text query are directly converted into a color palette and an image search system can return one or more search images with corresponding color palettes that are relevant to (e.g., within a threshold distance from) the color palette of the text query.

Text to color palette generator

Text-to-visual machine learning embedding techniques are described that overcome the challenges of conventional techniques in a variety of ways. These techniques include use of query-based training data which may expand availability and types of training data usable to train a model. Generation of negative digital image samples is also described that may increase accuracy in training the model using machine learning. A loss function is also described that also supports increased accuracy and computational efficiency by losses separately, e.g., between positive or negative sample embeddings a text embedding.

Text-to-visual machine learning embedding techniques

Systems and methods for color replacement are described. Embodiments of the disclosure include a color replacement system that adjusts an image based on a user-input source color and target color. For example, the source color may be replaced with the target color throughout the entire image. In some embodiments, a user provides a speech or text input that identifies a source color to be replaced. The user may then provide a speech or text input identifying the target color, replacing the source color. A color replacement system creates and embedding of the source color, segments the image based on the source color embedding, and then replaces the color of segmented portion of the image with the target color.

Multi-modal image color segmenter and editor

The present disclosure relates to methods, systems, and non-transitory computer-readable media for retrieving digital images in response to queries. For example, in one or more embodiments, the disclosed systems receive a query comprising text and generates a cross-lingual-multimodal embedding for the text within a multimodal embedding space. The disclosed systems further identifies an image embedding for a digital image that corresponds to (e.g., is relevant to) the text from the query based on an embedding distance between the image embedding and the cross-lingual-multimodal embedding for the text within the multimodal embedding space. Accordingly, the disclosed systems retrieve the digital image associated with the image embedding for display on a client device, such as the client device that submitted the query.

Generating embeddings in a multimodal embedding space for cross-lingual digital image retrieval

Disclosed are computer-implemented methods and systems for generating text descriptive of digital images, comprising using a machine learning model to pre-process an image to generate initial text descriptive of the image; adjusting one or more inferences of the machine learning model, the inferences biasing the machine learning model away from associating negative words with the image; using the machine learning model comprising the adjusted inferences to post-process the image to generate updated text descriptive of the image; and processing the generated updated text descriptive of the image outputted by the machine learning model to fine-tune the updated text descriptive of the image.

Systems and methods for generating text descriptive of digital images

Introduced here is an approach to translating tags assigned to digital images. As an example, embeddings may be extracted from a tag to be translated and the digital image with which the tag is associated by a multimodal model. These embeddings can be compared to embeddings extracted from a set of target tags associated with a target language by the multimodal model. Such an approach allows similarity to be established along two dimensions, which ensures the obstacles associated with direct translation can be avoided.

Growing community of inventors

San Jose, CA, United States of America

Pranav Vineet Aggarwal