Lemon Inc.

Bingchen Liu

Xiao Yang

Yizhe Zhu

Kin Chung Wong

Qing Yan

Lingyiqing Zhou

Blake Garrett Fuselier

Diana Lee

Peilin Li

Lexin Tang

The present disclosure describes techniques for generating content. Text may be received. The text is associated with a video to be created by at least one user. At least one image may be generated based at least in part on the text using at least one machine learning model. The video may be generated based at least in part on the at least one image. The video comprises content overlaid on the at least one image.

Content creation based on text-to-image generation

The present disclosure provides systems and methods for generating a synthesized image of a user with a trained machine learning diffusion model. In one example, a computing system includes one or more processors configured to execute instructions stored in memory to execute a trained machine learning diffusion model including an image encoder, a text encoder, and a diffusion model. The image encoder is configured to receive an image of a user and generate a set of embeddings that semantically describe visual features of the user based at least on the image of the user. The text encoder is configured to receive the set of embeddings and generate an input feature vector based at least on the set of embeddings. The diffusion model is configured to receive the input feature vector and generate a synthesized image of the user based at least on the input feature vector.

Machine learning diffusion model with image encoder trained for synthetic image generation

The present disclosure provides systems and methods for video generation corresponding to a user input. Given a user input, a story video with content relevant to the user input can be generated. One aspect includes a computing system comprising a processor and memory. The processor can be configured to execute a program using portions of the memory to receive the user input, generate a story text based on the user input, generate a plurality of story images based on the story text, and output a story including the story text and a story video having content corresponding to the story text, wherein the story video includes the plurality of story images. Additionally or alternatively, the story video can include audio data and a plurality of generated animated videos, each animated video corresponding to a story image in the plurality of story images.

Generation of story videos corresponding to user input using generative models

Systems and methods are provided that include a processor executing a program to receive input text from a user. The processor is further configured to, for a predetermined number of iterations, input an initial image into a diffusion process to generate a processed image, back-propagate the processed image through a text-image match gradient calculator to calculate a gradient against the input text, and update the initial image with an image generated by applying the calculated gradient to the processed image. The pixel values of the processed image during a first portion of the predetermined number of iterations are value clamped to a first range, and pixel values of the processed image during a second portion of the predetermined number of iterations are value clamped to a second range that is a subset of the first range.

Generation of image corresponding to input text using dynamic value clipping

Systems and methods are provided that include a processor executing a program to receive an input from a user, where the input including a first input text and a second input text. The processor is further configured to provide an initial image and, for a predetermined number of iterations, define a first and second regions of the initial image associated with the first and second input texts, respectively, define a plurality of patches of the initial image, input the initial image into a diffusion process to generate a processed image, back-propagate the processed image through a text-image match gradient calculator by generating an image embedding based on the processed image, generating a text embedding based on the region and the input text that are associated with a patch, and calculating a differential between the image embedding and the text embedding.

Generation of image corresponding to input text using multi-text guided image cropping

Systems and methods are provided that include a processor executing a program to process an initial image through a first diffusion stage to generate a final first stage image, wherein the first diffusion stage includes using a diffusion model, a gradient estimator model smaller than the diffusion model, and a text-image match gradient calculator. The processor further executes the program to process the final first stage image through a second diffusion stage to generate a final second stage image. The second diffusion stage includes, for a second predetermined number of iterations, inputting the final first stage image to through the diffusion model, back-propagate the image through the text-image match gradient calculator to calculate a second stage gradient against the input text, and update the final first stage image by applying the second stage gradient to the final first stage image.

Generation of images corresponding to input text using multi-algorithm diffusion sampling

The present disclosure describes techniques for generating visual effects based on text. The techniques comprise initiating a process of generating a visual effect in response to a selection of an interface element; generating the visual effect, wherein the generating the visual effect comprises generating a first image based on a first text input by a first user, and wherein content of the first image correlates with the first text; and causing to display the first image on a screen of a computing device.

Growing community of inventors

Los Angeles, CA, United States of America

Bingchen Liu