Salesforce, Inc.

Nikhil Naik

Ali Madani

Ramprasaath Ramasamy Selvaraju

Nitish Shirish Keskar

Bram Wallace

Bryan McCann

Juan Carlos Niebles Duque

Brian Chen

Embodiments described herein provide a method of generating an image. the method comprises receiving, via a data interface, a natural language prompt, obtaining a noised image vector, and generating a denoised image vector by a first forward pass of a plurality of iterations of a denoising diffusion model with the noised image vector as an input and conditioned on the natural language prompt. The method further includes calculating a gradient of a loss function based on the denoised image vector with respect to the noised image vector, and updating the noised image vector based on the gradient. A final image is generated using a final forward pass of the denoising diffusion model with the updated noised image vector as an input and conditioned on the natural language prompt.

Systems and methods for image generation via diffusion

Embodiments described herein provide systems and methods for image editing, a first copy and a second copy of an input image are generated; noise is iteratively added to the first copy and the second copy by: updating the first copy based on a first inverted output of a denoising diffusion model (DDM) based on the second copy and a first caption and updating the second copy based on a second inverted output of the DDM based on the first copy and the first caption. A resultant noised image is iteratively denoised by a reverse process using the DDM conditioned on a second caption, thereby producing a final image.

Systems and methods for reversible transformations using diffusion models

Embodiments described herein embodiments described herein provide Contrastive Attention-Supervised Tuning (CAST), a training method to fix the visual grounding ability of contrastive SSL methods based on a data augmentation strategy using unsupervised saliency maps. In addition to the contrastive loss that encourages the model to pick the crop that comes from the corresponding image, CAST provides an explicit grounding supervision through a Grad-CAM based attention loss that enforces models to look at the specified object of interest that is common across different crops when making this decision. A new geometric transform is introduced for randomly cropping different views from an input image based on certain constraints derived from a saliency map.

Systems and methods for contrastive attention-supervised tuning

Embodiments described herein provide an intelligent method to select instances, by utilizing unsupervised tracking for videos. Using this freely available form of supervision, a temporal constraint is adopted for selecting instances that ensures that different instances contain the same object while sampling the temporal augmentation from the video. In addition, using the information on the spatial extent of the tracked object, spatial constraints are applied to ensure that sampled instances overlap meaningfully with the tracked object. Taken together, these spatiotemporal constraints result in better supervisory signal for contrastive learning from videos.

Systems and methods for contrastive pretraining with video tracking supervision

The present disclosure provides systems and methods for controllable protein generation. According to some embodiments, the systems and methods leverage neural network models and techniques that have been developed for other fields, in particular, natural language processing (NLP). In some embodiments, the systems and methods use or employ models implemented with transformer architectures developed for language modeling and apply the same to generative modeling for protein engineering.

Systems and methods for language modeling of protein engineering

An analytics system uses one or more machine-learned models to predict a hormone receptor status from a H&E stain image. The system partitions H&E stain images each into a plurality of non-overlapping image tiles. Bags of tiles are created through sampling of the image tiles. For each H&E stain image, the system generates a feature vector from a bag of tiles sampled from the partitioned image tiles. The analytics system trains one or more machine-learned models with training H&E stain images having a positive or negative receptor status. With the trained models, the analytics system predicts a hormone receptor status by applying a prediction model to the feature vector for a test H&E stain image.

Machine-learned hormone status prediction from image analysis

An analytics system uses one or more machine-learned models to predict a hormone receptor status from a H&E stain image. The system partitions H&E stain images each into a plurality of image tiles. Bags of tiles are created through sampling of the image tiles. The analytics system trains one or more machine-learned models with training H&E stain images having a positive or negative receptor status. The analytics system generates, via a tile featurization model, a tile feature vector for each image tile a test bag for a test H&E stain image. The analytics system generates, via an attention model, an aggregate feature vector for the test bag by aggregating the tile feature vectors of the test bag, wherein an attention weight is determined for each tile feature vector. The analytics system predicts a hormone receptor status by applying a prediction model to the aggregate feature vector for the test bag.

Growing community of inventors

Palo Alto, CA, United States of America

Nikhil Naik