Amazon Technologies, Inc.

Markable, Inc.

Suren Kumar

Rui Zheng

Dirk Padfield

Nikhil Garg

Huang Tang

Anders Jonas Marcus Grimfelt

Jugal Sheth

Thomas Balestri

Systems and methods are provided for classifying images associated with an item, and generating an image set for that item which includes image classifications determined to be helpful for the item type of the item. To classify images, an image classification model is generated and trained using two phases. The first phase uses intermediate model with text and visual processing to teach the model to recognize patterns created by text without requiring OCR at inference. The second phase uses visual processing to refine the model for use at inference. To generate an image set, image classifications helpful to an item type are identified, items are associated with item types, images are obtained for an item, the images are classified using the image classification model, missing image classifications set out in the preferred image set are identified, and a request or requests is generated for the missing image classifications.

Image classification with modality dropout

Systems that search databases of videos or images to identify similar products in a given video or image of a product are disclosed. The content of the given video is represented by a feature vector used to measure the given video's similarity to either a video or an image. When the system is deployed to recognize particular fashion items in videos, some such videos are taken in uncontrolled settings, and as a result, may have low resolution, poor contrast, minimal focus, motion blur, or low lighting. By recognizing and removing poor quality video frames from the image recognition pipeline, associating products across video frames to form tracklets of each product, and enriching the feature representation of each item for best retrieval result by fusing information from multiple video frames depicting the item, the system addresses the aforementioned shortcomings.

Systems and methods for improving visual search using summarization feature

An image processing system identifies objects within images or video segments. To identify an object within an image, the system identifies one or more regions of an image that contain an object. In some examples, a tracklet is used to track an object though a plurality of image frames within a video segment allowing more than one image frame to be used in object detection, and thereby increasing detection accuracy. Various embodiments utilize a deep learning based object detection framework and similar object search framework that models the correlations present between various object categories. The system determines a category for each object detected using a hierarchical tree of categories to learn the visual similarities between various object categories. The hierarchical tree is estimated by analyzing the errors of an object detector which does not use any correlation between the object categories.

Image processing system

Machine learning techniques are disclosed for training a model to identify each of multiple different classes in images, based on training data where each training image may not be labeled in a complete manner with respect to the classes. The disclosed training techniques use a new label value to indicate when a ground truth value is unknown for a particular class, and do not penalize the machine learning model for output predictions that do not match the label value representing unknown ground truth. The disclosed processes may, for example, be used to train a model to detect each multiple types of image defects based on incomplete information provided by human reviewers who accept and reject images based on whether any of the types of image defects are found.

Model training using incomplete indications of types of defects present in training images

Methods and systems for training a metric learning convolutional neural network (CNN)-based model for cross-domain image retrieval are disclosed. The methods and systems perform steps of generating a plurality of batches sampled from a cross-domain training dataset to train the CNN-based model to match images of different sub-categories from one domain to another, and training the CNN-based model using the generated batches. The CNN-based model comprises various pooling, normalization, and concatenation layers that enable it to concatenate the normalized outputs of multiple concatenation layers. Use of the generated batches comprises executing a loss function based on one or more batches, where the loss function is a triplet, contrastive, or cluster loss function. Embodiments of the present invention enable the CNN-based model to summarize information from multiple convolutional layers, thus improving visual search. Also disclosed are benefits of the new methods, and alternative embodiments of implementation.

Aspects of the present disclosure relate to machine learning techniques for training a model to identify each of a number of different classes in images, based on training data where each training image may not be labeled in a complete manner with respect to the classes. The disclosed training techniques use a new label value to indicate when a ground truth value is unknown for a particular class, and do not penalize the machine learning network for output predictions that do not match the label value representing unknown ground truth. Some implementations of the training process can be regularized to impose sparsity on predicted classes in order to avoid false positive predictions.

Growing community of inventors

Seattle, WA, United States of America

Suren Kumar