A9.com, Inc.

Aishwarya Natesh

Arnab Sanat Kumar Dhua

Ming Du

Pinkee Rasik Patel Gupta

Sharmila Nagaraja Reddy

Colin Jon Taylor

Daniya Zamalieva

Andrea Joyce Diane Zehr

Mehmet Nejat Tek

Andrea Zehr

Shruthi R Bathina

Whitney Chan

Deborah S Hoo

Douglas Ryan Gray

Gautam Bhargava

Rupa Chaturvedi

R Manmatha

Son Dinh Tran

Shabnam Ghadar

Ketan Ramesh Deshpande

Krystle Elaine De Mesa

Smita Malaviya

Julie Chang

Dheeraj Soti

Natasha Duggal

Maya Kabkab

The present embodiments provide visual search techniques which produces results that include both accurate similar items as well diversified items through attribute manipulation. In some embodiments, a feature vector describing the item of interest is obtained. A target feature vector is then generated at least partially from the original feature vector, in which the target feature vector shares only a subset of attribute values with the original feature vector and includes at least some values that are different from the original feature vector. An electronic catalog of items is then queried using the target feature vector, and a set of candidate items are determined from the electronic catalog based at least in part on similarity to the target feature vector. The original feature vector may be used to query for a set of similar items that are as similar as possible to the item of interest.

Systems and method for visual search with attribute manipulation

Various embodiments provide for recommending products from an electronic catalog that are aesthetically compatible with an existing item in a physical space based on a live camera view of the space, and rendering augmented reality views of the recommended products into the live camera view. Specifically, an image of the existing item is used to search a database of images of designed spaces that includes a corresponding object visually similar to the existing item. The image of the designed space may also include other items that are visually compatible with the corresponding object and therefore visually compatible with the existing item. Thus, the images of the other items are used to search an electronic catalog for available items that are visually similar to the other items and therefore would be visually compatible with the existing item.

Recommendations based on object detected in an image

A color selection image matching system can receive, from a computing device, image data captured by a camera in the user device, where the image data includes one or more colors. The computing device can extrapolate a subset of the colors and receive an indication of a target color to initiate a product search based on the target color. The subset of colors may be determined based on other image data, and selectable color elements may be generated for the subset of colors. In some embodiments, the system may generate a palette of colors visually similar to the target color. The target color and/or palette of colors can be cross-referenced with products colors, based on standardized or quantitative color descriptors, to determine relevant product search results matching the target color, which may be displayed on the user device.

Color selection for image matching visual search

Disclosed are various embodiments of systems and methods for dynamically generating and providing personalized recommendations of newer products or services potentially of interest to a particular user who has previously purchased a similar product or service. Historical purchase data or other information indicating the user's preferences is analyzed to determine personal preference data. Candidate content is identified based on attributes found in the preference data. Similarity strategies and criteria can be used to test features and qualities in candidate content. Recommended product or service content comes in the form of candidate content which reaches a similarity threshold or otherwise achieves a sufficient confidence score based at least in part on a similarity metric is determined. In accordance with various embodiments, the generation of newer product and service recommendations can be accomplished or optimized by training machine learning models, such as deep neural networks, using preference data for the particular user.

Dynamic recommendations personalized by historical data

Various embodiments provide techniques for generating a style profile in which items are recognized in image data, analyzed for their attributes, and categorized based on those attributes. In various embodiments, computer vision techniques can be used to analyze the image data and subsequent image data to update the generated style profiles. The style profiles may be associated with a person, event, or the like and provide users with items having similar characteristics or attributes, which may be available for purchase in an electronic marketplace. The style profiles may be updated by analyzing user interaction with the provided items and by receiving additional image data. Additionally, recommendations may be provided based on search queries that identify certain events or style profiles.

Computer vision based style profiles

Various embodiments utilize a machine learning-based approach to filter items, such as apparel items, based on non-binary gender styles. For example, an electronic catalog of apparel items can be assigned gender scores on a gender scale by a neural network trained to determine a gender score of an apparel item based on an image representation of the apparel item. The neural network may be trained on training data that includes images of various apparel items with gender designations. The apparel items in the electronic catalog are assigned a gender score attribute that reflects how masculine or feminine the apparel item may be. As such, the apparel items can be organized (e.g., sorted, filtered, ranked) based on a non-binary gender score in addition to other attributes, such as item type, size, color, brand, etc. Thus, a user can include non-binary gender style as a search or filtering criteria.

Non-binary gender filter

Approaches provide for an interactive interface for determining annotation information that can be used to recommend content, improve user interaction and satisfaction, as well as improve various image recognition approaches. For example, a user can be presented an interactive interface, such as a game. The interactive interface can include colored graphical elements, where each graphical element can be associated with and display a visual feature of an item. A user can be provided a task or goal, such as to select, move, or otherwise arrange graphical elements based on a visual feature of the elements. The user interaction can generate annotation information that can be utilized to determine related content as well as improve various image recognition approaches.

Interactive interfaces for generating annotation information

Images may be analyzed to determine a visually cohesive color palette, for example by comparing a subset of the colors most frequently appearing in the image to a plurality of color schemes (e.g., complementary, analogous, etc.), and potentially modifying one or more of the subset of colors to more accurately fit the selected color scheme. Various regions of the image are selected and portions of the regions having one or more colors of the color palette are extracted and classified to generate and compare feature vectors of the patches to previously-determined feature vectors of items to identify visually similar items. The visually similar items are selected for presentation in various ways, such as by choosing an outfit of visually-similar apparel items based on the locations of the corresponding colors in the image, etc.

Item recommendation based on feature match

Approaches attempt to determine information that can help to produce more useful recommendations to be displayed in a situation where no, or little, information is available that indicates a relationship between content provided through an electronic marketplace or other content provider. For example, data available that relates to an item in a product catalog, for example color data, can be analyzed and aggregated in order to attempt to locate other items that are related and relevant to the item, at least as it relates to color and categorization of the content. Such approaches can include, for example, analyzing images, articles, and other sources of electronic content to attempt to locate items that might be relevant to the item of interest. In a clothing example, this can include accessory items that might be worn with an apparel item of interest, match the apparel item of interest, be frequently utilized or exhibited with the apparel item of interest, include a matching and/or complementary color to the apparel item of interest, etc.

Content recommendation based on color match

Training images can be synthesized in order to obtain enough data to train a convolutional neural network to recognize various classes of a type of item. Images can be synthesized by blending images of items labeled using those classes into selected background images. Catalog images can represent items against a solid background, which can be identified using connected components or other such approaches. Removing the background using such approaches can result in edge artifacts proximate the item region. To improve the results, one or more operations are performed, such as a morphological erosion operation followed by an opening operation. The isolated item portion then can be blended into a randomly selected background region in order to generate a synthesized training image. The training images can be used with real world images to train the neural network.

Synthesizing training data

A set of training images is obtained by analyzing text associated with various images to identify images likely demonstrating a visual attribute. Localization can be used to extract patches corresponding to these attributes, which can then have features or feature vectors determined to train, for example, a convolutional neural network. A query image can be received and analyzed using the trained network to determine a set of items whose images demonstrate visual similarity to the query image at least with respect to the attribute of interest. The similarity can be output from the network or determined using distances in attribute space. Content for at least a determined number of highest ranked, or most similar, items can then be provided in response to the query image.

Growing community of inventors

Mountain View, CA, United States of America

Aishwarya Natesh