Microsoft Technology Licensing, LLC

Kuang-Huei Lee

Arun Sacheti

Houdong Hu

Saurajit Mukherjee

Yan Wang

Li Huang

Meenaz Merchant

Surendra Ulabala

Fnu Yokesh Kumar

Nikesh Srivastava

Xi Chelsea Chen

Gang Hua

Linjun Yang

Tianjun Xiao

Jiapei Huang

Rui Xia

Sean Chang Culatana

He Xiaodong

Xi Chen

Yokesh Kumar

Representative embodiments disclose mechanisms to perform visual intent classification or visual intent detection or both on an image. Visual intent classification utilizes a trained machine learning model that classifies subjects in the image according to a classification taxonomy. The visual intent classification can be used as a pre-triggering mechanism to initiate further action in order to substantially save processing time. Example further actions include user scenarios, query formulation, user experience enhancement, and so forth. Visual intent detection utilizes a trained machine learning model to identify subjects in an image, place a bounding box around the image, and classify the subject according to the taxonomy. The trained machine learning model utilizes multiple feature detectors, multi-layer predictions, multilabel classifiers, and bounding box regression.

Visual intent triggering for visual search

The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Image annotation

The present concepts relate to matching data of two different modalities using two stages of attention. First data is encoded as a set of first vectors representing components of the first data, and second data is encoded as a set of second vectors representing components of the second data. In the first stage, the components of the first data are attended by comparing the first vectors and the second vectors to generate a set of attended vectors. In the second stage, the components of the second data are attended by comparing the second vectors and the attended vectors to generate a plurality of relevance scores. Then, the relevance scores are pooled to calculate a similarity score that indicates a degree of similarity between the first data and the second data.

Stacked cross-modal matching

Non-limiting examples described herein relate to ensemble model processing for image recognition that improves precision and recall for image recognition processing as compared with existing solutions. An exemplary ensemble model is configured enhance image recognition processing through aggregate data modeling processing that evaluates image recognition prediction results obtained through processing that comprises: nearest neighbor visual search analysis, categorical image classification analysis and/or categorical instance retrieval analysis. An exemplary ensemble model is scalable, where new segments/categories can be bootstrapped to build deeper learning models and achieve high precision image recognition, while the cost of implementation (including from a bandwidth and resource standpoint) is lower than what is currently available across the industry today. Processing described herein, including implementation of an exemplary ensemble data model, may be exposed as a web service that is standalone or integrated within other applications/services to enhance processing efficiency and productivity applications/services such as productivity applications/services.

Ensemble model for image recognition processing

Growing community of inventors

Redmond, WA, United States of America

Kuang-Huei Lee