Microsoft Technology Licensing, LLC

Ye Wu

Yan Wang

Arun Sacheti

Houdong Hu

Xi Chelsea Chen

Linjun Yang

Li Huang

Meenaz Merchant

Surendra Ulabala

Vishal Chhabilbhai Thakkar

Jiapei Huang

Described herein is a mechanism for visual recognition of items or visual search using Optical Character Recognition (OCR) of text in images. Recognized OCR blocks in an image comprise position information and recognized text. The embodiments utilize a location-aware feature vector created using the position and recognized information in each recognized block. The location-aware features of the feature vector utilize position information associated with the block to calculate a weight for the block. The recognized text is used to construct a tri-character gram frequency, inverse document frequency (TGF-IDP) metric using tri-character grams extracted from the recognized text. Features in location-aware feature vector for the block are computed by multiplying the weight and the corresponding TGF-IDF metric. The location-aware feature vector for the image is the sum of the location-aware feature vectors for the individual blocks.

Layout-aware, scalable recognition system

A computer-implemented technique generates a dense embedding vector that provides a distributed representation of input text. The technique includes: generating an input term-frequency (TF) vector of dimension g that includes frequency information relating to frequency of occurrence of terms in an instance of input text; using a TF-modifying component to modify the term-specific frequency information in the input TF vector by respective machine-trained weighting factors, to produce an intermediate vector of dimension g; using a projection component to project the intermediate vector of dimension g into an embedding vector of dimension k, where k is less than g. Both the TF-modifying component and the projection component use respective machine-trained neural networks. An application performs any of a retrieval-based function, a recognition-based function, a recommendation-based function, a classification-based function, etc. based on the embedding vector.

Encoder using machine-trained term frequency weighting factors that produces a dense embedding vector

Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Multi-modal visual search pipeline for web scale images

Growing community of inventors

Redmond, WA, United States of America

Ye Wu