Microsoft Technology Licensing, LLC

Adit Krishnan

Ji Li

Amit Srivastava

Han Hu

Qi Dai

Yixuan Wei

Dachuan Zhang

Yue Cao

Xiaozhi Yu

Varun Tandon

Aman Malik

A system for retrieving multimodal assets using domain-specific knowledge includes receiving a search query for searching for multimodal assets; encoding the search query into a first query representation via a first trained query representation machine-learning (ML) model and a second query representation via a second trained query representation ML model; comparing the first query representation to a plurality of multimodal representations to calculate a first similarity score, each of the plurality of multimodal representations being a representation of one of the plurality of candidate multimodal assets; comparing the second query representation to a plurality of domain-specific representations to calculate a second similarity score, the domain-specific representations being representations of domain-specific data associated with one or more of the plurality of the multimodal representations; calculating a third similarity score based on keyword matching between the domain-specific data and the one or more search terms in the search query; aggregating the first, second and third similarity scores to calculate a total similarity score for each of the plurality of candidate multimodal assets; ranking the plurality of candidate multimodal assets based on the total similarity scores to identify search results for the search query; and providing the identified candidate multimodal assets for display as the search results.

Method and system of using domain specific knowledge in retrieving multimodal assets

A system and method and for retrieving one or more one or more multimodal assets includes receiving a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, encoding the search query into one or more query embedding representations via a trained query representation machine-learning (ML) model, comparing, via a matching unit, the one or more query embedding representations to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal assets, and identifying, based on the comparison, at least one of the plurality of the candidate multimodal assets as a search result for the search query, and providing the at least one of the plurality of the candidate multimodal assets for display as the search result.

Method and system of retrieving multimodal assets

A data processing system implements a dynamic knowledge distillation process including dividing training data into a plurality of batches of samples and distilling a student model from a teacher model using an iterative knowledge distillation. The process includes instantiating an instance of the teacher model and the student model in a memory of the data processing system and obtaining a respective batch of training data from the plurality of batches of samples in the memory. The process includes training the teacher and student models using each of the samples in the respective batch of the training data, evaluating the performance of the student model compared with the performance of the teacher model, and providing feedback to student model to adjust the behavior of the student model based on the performance of the student model.

Scalable knowledge distillation techniques for machine learning

A system and method and for retrieving assets from a personalized asset library includes receiving a search query for searching for assets in one or more asset libraries, the one or more asset libraries including a personalized asset library; encoding the search query into embedding representations via a trained query representation machine-learning (ML) model; comparing, via a matching unit, the query embedding representations to a plurality of asset representations, each of the plurality of asset representations being a representation of one of the plurality of candidate assets; identifying, based on the comparison, at least one of the plurality of the candidate assets as a search result for the search query; and providing the identified plurality of candidate assets for display as the search result. The plurality of asset representations for the one or more assets in the personalized content library are generated automatically without human labeling.

Method and system of retrieving assets from personalized asset libraries

A system and method and for retrieving one or more visual assets includes receiving a search query for the one or more visual assets, the search query including textual data, encoding the textual data into one or more text embedding representations via a trained text representation machine-learning (ML) model, transmitting the one or more text embedding representations to a matching and selection unit, providing visual embedding representations of one or more visual assets to the matching and selection unit, comparing, by the matching and selection unit, the one or more text embedding representations to the visual embedding representations to identify one or more visual asset search results, and providing the one or more visual asset search results for display.

Method and system of content retrieval for visual data

A data processing system implements receiving query text for a search query for textual content recommendation. The query text includes one or more words indicating a type of textual content items being sought. The system implements analyzing the query text using a first machine learning (ML) model to obtain encoded query text, where the first ML model is trained to identify features within the query text and to generate the encoded query text by mapping the features to a hyper-dimensional latent space (HDLS). The system implements identifying one or more content items in a database of encoded content items mapped to the HDLS that satisfy the search query by comparing attributes of the encoded query text with attributes of the encoded content items to identify content items that are closest to the encoded query text within the HDLS, and causing the one or more content items to be displayed.

Growing community of inventors

Mountain View, CA, United States of America

Adit Krishnan