Microsoft Technology Licensing, LLC

Rovi Technologies Corporation

Other

Tel Aviv University

Noam Koenigstein

Nir Nice

Oren Barkan

Itzik Malkiel

Ulrich Paquet

Ori Katz

Omri Armstrong

Roy Hirsch

Noam Razin

Shahar Keren

Avi Caciularu

Yonathan Weill

Dvir Ginzburg

Ran Gilad-Bachrach

Daniel Sitton

Liran Katzir

Amir Hertz

Nimrod Ben Simhon

Dror Kremer

Gal Lavee

Paul Nogues

Yehuda Finkelstein

Shimon Shlevich

Shay Ben-Elazar

Shai Roitman

Jonathan Ephrath

Eylon Yogev

Yehiel Berezin

Shay Ben Elazar

Yael Brumer

Tal Zaccai

Idan Rejwan

Ilona Kifer

Amit Perelstein

Pavel Roit

Yuval Shavitt

Tomer Tankel

A method of training a machine learning model is provided. The method includes receiving labeled training data in the machine learning model, the received labeled training data including content data for items accessible to a user and input usage data representing recorded interaction between the user and the items, wherein the received content data for each item includes data representing intrinsic attributes of the item. The method further includes selecting a set of the input usage data that excludes input usage data for a proper subset of the items and training the machine learning model based on both the content data and the selected set of input usage data of the received labeled training data for the items.

Model with usage data compensation

The disclosure herein describes a system and method for attentive sentence similarity scoring. A distilled sentence embedding (DSE) language model is trained by decoupling a transformer language model using knowledge distillation. The trained DSE language model calculates sentence embeddings for a plurality of candidate sentences for sentence similarity comparisons. An embedding component associated with the trained DSE language model generates a plurality of candidate sentence representations representing each candidate sentence in the plurality of candidate sentences which are stored for use in analyzing input sentences associated with queries or searches. A representation is created for the selected sentence. This selected sentence representation is used with the plurality of candidate sentence representations to create a similarity score for each candidate sentence-selected sentence pair. A retrieval component identifies a set of similar sentences from the plurality of candidate sentences responsive to the input query based on the set of similarity scores.

Sentence similarity scoring using neural network distillation

A relational similarity determination engine receives as input a dataset including a set of entities and co-occurrence data that defines co-occurrence relations for pairs of the entities. The relational similarity determination engine also receives as input side information defining explicit relations between the entities. The relational similarity determination engine jointly models the co-occurrence relations and the explicit relations for the entities to compute a similarity metric for each different pair of entities within the dataset. Based on the computed similarity metrics, the relational similarity determination engine identifies a most similar replacement entity from the dataset for each of the entities within the dataset. For a select entity received as an input, the relational similarity determination engine outputs the identified most similar replacement entity.

Representation learning with side information

A method of generating an aggregate saliency map using a convolutional neural network. Convolutional activation maps of the convolutional neural network model are received into a saliency map generator, the convolutional activation maps being generated by the neural network model while computing the one or more prediction scores based on unlabeled input data. Each convolutional activation map corresponds to one of the multiple encoding layers. The saliency map generator generates a layer-dependent saliency map for each encoding layer of the unlabeled input data, each layer-dependent saliency map being based on a summation of element-wise products of the convolutional activation maps and their corresponding gradients. The layer-dependent saliency maps are combined into the aggregate saliency map indicating the relative contributions of individual components of the unlabeled input data to the one or more prediction scores computed by the convolutional neural network model on the unlabeled input data.

Deep gradient activation map model refinement

Aspects of the technology described herein use acoustic features of a music track to capture information for a recommendation system. The recommendation can work without analyzing label data (e.g., genre, artist) or usage data for a track. For each audio track, a descriptor is generated that can be used to compare the track to other tracks. The comparisons between track descriptors result in a similarity measure that can be used to make a recommendation. In this process, the audio descriptors are used directly to form a track-to-track similarity measure between tracks. By measuring the similarity between a track that a user is known to like and an unknown track, a decision can be made whether to recommend the unknown track to the user.

Song similarity determination

A diagnostic tool for deep learning similarity models and image classifiers provides valuable insight into neural network decision-making. A disclosed solution generates a saliency map by: receiving a test image; determining, with an image classifier, an image classification of the test image; determining, for the test image, a first activation map for at least one model layer using the determined image classification; determining, for the test image, a first gradient map for the at least one model layer using the determined image classification; and generating a first saliency map as an element-wise function of the first activation map and the first gradient map.

Diagnostic tool for deep learning similarity models

An anchor-based collaborative filtering system receives a training dataset including user-item interactions each identifying a user and an item that the user has positively interacted with. The system defines a vector space and distributes the items of the training dataset within the vector based on a determined similarity of the items. The system further defines a set of taste anchors that are each associated in memory with a subgroup of the items in a same neighborhood of the vector space. To make a recommendation to an individual user, the system identifies an anchor-based representation for the individual user that includes a subset of the defined taste anchors that best represents the types of items that the user has favorably interacted with in the past. The taste anchors included in the identified anchor-based representation for the individual user are used to make recommendations to the user in the future.

Anchor-based collaborative filtering

Machine learning multiple features of an item depicted in images. Upon accessing multiple images that depict the item, a neural network is used to machine train on the plurality of images to generate embedding vectors for each of multiple features of the item. For each of multiple features of the item depicted in the images, in each iteration of the machine learning, the embedding vector is converted into a probability vector that represents probabilities that the feature has respective values. That probability vector is then compared with a value vector representing the actual value of that feature in the depicted item, and an error between the two vectors is determined. That error is used to adjust parameters of the neural network used to generate the embedding vector, allowing for the next iteration in the generation of the embedding vectors. These iterative changes continue thereby training the neural network.

Machine learning multiple features of depicted item

Examples provide a self-supervised language model for document-to-document similarity scoring and ranking long documents of arbitrary length in an absence of similarity labels. In a first stage of a two-staged hierarchical scoring, a sentence similarity matrix is created for each paragraph in the candidate document. A sentence similarity score is calculated based on the sentence similarity matrix. In the second stage, a paragraph similarity matrix is constructed based on aggregated sentence similarity scores associated with the first candidate document. A total similarity score for the document is calculated based on the normalize the paragraph similarity matrix for each candidate document in a collection of documents. The model is trained using a masked language model and intra-and-inter document sampling. The documents are ranked based on the similarity scores for the documents.

Self-supervised document-to-document similarity system

The disclosure herein describes a system for interpreting text-based similarity between a seed item and a recommended item selected by a pre-trained language model from a plurality of candidate items based on semantic similarities between the seed item and the recommended item. The system analyzes similarity scores and contextual paragraph representations representing text-based descriptions of the seed item and recommended item to generate gradient maps and word scores representing the text-based descriptions. A model for interpreting text-based similarity utilizes the calculated gradients and word scores to match words from the seed item description with words in the recommended item description having similar semantic meaning. The word-pairs having the highest weight are identified by the system as the word-pairs having the greatest influence over the selection of the recommended item from the candidate items by the original pre-trained language model.

Interpreting text-based similarity

Semantic search techniques via focused summarizations are described. For example, a search query is received for a text-based content item in a data set comprising a plurality of text-based content items. A first feature vector representative of the search query is obtained. A respective semantic similarity score is determined between the first feature vector and each of a plurality of second feature vectors. Each of the second feature vectors is representative of a machine-generated summarization of a respective text-based content item. The machine-generated summarization comprises a plurality of multi-word fragments that are selected from the respective text-based content item via a transformer-based machine learning model. A search result is provided responsive to the search query. The search result comprises a subset of the plurality of text-based content items associated with a respective second feature vector having a semantic similarity score that has a predetermined relationship with a predetermined threshold value.

Systems and methods for semantic search via focused summarizations

A diagnostic tool for deep learning similarity models and image classifiers provides valuable insight into neural network decision-making. A disclosed solution generates a saliency map by: receiving a baseline image and a test image; determining, with a convolutional neural network (CNN), a first similarity between the baseline image and the test image; based on at least determining the first similarity, determining, for the test image, a first activation map for at least one CNN layer; based on at least determining the first similarity, determining, for the test image, a first gradient map for the at least one CNN layer; and generating a first saliency map as an element-wise function of the first activation map and the first gradient map. Some examples further determine a region of interest (ROI) in the first saliency map, cropping the test image to an area corresponding to the ROI, and determine a refined similarity score.

Solutions for localizing relevant objects in multi-object images include receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the relevant detected object as an output result but not indicating the irrelevant detected object as an output result. Some examples identify a plurality of objects that are visually similar to the relevant object and displaying the visually similar objects to a user, for example as recommendations of alternative catalog items on an e-commerce website. Some examples are able to identify a plurality of relevant objects and display multiple sets of visually similar objects.

Localizing relevant objects in multi-object images

Growing community of inventors

Tel Aviv, Israel

Noam Koenigstein