Adobe, Inc.

Intel Corporation

Google Inc.

Interdigital Ce Patent Holdings

Interdigital Madison Patent Holdings, Sas

Branislav Kveton

Hung Hai Bui

Zheng Wen

Yasin Abbasi Yadkori

Sungchul Kim

Mohammad Ghavamzadeh

Jaya Kawale

Nikolaos Vlassis

Eunyee Koh

Vishwa Vinay

Atanu Ranjan Sinha

Nikhil Sheoran

Deepali Jain

Deepali Gupta

Charles Li Chen

Claire Vernade

Suvash Sedhain

Shuai Li

Victor B Lortz

Vijay Sarathi Kesavan

Anand P Rangarajan

Trevor A Pering

Handong Zhao

Gaurush Hiranandani

Iftikhar Ahamath Burhanuddin

Kenton M Lyons

Trevor Paulsen

Somya Rathi

Meiyuan Zhao

Prakhar Gupta

Harvineet Singh

Sheng Li

Mariano J Phielipp

Craig Mathis

Ivan Andrus

Kristopher Paries

Muhammad Jehangir Amjad

Christophe Diot

Michael Rimer

Sandilya Bhamidipati

Magdiel F Galán

Xiaolong Yang

Augustin Soule

Dimitris Konomis

Hamid Dadkhahi

Yang Cao

Systems and techniques are provided for monitoring of a large-scale network, such as a large-scale network supporting cloud infrastructures. A minimum fixed probe allocation and/or a sampling budget for monitoring may be set. A probing and/or sampling strategy may be optimized in order to measure network metrics, such as error metrics associated with the latency of a network, with a known accuracy, given a particular probe allocation. The systems and techniques provided may leverage particular designs to determine efficient probing strategies for the network, while simultaneously conserving computing resources. In some examples, instead of using these frameworks and techniques directly in production networks, a scalable and near optimal approximation technique based on the Frank-Wolfe algorithm may be used.

Scalable and low computation cost method for optimizing sampling/probing in a large scale network

Systems and techniques for multi-task equidistant embedding are described that process categorical feature data to explore feature interactions. A digital analytics system enforces an equidistant relationship among features within a category while extracting high-order feature interactions by punishing both positive correlations and negative correlations among low-dimensional representations of different features. By enforcing an equidistant embedding, information is retained and accuracy is increased while higher order feature interactions are determined. Further, the digital analytics system shares knowledge among different tasks by connecting a shared network representation common to multiple tasks with exclusive network representations specific to particular tasks.

Multi-task equidistant embedding

The present disclosure is directed toward systems, methods, and computer readable media for training and utilizing an item-level importance sampling model to evaluate and execute digital content selection policies. For example, systems described herein include training and utilizing an item-level importance sampling model that accurately and efficiently predicts a performance value that indicates a probability that a target user will interact with ranked lists of digital content items provided in accordance with a target digital content selection policy. Specifically, systems described herein can perform an offline evaluation of a target policy in light of historical user interactions corresponding to a training digital content selection policy to determine item-level importance weights that account for differences in digital content item distributions between the training policy and the target policy. In addition, the systems described herein can apply the item-level importance weights to training data to train item-level importance sampling model.

Method, medium, and system for utilizing item-level importance sampling models for digital content selection policies

Methods, systems, and non-transitory computer readable storage media are disclosed for determining parameters for digital campaign content in connection with executing digital campaigns using a rank-one assumption and a best-arm identification algorithm. For example, the disclosed system alternately explores response data in the first dimension and response data in the second dimension using the rank-one assumption and the best-arm identification algorithm to estimate highest sampling values from each dimension. In one or more embodiments, the disclosed system uses the estimated highest sampling values from the first and second dimension to determine a combination with a highest sampling value in a parameter matrix constructed based on the first dimension and the second dimension, and then executes the digital campaign using the determined combination.

Multivariate digital campaign content exploration utilizing rank-1 best-arm identification

Techniques disclosed herein provide more efficient and more relevant item recommendations to users in large-scale environments in which only positive interest information is known. The techniques use a rank-constrained formulation that generalizes relationships based on known user interests in items and/or use a randomized singular value decomposition (SVD) approximation technique to solve the formulation to identify items of interest to users in an efficiently, scalable manner.

Item recommendation techniques

Systems and methods are disclosed herein for providing a user interface representing differences between segments of end users. The systems and methods receive user input on a user interface identifying a first segment, the first segment being a subset of the end users having a particular characteristic, determine differences between the first segment and a second segment, and represent, on the user interface, the differences between the first segment and the second segment based on relative significances of the differences. The marketer using the user interface is able to quickly and easily identify the metrics, dimensions, and/or relationships to other segments that most distinguish the compared segments from one another.

Audience comparison

Multivariate digital campaign content testing utilizing rank-1 best-arm identification

Certain embodiments involve determining and outputting correlations between metrics in large-scale web analytics datasets. For example, a processor identifies pairs of data metrics in a web analytics data set and determines a Maximal Information Coefficient (MIC) score for each pair of data metrics that indicates a strength of a correlation between the pair of data metrics. The processor generates an interactive user interface that graphically displays each pair of correlated data metrics having an MIC score above a threshold and the interactive user interface indicates the strength of the correlation between each displayed pair of correlated data metrics. The processor receives user input indicating an adjustment to the threshold and modifies the interactive user interface in response to receiving the user input by adding pairs of correlated data metrics to, or removing pairs of correlated metrics from, the interactive user interface based on the adjustment to the threshold.

Methods and systems for determining and outputting correlations between metrics in a web analytic dataset

A machine-learning framework uses partial-click feedback to generate an optimal diverse set of items. An example method includes estimating a preference vector for a user based on diverse cascade statistics for the user, the diverse cascade statistics including previously observed responses and previously observed topic gains. The method also includes generating an ordered set of items from the item repository, the items in the ordered set having highest topic gain weighted by similarity with the preference vector, providing the ordered set for presentation to the user, and receiving feedback from the user on the ordered set. The method also includes, responsive to the feedback indicating a selected item, updating the diverse cascade statistics for observed items, wherein the updating results in penalizing the topic gain for items of the observed items that are not the selected item and promoting the topic gain for the selected item.

Online diverse set generation from partial-click feedback

Recommendation systems and techniques are described that employ change point detection to generate recommendations for digital content. In one example, a change point detection technique is employed by a recommendation system to identify when a change point has occurred at a respective time step of a series of time steps. Detection of this change point may then be used by the recommendation system to reset the statistical model to address this change as well as generate a subsequent recommendation configured for exploration of reward distributions of the items of digital marketing content.

Change point detection in a multi-armed bandit recommendation system

Systems and methods provide for generating predictive models that are useful in predicting next-user-actions. User-specific navigation sequences are obtained, the navigation sequences representing temporally-related series of actions performed by users during navigation sessions. To each navigation sequence, a Recurrent Neural Network (RNN) is applied to encode the navigation sequences into user embeddings that reflect time-based, sequential navigation patterns for the user. Once a set of navigation sequences is encoded to a set of user embeddings, a variety of classifiers (prediction models) may be applied to the user embeddings to predict what a probable next-user-action may be and/or the likelihood that the next-user-action will be a desired target action.

Predictive analysis of target behaviors utilizing RNN-based user embeddings

Method, medium, and system for training and utilizing item-level importance sampling models

Particle Thompson Sampling for online matrix factorization recommendation is described. In one or more implementations, a recommendation system provides a recommendation of an item to a user using Thompson Sampling. The recommendation system then receives a rating of the item from the user. Unlike conventional solutions which only update the user latent features, the recommendation system updates both user latent features and item latent features in a matrix factorization model based on the rating of the item. The updating is performed in real time which enables the recommendation system to quickly adapt to the user ratings to provide new recommendations. In one or more implementations, to update the user latent features and the item latent features in the matrix factorization model, the recommendation system utilizes a Rao-Blackwellized particle filter for online matrix factorization.

Particle thompson sampling for online matrix factorization recommendation

Growing community of inventors

San Jose, CA, United States of America

Branislav Kveton