Adobe, Inc.

Intel Corporation

Georgios Theocharous

Anup Bandigadi Rao

William Noah Schilit

Uttam K Sengupta

Nilesh N Shah

Ryan A Rossi

Robert A Dunstan

Kelan Craig Silvester

Sana Malik Lee

Tung Mai

Nikolaos Vlassis

Yash Chandak

Eunyee Koh

Fan Du

Michele Saad

Mohammad Ghavamzadeh

David Arbour

Sridhar Mahadevan

Kirankumar Shiragur

Balaji Krishnamurthy

Jennifer Anne Healey

Gang Wu

Handong Zhao

Iftikhar Ahamath Burhanuddin

Kent Andrew Edmonds

Hung Hai Bui

Trevor Paulsen

Haoliang Wang

Zheng Wen

Pinkesh Badjatiya

Kanak Vivek Mahadik

Sunny Dhamnani

Yasin Abbasi Yadkori

Atanu R Sinha

Navita Goyal

Tanay Anand

Zhao Song

Jayakumar Subramanian

Karan Aggarwal

Chunyuan Li

Qingyun Wu

Tanay Asija

Kaarthik Raja Meenakshi Viswanathan

Raja Kumar Dubey

Mandana Hamidi Haines

Sriyash Poddar

Assaf Joseph Hallak

Kai Wang

Sougata Chaudhuri

James Kostas

Tong Mu

Shreyas Chaudhari

Sorathan Chaturapruek

Frits De Nijs

Youngsuk Park

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate digital item recommendations for client devices utilizing coagent recommendation models of a distributed asynchronous coagent network. Indeed, in one or more embodiments, the disclosed systems operate on an edge computing device of a distributed asynchronous coagent network. In some cases, the disclosed systems utilize recommendation scores generated at the edge computing device via local coagents and additional recommendation scores received from other coagents of other edge computing devices to generate a digital item recommendation. In some cases, the disclosed systems progressively refines the recommendation as delayed scores from the other coagents are received. Further, in some embodiments, the disclosed systems update parameters of the local coagents using local policy gradients determined from responses to the generated recommendations.

Method, system, and medium for generating recommendations utilizing an edge-computing-based asynchronous coagent network

The present disclosure relates to systems, non-transitory computer-readable media, and methods for performing off-policy evaluations of slate recommendation policies through additive decomposition. In particular, in one or more embodiments, the disclosed systems receive historical data corresponding to digital slate recommendations performed by a first slate recommendation policy, with each slate recommendation comprising a plurality of digital slot recommendations. Additionally, in some embodiments, the disclosed systems generate a second slate action using a second slate recommendation policy conditioned on user context. Further, in some embodiments, the disclosed systems generate a plurality of importance weights by summing a plurality of slot-level density ratios generated by comparing the slate actions of the second slate recommendation policy to the slate actions of the first slate recommendation policy. In some embodiments, the disclosed systems apply the plurality of importance weights to generate a predicted reward distribution for evaluation of the second slate recommendation policy.

Utilizing additive decomposition for universal off-policy evaluation of digital content slate recommendations

Systems and methods for sequential recommendation receive a user interaction history including interactions of a user with a plurality of items, select a constraint from a plurality of candidate constraints based on lifetime values observed for the candidate constraints, wherein the lifetime values are based on items predicted for other users using a recommendation network subject to the candidate constraints, and predict a next item for the user based on the user interaction history using the recommendation network subject to the selected constraint.

Constraint sampling reinforcement learning for recommendation systems

The present disclosure relates to generating proposed digital actions in high-dimensional action spaces for client devices utilizing reinforcement learning models. For example, the disclosed systems can utilize a supervised machine learning model to train a latent representation decoder to determine proposed digital actions based on latent representations. Additionally, the disclosed systems can utilize a latent representation policy gradient model to train a state-based latent representation generation policy to generate latent representations based on the current state of client devices. Subsequently, the disclosed systems can identify the current state of a client device and a plurality of available actions, utilize the state-based latent representation generation policy to generate a latent representation based on the current state, and utilize the latent representation decoder to determine a proposed digital action from the plurality of available actions by analyzing the latent representation.

Generating and providing proposed digital actions in high-dimensional action spaces using reinforcement learning models

A single unified machine learning model (e.g., a neural network) is trained to perform both supervised event predictions and unsupervised time-varying clustering for a sequence of events (e.g., a sequence representing a user behavior) using sequences of events for multiple users using a combined loss function. The unified model can then be used for, given a sequence of events as input, predict a next event to occur after the last event in the sequence and generate a clustering result by performing a clustering operation on the sequence of events. As part of predicting the next event, the unified model is trained to predict an event type for the next event and a time of occurrence for the next event. In certain embodiments, the unified model is a neural network comprising a recurrent neural network (RNN) such as an Long Short Term Memory (LSTM) network.

Unified framework for dynamic clustering and discrete time event prediction

Systems and methods for machine learning are described. Embodiments of the present disclosure receive state information that describes a state of a decision making agent in an environment; compute an action vector from an action embedding space based on the state information using a policy neural network of the decision making agent, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and perform an action that modifies the state of the decision making agent in the environment based on the action vector, wherein the action is selected based on the mapping.

Optimal sequential decision making with changing action space

A control system facilitates active management of a streaming data system. Given historical data traffic for each data stream processed by a streaming data system, the control system uses a machine learning model to predict future data traffic for each data stream. The control system selects a matching between data streams and servers for a future time that minimizes a cost comprising a switching cost and a server imbalance cost based on the predicted data traffic for the future time. In some configurations, the matching is selected using a planning window comprising a number of future time steps dynamically selected based on uncertainty associated with the predicted data traffic. Given the selected matching, the control system may manage the streaming data system by causing data streams to be moved between servers based on the matching.

Active control system for data stream allocation

Techniques are provided for analyzing user actions that have occurred over a time period. The user actions can be, for example, with respect to the user's navigation of content or interaction with an application. Such user data is provided in an action string, which is converted into a highly searchable format. As such, the presence and frequency of particular user actions and patterns of user actions within an action string of a particular user, as well as among multiple action strings of multiple users, are determinable. Subsequences of one or more action strings are identified and both the number of action strings that include a particular subsequence and the frequency that a particular subsequence is present in a given action string are determinable. The conversion involves breaking that string into a sorted list of locations for the actions within that string. Queries can be readily applied against the sorted list.

Determining patterns within a string sequence of user actions

The present disclosure relates to generating and modifying recommended event sequences utilizing a dynamic user preference interface. For example, in one or more embodiments, the system generates a recommended event sequence using a recommendation model trained based on a plurality of historical event sequences. The system then provides, for display via a client device, the recommendation, a plurality of interactive elements for entry of user preferences, and a visual representation of historical event sequences. Upon detecting input of user preferences, the system can modify a reward function of the recommendation model and provide a modified recommended event sequence together with the plurality of interactive elements. In one or more embodiments, as a user enters user preferences, the system additionally modifies the visual representation to display subsets of the plurality of historical event sequences corresponding to the preferences.

Generating digital event recommendation sequences utilizing a dynamic user preference interface

In implementations of item transfer control systems, a computing device implements a transfer system to receive input data describing types of requested items and corresponding quantities of the types of requested items to receive at each of a plurality of destination sites and types of available items and corresponding quantities of the types of available items that are available at each of a plurality of source sites. The transfer system constructs a flow network having a source node for each of the plurality of the source sites and a destination node for each of the plurality of the destination sites. An integral approximate solution is generated that transfers the corresponding quantities of the types of requested items to each of the plurality of the destination sites using a maximum flow solver and the flow network. The transfer system causes transferences of the corresponding quantities of the types of requested items to each of the plurality of the destination sites based on the integral approximate solution.

Item transfer control systems

The present disclosure relates to methods, systems, and non-transitory computer-readable media for determining a cognitive, action-selection bias of a user that influences how the user will select a sequence of digital actions for execution of a task. For example, the disclosed systems can identify, from a digital behavior log of a user, a set of digital action sequences that correspond to a set of sessions for a task previously executed by the user. The disclosed systems can utilize a machine learning model to analyze the set of sessions to generate session weights. The session weights can correspond to an action-selection bias that indicates an extent to which a future session for the task executed by the user is predicted to be influenced by the set of sessions. The disclosed systems can provide a visual indication of the action-selection bias of the user for display on a graphical user interface.

Detecting cognitive biases in interactions with analytics data

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing offline models to warm start online bandit learner models. For example, the disclosed system can determine relevant offline models for an environment based on reward estimate differences between the offline models and the online model. The disclosed system can then utilize the relevant offline models (if any) to select an arm for the environment. The disclosed system can update the online model based on observed rewards for the selected arm. Additionally, the disclosed system can also use entropy reduction of arms to determine the utility of the arms in differentiating relevant and irrelevant offline models. For example, the disclosed system can select an arm based on a combination of the entropy reduction of the arm and the reward estimate for the arm and use the observed reward to update an observation history.

Utilizing relevant offline models to warm start an online bandit learner model

Metric forecasting techniques and systems in a digital medium environment are described that leverage similarity of elements, one to another, in order to generate a forecast value for a metric for a particular element. In one example, training data is received that describes a time series of values of the metric for a plurality of elements. The model is trained to generate the forecast value of the metric, the training using machine learning of a neural network based on the training data. The training includes generating dimensional-transformation data configured to transform the training data into a simplified representation to determine similarity of the plurality of elements, one to another, with respect to the metric over the time series. The training also includes generating model parameters of the neural network based on the simplified representation to generate the forecast value of the metric.

Metric forecasting employing a similarity determination in a digital medium environment

Systems and methods are described for a decision-making process including actions characterized by stochastic availability, provide an Markov decision process (MDP) model that includes a stochastic action set based on the decision-making process, compute a policy function for the MDP model using a policy gradient based at least in part on a function representing the stochasticity of the stochastic action set, identify a probability distribution for one or more actions available at a time period using the policy function, and select an action for the time period based on the probability distribution.

Reinforcement learning with a stochastic action set

Growing community of inventors

San Jose, CA, United States of America

Georgios Theocharous