Adobe, Inc.

Arize Ai, Inc.

Tubemogul, Inc.

Jason Lopatecki

John Hughes

Adam Rose

Brett Wilson

Aparna Dhinakaran

Alexander R Hood

Greg Collison

John M Trenkle

Justin K Sung

David Innes-Gawn

Kevin Thakkar

Michael Schiff

Francisco Castillo Carrasco

Boaz Ram

Christopher Cleophus Bell

Julie Lee

Nathaniel Mar

Thomas Riordan

Narayan Kinhal

Darren Sue

Matthew Ellinwood

A system for optimizing a machine learning model. The machine learning model generates predictions based on at least one input feature vector, each input feature vector having one or more vector values; and an optimization module with a processor and an associated memory, the optimization module being configured to: create at least one slice of the predictions based on at least one vector value, determine at least one optimization metric of the slice that is based on at least a total number of predictions for the vector value, and optimize the machine learning model based on the optimization metric.

Systems and methods for optimizing a machine learning model

Techniques for determining a drift impact score in a machine learning model are disclosed. The techniques can include: obtaining a reference distribution of a machine learning model; obtaining a current distribution of the machine learning model; determining a statistical distance based on the reference distribution and the current distribution; determining a local feature importance parameter for each feature associated with a prediction made by the machine learning model; determining a cohort feature importance parameter for a cohort of multiple features based on the local feature importance parameter of each feature in the cohort; and determining a drift impact score for the cohort based on the statistical distance and the cohort feature importance parameter.

Data drift impact in a machine learning model

Techniques for optimizing a machine learning model. The techniques can include: obtaining one or more embedding vectors based on a prediction of a machine learning model; mapping the embedding vectors from a higher dimensional space to a 2D/3D space to generate one or more high density points in the 2D/3D space; clustering the high-density points by running a clustering algorithm multiple times, each time with a different set of parameters to generate one or more clusters; applying a purity metric to each cluster to generate a normalized purity score of each cluster; identifying one or more clusters with a normalized purity score lower than a threshold; and optimizing the identifying one or more clusters.

Optimizing machine learning based on embedding smart data drift

Techniques for determining embedding drift score in a machine learning model. The techniques can include: obtaining one or more first embedding vectors based on at least one first prediction of a machine learning model; filtering the first embedding vectors based on a slice of the first prediction; determining a first average vector by averaging each dimension of the filtered first embedding vectors; obtaining one or more second embedding vectors on at least one second prediction of the machine learning model; filtering the second embedding vectors based on a slice of the second prediction; generating a second average vector by averaging each dimension of the filtered second embedding vectors; and determining an embedding drift score based on a distance measure of the first average vector and the second average vector.

Embedding drift in a machine learning model

Systems and methods are disclosed for characterizing websites and viewers, for predicting GRPs (Gross Rating Points) for online advertising media campaigns, and for pricing media campaigns according to GRPs delivered as opposed to impressions delivered. To predict GRPs for a campaign, systems and methods are disclosed for first characterizing polarized websites and then characterizing polarized viewers. To accomplish this, a truth set of viewers with known characteristics is first established and then compared with historic and current media viewing activity to determine a degree of polarity for different Media Properties (MPs)&#x2014;typically websites offering ads&#x2014;with respect to gender and age bias. A broader base of polarized viewers is then characterized for age and gender bias, and their propensity to visit a polarized MP is rated. Based on observed and calculated parameters, a GRP total is then predicted and priced to a client/advertiser for an online ad campaign.

Systems and methods for predicting and pricing of gross rating point scores by modeling viewer data

Systems and methods are disclosed for conducting media lift studies for online advertising concurrently with operating an advertising campaign. While operating an advertising campaign for a first advertiser/client focused primarily on a set of intended ads and a specific targeted viewer audience, a non-intended ad is occasionally substituted to run in an ad slot, and is tracked as belonging to the first advertiser/client. The non-intended ad can be for example one of: an ad for a second advertiser/client; an alternate ad for the first advertiser/client; or a blank/unrelated ad. After the campaign, attribution results for the intended ads are adjusted according to those for non-intended ads to provide an indication of net media lift resulting from the intended ads&#x2014;typically at no additional cost to the first advertiser/client. Analysis results may also be compared between different attribution data providers to determine which provide the more accurate attribution data.

Conducting dynamic media lift studies concurrently with operating online advertising campaigns

Systems and methods are disclosed for employing supervised machine learning methods with activities and attributes of viewers with truth as input, to produce models that are utilized in determining probabilities that an unknown viewer belongs to one or more demographic segment categories. Using these models for processing viewer behavior, over a period of time a database of known categorized viewers is established, each categorized viewer having a probability of belonging to one or more segment categories. These probabilities are then used in bidding for online advertisements in response to impression opportunities offered in online media auctions. The probabilities are also used in predicting on-target impressions and GRPs (Gross Rating Points) in advance of online advertising media campaigns, and pricing those campaigns to advertiser/clients. Strategies are also disclosed for fulfilling a campaign when an available inventory of known categorized viewers is not adequate to fulfill a campaign in a required runtime.

Methods for viewer modeling and bidding in an online advertising campaign

Systems and methods are disclosed for optimizing an online advertising campaign both before the campaign begins, and dynamically during the campaign. Optimizations are performed comparatively between a plurality of MPs (Media Properties) based on their relative cost-per-engagement. Comparisons are performed by first stack ranking MP inventory including any of sites, feeds, and verticals, based on cost per engagement. Once ranked, scores are assigned to the targeted inventory and a mean score is determined. Then, the inventory is rated as high, normal, or low impact based on their scores compared with the mean and a standard deviation for all scores. Higher impact sites with scores at least a standard deviation above the mean are initially favored, and the MP targeting strategy is dynamically adjusted during the campaign based on periodically re-evaluating the MP rankings, frequencies of engagement, and campaign progress relative to fulfillment in an allotted run time.

Real-time bidding system that achieves desirable cost per engagement

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a plurality of recurrent neural networks to generate media consumption predictions and providing media content to a target audience. For example, the disclosed system can train a plurality of long short-term memory neural networks for a plurality of users based on historical media consumption data over a plurality of time periods. In one or more embodiments, the disclosed system identifies a target audience including a subset of users and the corresponding neural networks. The disclosed system can then utilize the neural networks of the subset of users to generate a plurality of predictions for a future time period for the users. In some embodiments, the disclosed system then combines the predictions for the users to generate a media consumption prediction for the target audience for the future time period.

Utilizing combined outputs of a plurality of recurrent neural networks to generate media consumption predictions

The present disclosure relates to a content campaign system that improves the design and implementation of content campaigns. In particular, the content campaign system can receive television viewer information corresponding to television client devices and online activity information corresponding to client computing devices. Further, the content campaign system can identify a correspondence between the television viewer information and the online activity information for individual users and/or households. Based on the correspondence, the content campaign system can automatically generate targeting parameters for audiovisual content campaigns. For example, the content campaign system can recommend audiovisual content (e.g., a television advertisement) to provide to a target audience of users via a television broadcast based on the correlated television viewer information and online activity information of a particular user.

Digital audiovisual content campaigns using merged television viewer information and online activity information

Systems and methods for operating placebo-based experiments are described for online advertisements. One or more embodiments of the disclosed systems and methods utilizes an ad swapping approach to offer placebo media exposures, at no additional cost to an advertiser. One or more embodiments further provide a native experimentation platform that allows users to run tests of ad placements to measure the effectiveness of ads and view results displayed on a user interface. The disclosed systems and methods can assign viewers into a test group if shown the test ad or a control group if shown a control ad. The control ad can be provided at no cost to the advertiser for embodiments where the placebo ad belongs to an alternative advertiser. Effectiveness of third party attribution can also be evaluated. The disclosed systems and methods can define experiment parameters, including control frequency, test viewer groups, and control viewer groups.

Real-time bidding through placebo-based experimentation

Systems and methods are disclosed for planning, executing, reviewing, and reporting the results of an advertising campaign to be run on TV. A demand-side platform receives ad slot opportunities from TV programming sources, and analyzes the ad slots to produce a prioritized list of placement opportunities for the advertising campaign to be presented to advertiser/clients. Each ad slot is analyzed with respect to past viewership data and with respect to desired targeting characteristics that may include conventional age and gender targeting, or additionally strategic targeting characteristics. Scores are established for each ad slot with respect to numbers of projected on-target impressions and/or a cost for projected on-target impressions. The scores are sorted to produce the prioritized list. Projected results can be viewed with respect to any or all of network, day, and daypart. After a campaign has completed, viewership data representing actual results is acquired, processed, and reported.

Planning and executing a strategic advertising campaign

Presentation of a video clip is made to persons having a high probability of viewing the clip. A database containing viewers of previously offered video clips is analyzed to determine similarities of preferences among viewers. When a new video clip has been watched by one or more viewers in the database, those viewers who have watched the new clip with positive results are compared with others in the database who have not yet seen it. Prospective viewers with similar preferences are identified as high likelihood candidates to watch the new clip when presented. Bids to offer the clip are based on the degree of likelihood. For one embodiment, a data collection agent (DCA) is loaded to a player and/or to a web page to collect viewing and behavior information to determine viewer preferences. Viewer behavior may be monitored passively by different disclosed methods.

Growing community of inventors

San Francisco, CA, United States of America

Jason Lopatecki