Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Sebastian Kohlmeier

Alexandru Burciu

Alexandru Tronciu

Aurel Dumitrascu

Rui Chen

Krishnan Thazhathekalam

Daniel R Parshall

Sameer R Rajyaguru

Serguei B Stepaniants

Michael Luis Collado

Jorge Banuelos

Mahaveer Bhavarlal Kothari

Kevin Donald Kelly

Anjali Muralidhar

Kyle Leon Battisti

Rusli Ciandy

Kevin McAlister

Akshay Gandhi

James S Luke

Khalid Mahmood

A machine may be configured to assign digital content campaign treatments to targets of a digital content campaign, at a server. For example, the machine access data pertaining to one or more characteristics of an application. The application is hosted on a client device. The data is stored at a server-side database. The machine determines that the one or more characteristics of the application match one or more targeting criteria associated with a digital content campaign. The digital content campaign includes a plurality of flights of a digital content item. The machine assigns a flight of the plurality of flights of the digital content campaign to the client device. The flight identifies a variant of the digital content item. The machine transmits a communication to the client device via a network. The communication includes a flight identifier associated with the flight.

Running client experiments based on server-side user segment data

Systems and methods are disclosed which facilitate management and monitoring of page-level usage data. Specifically, a client computing device may retrieve content from a remote system for local display. A local interaction monitoring component may monitor local interaction with the content, and transmit local interaction information to the remote system. In one embodiment, interaction may be monitored with regard to predefined portions of the content. In another embodiment, interaction may be transmitted to the remote server for processing, in order to generate page-level usage data. In still more embodiments, page level-usage data may be processed or formatted in order to facilitate ease of retrieval and display. In yet more embodiments, the remote system may enable clients, such as administrators, to view aggregate page-level usage data why viewing a corresponding content.

Managing page-level usage data

Described are techniques for determining long-term effects of an experimental change to a user experience after the end of the experiment. A control state and a treatment state of a statistical hypothesis experiment may be assigned to first and second client devices, respectively, during an experiment time period. Subsequent to the end of the experiment, presentation of the control state may be discontinued. Result data corresponding to the treatment state may be determined during the experiment time period and for a length of time subsequent to the experiment time period. Result data corresponding to the control state may be determined during the experiment time period, and for a length of time prior to assignment of the control state to the first client device.

Systems for determining long-term effects in statistical hypothesis testing

In some embodiments, the disclosed subject matter involves a system and method relating to automation test runs of a software build, where the test runs effect impersonating an automation audience and selecting optional features to test with the automation audience. The automation audience is associated with a software build having static features, and the test runs include optional features, where each optional feature may have more than one associated treatment. Each feature/treatment combination may be tested in a test scenario associated with the feature/treatment combination. New features that are exposed in other audiences may be enabled for the automation audience to assist in verification and test of the new features. Dynamic selection of the optional features is effected using an application program interface (API) in the source code for the features, where the API accesses feature configuration information during runtime. Other embodiments are described and claimed.

Feature targeting of test automation lab machines

Described are techniques for performing statistical hypothesis experiments and determining user responses that do not match an assigned treatment. One of multiple experiment states may be assigned to a set of client devices. Response data that includes indications of the experiment state that was executed may be compared to the assigned experiment state to determine allocation matches and allocation mismatches. The allocation mismatches may be separated based on the assigned and executed experiment states to determine mismatches caused by preexisting content stored on a client device and mismatches caused by other experiment parameters.

Systems for determining improper assignments in statistical hypothesis testing

Growing community of inventors

Mountlake Terrace, WA, United States of America

Sebastian Kohlmeier