Microsoft Technology Licensing, LLC

Weitao Duan

Ya Xu

Bryan G Ng

Nanyu Chen

Bo Liu

Kylan Matthew Nieh

A machine may be configured to perform A/B testing on customized experiment units. For example, the machine receives a specification of an experiment unit that identifies a type of subject of an experiment for execution on a social networking service (SNS), and a value of the experiment unit. The machine generates, for the value of the experiment unit, tracking data that tracks user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment. The machine generates, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit. The machine generates an experiment report based on the tracking data and the metric data. The machine causes a presentation of the experiment report in a user interface of a client device.

Flexible units for experimentation

The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Model-based matching for removing selection bias in quasi-experimental testing of mobile applications

The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Growing community of inventors

Mountain View, CA, United States of America

Weitao Duan