Microsoft Technology Licensing, LLC

Ya Xu

Weitao Duan

Nanyu Chen

Bryan G Ng

Bo Liu

Ryan M Rogers

Ron Kohavi

Maneesh Varshney

Kylan Matthew Nieh

Noor-E-Gagan Singh

Sean S Peng

David Anthony Durfee

Guillaume B Saint-Jacques

Alexander Ivaniuk

Jeremy John Simpson

Ruirui Xiang

The disclosed embodiments provide a system for performing A/B testing of service-level metrics. During operation, the system obtains service-level metrics for service calls made during an A/B test, wherein the service-level metrics are aggregated by user identifiers of multiple users. Next, the system matches the service-level metrics to treatment assignments of the users to a treatment group and a control group in the A/B test. The system then applies the A/B test to a first grouping of the service-level metrics for the treatment group and a second grouping of the service-level metrics for the control group. Finally, the system outputs a result of the A/B test for use in assessing an effect of a treatment variant in the A/B test on the service-level metrics.

A/B testing of service-level metrics

Techniques for ensuring differential privacy in top-K selection are provided. In one technique, multiple items and multiple counts are identified in response to a query. For each count, which corresponds to a different item, a noise value is generated and added to the count to generate a noisy value, and the noisy value is added to a set of noisy values that is initially empty. A particular noise value is generated for a particular count and added to the particular count to generate a noisy threshold. The particular noise value is generated using a different technique than the technique used to generate each noise value in the set. Based on the noisy threshold, a subset of the noisy values is identified, where each noisy value in the subset is less than the noisy threshold. A response to the query is generated that excludes items that correspond to the subset.

Differentially private top-k selection

During operation, a system obtains a graph that includes a set of nodes representing users in an online network and a set of edges representing relationships or interactions between pairs of the nodes. Next, the system selects egos containing a subset of the users that are representative of a distribution of an attribute across the users. The system then generates clusters of users, such that each cluster contains an ego and alters selected from the ego's connections in the graph. The system further divides the clusters between a treatment group and a control group, exposes alters in the treatment group to a treatment variant of an A/B test, and exposes alters in the control group to a control variant of the A/B test. Finally, the system outputs a result of the A/B test.

A/B testing using ego network clusters

A machine may be configured to perform A/B testing on customized experiment units. For example, the machine receives a specification of an experiment unit that identifies a type of subject of an experiment for execution on a social networking service (SNS), and a value of the experiment unit. The machine generates, for the value of the experiment unit, tracking data that tracks user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment. The machine generates, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit. The machine generates an experiment report based on the tracking data and the metric data. The machine causes a presentation of the experiment report in a user interface of a client device.

Flexible units for experimentation

The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Model-based matching for removing selection bias in quasi-experimental testing of mobile applications

The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Model validation and bias removal in quasi-experimental testing of mobile applications

Enhancements of the user experience are provided when a user returns to a previously viewed page, such as a previously viewed page of search results. When a user returns to a previously viewed page, additional context information from a user's actions since the initial view of a page can be used to modify the previously viewed page and/or obtain a new version of the previously viewed page. In situations where the previously viewed page corresponds to a page of responsive results from a search engine, the modified and/or new version of the search engine results page can include an expanded or reduced group of results, different types of results, different rankings for existing results, or a combination thereof.

Growing community of inventors

Los Altos, CA, United States of America

Ya Xu