Microsoft Technology Licensing, LLC

Cenzhuo Yao

Neelakantan Sundaresan

Xi Cheng

Jorge Banuelos

Kexin Xu

A dynamic experimentation evaluation system provides a framework in which a continuous stream of metric data is monitored to establish a causal relationship between changes in a software program and the effect of user-observable behavior. In one aspect, an A/B test is performed continuously on a stream of metric data representing the usage of a control version of software product and the usage of a treatment version of the software product. A sequential probability ratio test (SPRT) is used as the test statistic to determine when to terminate the test and produce results within a specific confidence interval and controlled error rate.

Dynamic experimentation evaluation system

Machine learning techniques are used to determine the viability of user data measuring the behavior of a new version of the program when compared with user data that measured the behavior of a previous version of the program. The machine learning techniques utilize statistical techniques in a non-conventional manner to train a system to learn from data obtained from the usage of both a new version of the program and a previous version that accounts for the variability in the user population, time variability of the results of the previous version, and feature coverage between the two test results in order to ensure the suitability of the user data in making estimations or predictions about the performance and reliability of the new version.

Growing community of inventors

Bellevue, WA, United States of America

Cenzhuo Yao