Dengyong Zhou

Title: Innovations by Dengyong Zhou in Language Models and Forecasting

Introduction

Dengyong Zhou, a prolific inventor based in Redmond, WA, has made significant contributions to the fields of language model compression and forecasting. With a total of 13 patents to his name, his work is characterized by innovative techniques that enhance the efficiency and accuracy of machine learning models.

Latest Patents

Among his most recent patents, "Extreme language model compression with optimal sub-words and shared projections" stands out. This patent introduces a knowledge distillation technique for training a student language model that, when compared to a larger teacher model, boasts a considerably smaller vocabulary and reduced dimensions in both embedding and hidden states. The technique employs a dual-training mechanism that allows the simultaneous training of both teacher and student models, leading to optimal word embeddings for the student vocabulary. This method is particularly powerful, achieving compression of the BERT model by over 60 times, all while maintaining a minimal drop in downstream task metrics, resulting in a language model with a footprint under 7 MB.

Another notable patent is the "Machine-learned state space model for joint forecasting." This invention enhances a deep state space generative model by incorporating intervention prediction. It provides a structured approach to understanding the interactions between observations, interventions, critical events, and uncertainties. The model outputs joint predictions of event risks and trajectories by analyzing the patterns in temporal progressions, making it a valuable tool in various predictive applications.

Career Highlights

Dengyong Zhou has had a distinguished career, working at prominent technology firms including Microsoft Technology Licensing, LLC and Google Inc. His role in these companies has facilitated his research and development in cutting-edge technologies, leading to his numerous patents.

Collaborations

Zhou’s innovative journey has been enriched through collaborations with esteemed colleagues, such as Christopher J C Burges and Tao Tao. These partnerships have allowed for a fruitful exchange of ideas and methodologies, pushing the boundaries of what is possible in their fields.

Conclusion

Dengyong Zhou’s contributions to machine learning and language model optimization reflect his deep understanding and commitment to innovation. His patents not only demonstrate his expertise but also pave the way for future advancements in technology, proving invaluable to the growing landscape of artificial intelligence and data forecasting.