The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
Oct. 31, 2023
Filed:
May. 22, 2020
Microsoft Technology Licensing, Llc, Redmond, WA (US);
Xiaodong Liu, Redmond, WA (US);
Hao Cheng, Bellevue, WA (US);
Yu Wang, Bellevue, WA (US);
Jianfeng Gao, Woodinville, WA (US);
Weizhu Chen, Kirkland, WA (US);
Pengcheng He, Sammamish, WA (US);
Hoifung Poon, Bellevue, WA (US);
Microsoft Technology Licensing, LLC, Redmond, WA (US);
Abstract
This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.