The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
Dec. 13, 2022
Filed:
Jun. 24, 2020
Microsoft Technology Licensing, Llc, Redmond, WA (US);
Pengcheng He, Sammamish, WA (US);
Xiaodong Liu, Redmond, WA (US);
Jianfeng Gao, Woodinville, WA (US);
Weizhu Chen, Kirkland, WA (US);
Microsoft Technology Licensing, LLC, Redmond, WA (US);
Abstract
Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.