The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
May. 26, 2026
Filed:
May. 28, 2021
Microsoft Technology Licensing, Llc, Redmond, WA (US);
Andy Wagner, Cupertino, CA (US);
Tiyasa Mitra, San Jose, CA (US);
Fanny Nina Paravecino, San Jose, CA (US);
Microsoft Technology Licensing, LLC, Redmond, WA (US);
Abstract
Embodiments of the present disclosure include systems and methods for training masked language models based on partial sequences of tokens. A sequence of tokens for training a transformer model is received. A defined proportion of the sequence of tokens is selected. Each value of the defined proportion of the sequence of tokens is replaced with a defined value. The transformer model is trained by using the sequence of tokens to train the transformer model during a forward pass and using a subset of the sequence of tokens that includes the defined the proportion of the sequence of tokens to train the transformer model during a backward pass.