The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
Feb. 11, 2025
Filed:
Jun. 10, 2022
Salesforce, Inc., San Francisco, CA (US);
Philippe Laban, New York, NY (US);
Chien-Sheng Wu, Mountain View, CA (US);
Wenhao Liu, Redwood City, CA (US);
Caiming Xiong, Menlo Park, CA (US);
Salesforce, Inc., San Francisco, CA (US);
Abstract
Embodiments described herein provide a method of evaluating a natural language processing model. The method includes receiving an evaluation dataset that may include a plurality of unit tests, the unit tests having: an input context, and a first candidate and a second candidate that are generated in response to the input context, where the first test candidate is associated with a first quality notation, and the second candidate is associated with a second quality notation. The method includes determining, via a model, a first likelihood of generating the first candidate and a second likelihood of generating the second candidate in response to the input context. The method also includes determining whether the first likelihood being greater than the second likelihood. The method also includes determining whether the first model passed the unit test, where the first quality notation indicates a higher quality candidate and the second quality notation indicate a lower quality candidate.