Servicenow, Inc.

Sagar Davasam Suryanarayan

Edwin Vito Sapugay

Anil Kumar Madamala

Maxim Naboka

Omer Anil Turkkan

Aniruddha Madhusudan Thakur

Sriram Palapudi

Jeevan Anand Anne

Rammohan Narendula

Edward Cheung

Vipulkumar Popat Mahadik

An embodiment may involve: obtaining textual content including a plurality of token strings, wherein each of the plurality of token strings includes one or more tokens; determining, for the plurality of token strings, respectively corresponding sets of n-gram tuples; assigning respective weights to the plurality of token strings, wherein, for each of the plurality of token strings, the assignment is based on the respectively corresponding set of n-gram tuples; identifying a subset of the plurality of token strings, wherein each of the subset of the plurality of token strings is characterized by a respective weight that exceeds a predetermined threshold weight; and storing sets of n-gram tuples respectively corresponding to the subset of the plurality of token strings.

Language model preprocessing with weighted n-grams

A natural language understanding (NLU) framework includes a lookup source framework, which enables a lookup source system to be defined having one or more lookup sources. The lookup source system can operate in a number of different manners to facilitate repository-aware inference of user utterances, for example, by facilitating vocabulary injection during compilation of an utterance meaning model and/or an understanding model. Additionally, the lookup source system can be leveraged to cleanse client-specific training data of sensitive values to generate generic training data that can be used to train the NLU framework of other clients. The lookup sources can be compiled in a synchronous or asynchronous manner, which enables lookup sources to be compiled in an on-demand basis from test source data. Additionally, understanding models that reference lookup sources can be periodically recompiled while leveraging the latest versions of the lookup sources for vocabulary injection.

System and method for repository-aware natural language understanding (NLU) using a lookup source framework

A natural language understanding (NLU) framework includes a lookup source framework, which enables a lookup source system to be defined having one or more lookup sources. Each lookup source of the lookup source system includes a respective source data representation that is compiled from respective source data. For example, a source data representation may include source data arranged in a finite state transducer (IFST) structure as a set of finite-state automata (FSA) states, wherein each state is associated with a token that represents underlying source data. Different producers can be applied during compilation of a source data representation to derive additional states within the source data representation from the source data. Certain states of the source data representation that contain sensitive data can be selectively protected through encryption and/or obfuscation, while other portions of the source data representation that are not sensitive may remain in clear-text form.

Lookup source framework for a natural language understanding (NLU) framework

Growing community of inventors

Santa Clara, CA, United States of America

Sagar Davasam Suryanarayan