International Business Machines Corporation

Saloni Potdar

Ming Tan

Haoyu Wang

Yang Yu

Haode Qi

Ladislav Kunc

Navneet N Rao

Lakshminarayanan Krishnamurthy

Inkit Padhi

Ruijian Wang

Dakuo Wang

Abhishek Shah

Lin Pan

Panos Karagiannis

Ravi Nair

Chuang Gan

Robert Leslie Yates

Shiyu Chang

Mo Yu

Eric Donald Wayne

Shasha Lin

Cheng Qian

Ladislav Kune

A method, a structure, and a computer system for OOD sentence detection in dialogue systems. The exemplary embodiments may include receiving, for a domain corresponding to a particular topic, one or more on-topic text inputs and one or more off-topic text inputs. The exemplary embodiments may further include encoding the one or more on-topic text inputs and the one or more off-topic text inputs into a latent space, as well as decoding the one or more on-topic text inputs and the one or more off-topic text inputs from the latent space. The exemplary embodiments may additionally include minimizing a reconstruction error between the encoded one or more on-topic text inputs and the decoded one or more on-topic text inputs, and maximizing a reconstruction error between the encoded one or more off-topic text inputs and the decoded one or more off-topic text inputs.

Out of domain sentence detection

Mechanisms are provided for performing an automated machine learning (AutoML) operation to configure parameters of a machine learning model. AutoML logic is configured based on an initial parameter sampling configuration for sampling values of parameter(s) of the machine learning (ML) model. An initial AutoML process is executed on the ML model based on a dataset utilizing the initially configured AutoML logic, to generate at least one learned value for the parameter(s) of the ML model. The dataset is analyzed to extract a set of dataset characteristics that define properties of a format and/or a content of the dataset which are stored in association with the at least one learned value as part of a training dataset. A ML prediction model is trained based on the training dataset to predict, for new datasets, corresponding new sampling configuration information based on characteristics of the new datasets.

Learning parameter sampling configuration for automated machine learning

A system for classifying a language sample intent by receiving a language sample including a set of features, identifying language sample features, determining a tokenization score for the language sample according to the language sample features, eliminating duplicate features according to the tokenization score, determining a term frequency (tf) according to the identified features and the tokenization score, determining an inverse document frequency (idf) according to the identified features and the tokenization score, and generating a term frequency-inverse document frequency (tf-idf) matrix for the identified features.

Intent classification using non-correlated features

A method, computer system, and computer program product for multi-lingual chatlog training are provided. The embodiment may include receiving, by a processor, a plurality of data related to conversational data in multiple languages. The embodiment may also include assigning an intent label to each conversational data. The embodiment may further include assigning a language label to each conversational data. The embodiment may also include paring the plurality of the data related to the conversational data according to the intent label and the language label. The embodiment may further include training a machine learning model using a multi-lingual and multi-intent conversational data pairing. The embodiment may also include training the machine learning model using a single language and multi-intent conversational data paring.

Conversational AI with multi-lingual human chatlogs

A computer-implemented method includes using an embedding network to generate prototypical vectors. Each prototypical vector is based on a corresponding label associated with a first domain. The computer-implemented method also includes using the embedding network to generate an in-domain test vector based on at least one data sample from a particular label associated with the first domain and using the embedding network to generate an out-of-domain test vector based on at least one other data sample associated with a different domain. The computer-implemented method also includes comparing the prototypical vectors to the in-domain test vector to generate in-domain comparison values and comparing the prototypical vectors to the out-of-domain test vector to generate out-of-domain comparison values. The computer-implemented method also includes modifying, based on the in-domain comparison values and the out-of-domain comparison values, one or more parameters of the embedding network.

Out-of-domain encoder training

Domain specific model compression by providing a weighting parameter for a candidate operation of a neural network, applying the weighting parameter to an output vector of the candidate operation, performing a regularization of the weighting parameter output vector combination, compressing the neural network model according to the results of the regularization, and providing the neural network model after compression.

Domain specific model compression

In response to running at least one testing phrase on a previously trained text classifier and identifying a separate predicted classification label based on a score calculated for each respective at least one testing phrase, a text classifier decomposes extracted features summed in the score into word-level scores for each word in the at least one testing phrase. The text classifier assigns a separate heatmap value to each of the word-level scores, each respective separate heatmap value reflecting a weight of each word-level score. The text classifier outputs the separate predicted classification label and each separate heatmap value reflecting the weight of each word-level score for defining a heatmap identifying the contribution of each word in the at least one testing phrase to the separate predicted classification label for facilitating client evaluation of text classification anomalies.

Evaluating text classification anomalies predicted by a text classification model

A system includes a memory having instructions therein and at least one processor configured to execute the instructions to: receive a natural language question; determine, from a chat log comprising a plurality of chat session logs, a set of chat session logs most relevant to the natural language question; determine a respective plurality of non-overlapping text spans most relevant to the natural language question within each of a respective plurality of conceptual pseudo-documents; determine a conceptual pseudo-document most relevant to the natural language question; extract a question-answer pair most relevant to the natural language question from the most relevant pseudo-document; and convey the most relevant question-answer pair to a user. Each one of the conceptual pseudo-documents corresponds to a respective one of the most relevant chat session logs.

Contextual question answering using human chat logs

A method includes determining, based on an input data sample, a set of probabilities. Each probability of the set of probabilities is associated with a respective label of a set of labels. A particular probability associated with a particular label indicates an estimated likelihood that the input data sample is associated with the particular label. The method includes modifying the set of probabilities based on a set of adjustment factors to generate a modified set of probabilities. The set of adjustment factors is based on a first relative frequency distribution and a second relative frequency distribution. The first relative frequency distribution indicates for each label of the set of labels, a frequency of occurrence of the label among training data. The second relative frequency distribution indicates for each label of the set of labels, a frequency of occurrence of the label among post-training data provided to the trained classifier.

Intent classification distribution calibration

Mechanisms are provided for optimizing an automated machine learning (AutoML) operation to configure parameters of a machine learning model. AutoML logic is configured based on an initial default value and initial range for sampling of a parameter of the machine learning (ML) model and an initial AutoML process is executed on the ML model based on a plurality of datasets comprising a plurality of domains of data elements, utilizing the initially configured AutoML logic. For each domain, a cross-dataset default value and cross-dataset value range are derived from results of the execution of the initial AutoML process. For each domain, an entry is stored in a data structure, the entry storing the derived cross-dataset default value and cross-dataset value range for the domain. The AutoML logic performs a subsequent AutoML process on a new dataset based on one or more entries of the data structure.

Mechanisms for continuous improvement of automated machine learning

A mechanism is provided to implement an abnormal entity detection mechanism that facilitates detecting abnormal entities in real-time response systems through weak supervision. For each first intent from an entity labeled workspace that matches a second intent in labeled chat logs, when the entity score associated with each first entity or second entity is above a predefined significance level the first entity or the second entity is recorded. For each first intent from the entity labeled workspace that matches the second intent in the labeled chat logs: responsive to the first entity being recorded and the second entity failing to be recorded, that first entity is removed from the training data as being mistakenly included; or, responsive to the second entity being recorded and the first entity failing to be recorded, that second entity is added as a potential business case to the training data.

Weak supervised abnormal entity detection

A mechanism is provided to implement suggestion of new entity types with discriminative importance analysis. The mechanism obtains a list of predefined intents from a chatbot designer. The mechanism receives an input sentence having a target intent within the list of predefined intents. The mechanism performs intent-specific importance analysis on the input sentence to generate an importance score for each token in the input sentence. The mechanism ranks the tokens in the input sentence by importance score and outputs a token with a highest importance score as a candidate entity type.

Suggestion of new entity types with discriminative term importance analysis

A method of text classification includes generating a text embedding vector representing a text sample and applying weights of a regression layer to the text embedding vector to generate a first data model output vector. The method also includes generating a plurality of prototype embedding vectors associated with a respective classification labels and comparing the plurality of prototype embedding vectors to the text embedding vector to generate a second data model output vector. The method further includes assigning a particular classification label to the text sample based on the first data model output vector, the second data model output vector, and one or more weighting values.

Hybrid model for short text classification with imbalanced data

A mechanism is provided for implementing an intent segmentation mechanism that segments intent boundaries for multi-intent utterances in a conversational agent. For each term of a set of terms in the utterance from a real-time chat session, a set of adversarial utterances is generated for the utterance. An influence of changing each term is determined so as to identify a term importance value. Utilizing the term importance value, one or more of a change in ranking of the intent of the utterance or a change in confidence with regard to the intent of the utterance is identified. An entropy-based segmentation of the utterance into a plurality of candidate partitions is performed. An associated intent and entropy value are then assigned. Based on a segment with minimum entropy, a call associated with the real-time chat session is directed to an operation associated with an intent of the segment with minimum entropy.

Intent boundary segmentation for multi-intent utterances

Provided is a method, system, and computer program product for context-dependent spellchecking. The method comprises receiving context data to be used in spell checking. The method further comprises receiving a user input. The method further comprises identifying an out-of-vocabulary (OOV) word in the user input. An initial suggestion pool of candidate words is identified based, at least in part, on the context data. The method then comprises using a noisy channel approach to evaluate a probability that one or more of the candidate words of the initial suggestion pool is an intended word and should be used as a candidate for replacement of the OOV word. The method further comprises selecting one or more candidate words for replacement of the OOV word. The method further comprises outputting the one or more candidates.

Growing community of inventors

Arlington, MA, United States of America

Saloni Potdar