System and method for intent discovery from user logs using deep semi-supervised contrastive clustering

Abstract

Existing semi-supervised and unsupervised approaches for intent discovery require an estimate of the number of new intents present in the user logs. The present disclosure receives labeled utterances from known intents and update parameters of a pre-trained language model (PLM). Representation learning and clustering is performed iteratively using labeled and unlabeled utterances from known intents and unlabeled utterances from unknown intents to fine-tune PLM and a plurality of clusters is generated. Cluster merger algorithm is executed iteratively on generated plurality of clusters. A query cluster is obtained by randomly selecting one cluster from the plurality of clusters and by obtaining a corresponding plurality of nearest neighbors based on a cosine-similarity. A response for merging the query cluster and corresponding plurality of nearest neighbors is obtained, and a new cluster is created. The corresponding cluster representation is recalculated and each of the new cluster is interpreted as an intent.