Machine learning system for multi-domain long document clustering

Abstract

A model pipeline has been created that allows an organization to gain a clear understanding of contents of a collection of documents despite varying lengths and multiple domain content. A collection of documents with content of different domains are normalized by summarizing the documents. For summarizing, the pipeline uses a language model that has been trained for text summarization across domains to a constrained summary length or length limit. The pipeline extracts embeddings of the summaries and clusters the summary embeddings. The pipeline uses the summaries to label the clusters using another language model that has been trained to generate a label or name for a cluster based on a set of summaries corresponding to a sample of cluster members. The labeled clusters can then be used to generate an organized presentation of the content of the documents in the document collection.