Scalable pipeline for machine learning-based base-variant grouping

Abstract

A system comprises one or more processors and non-transitory computer-readable media storing computing instructions that, when executed, perform operations comprising: generating an adjacency list for candidate items using a distance threshold with the maximum allowable neighbor distance equal to a Siamese model cut-off. The operations can also comprise loading data for the candidate items in the adjacency list and generating graphs of the candidate items in the adjacency list. The operations further can comprise determining, using breakdown logic, first graphs of the graphs that exceed a predetermined size, and building hierarchy dendrograms of nested subclusters of the first graphs. The operations additional can comprise determining cut-off values based on p-th percentiles of density for the first graphs, and identifying recommended variant groups of the candidate item in the nested subclusters of the hierarchy dendrograms below the cut-off values. Other embodiments are disclosed.