Memory usage reduction for disparate data source aggregation

Abstract

Techniques are described herein a system accesses undefined string objects from a first, externally generated data store and defined string objects from a distinct second data store. It calculates similarity scores between each undefined and defined string object using a first function, producing a first ordered array of similarity scores with a reduced memory footprint compared to the original undefined objects. A second function processes this array to form candidate pairs, resulting in a second ordered array with an even smaller memory footprint. For each candidate pair with a similarity score meeting a predetermined threshold, the system transforms the associated undefined string objects into new defined string objects based on the defined string objects. The second data store is then updated with these new defined string objects.