Comparing datasets using hash values over a subset of fields

Abstract

Relevant content of datasets is compared by computing a hash value over a selected subset of fields to obtain a signature of a dataset, with other fields being disregarded. A hash value can be computed directly for all records of the dataset, or by combining individual hash values for each record. Comparison of the signature with that of other datasets leads to efficient determination whether two datasets match with respect to relevant content in the selected fields. For larger groups of datasets, lists of matched and mismatched datasets can be reported. Optional features include matches insensitive to permutation of the records, or identification of which records in a dataset fail to match.