System and method for automatically extracting latent structures and relationships in datasets

Abstract

The system and method provide for automatically extracting latent structures and relationships in datasets of any size, to enable effective management of that data. The system and method produce hierarchies of fields with relationships of a particular type and strength. The hierarchies of fields with relationships of a particular type and strength can be used to optimize the data storage profile of a company by separating large datasets into multiple smaller datasets, reducing operational costs, perform enhanced analysis of the data through a greater understanding of its internal logical structure, reduce privacy risk within the dataset by removing risky fields and substituting them in analytical processing with less risky fields that carry a similar amount of statistical information, to identify inference risks in the data to enable more effective data protection decision-making, and can be used to separate out wide datasets into multiple narrower/smaller datasets, thus aiding more efficient analytics.