International Business Machines Corporation

Svetlana Levitan

Damir Spisic

Jing James Xu

Jing-Yun Shyr

Dong Liang

In each iteration of the process of mining association rules from transaction data by a cluster of computing systems, each mapper node in the cluster receives a split of the transaction data. Each mapper node scans the split to count an absolute support value of each candidate itemset for current search level(s), and passes the candidate itemsets and their support values to reducer nodes in the cluster. The number of reducer nodes will be determined adaptively based on the number of the candidate itemsets and the number of maximum available resource nodes in the cluster. Each reducer node combines the absolute support value of each candidate itemset, and finds frequent itemsets among them using a minimum support threshold. For each frequent itemset it finds, the reducer node creates association rule(s) satisfying a minimum confidence threshold, and exports all discovered frequent itemsets and association rules to a file system for storage.

Mining association rules in the map-reduce framework

One or more processors initiate cluster feature (CF)-tree based hierarchical clustering on leaf entries of CF-trees included in a plurality of subsets. One or more processors, generate respective partial clustering solutions for the subsets. A partial clustering solution includes a set of regular sub-clusters and candidate outlier sub-clusters. One or more processors generate initial regular clusters by performing hierarchical clustering using the regular sub-clusters. For a candidate outlier sub-cluster, one or more processors determine a closest initial regular cluster, and a distance separating the candidate outlier sub-cluster and the closest initial regular cluster. One or more processors determine which candidate outlier sub-clusters are outlier clusters based on which candidate outlier sub-clusters have a computed distance to their respective closest initial regular cluster that is greater than a corresponding distance threshold associated with their respective closest initial regular cluster.

Distributed clustering with outlier detection

Embodiments of the present invention provide efficient systems and methods for processing large data sets using a composite function. Embodiments of the present invention can be used to compute a broad range of composite functions in a single map-reduce job. Each mapper computes an additive function G on a set of specified data partitions, and then passes the results to one or more reducers. The one or more reducers can then compute a function F, using the aggregate results of function G and data from a single partition.

Growing community of inventors

Skokie, IL, United States of America

Svetlana Levitan