International Business Machines Corporation

Dong Liang

Jing-Yun Shyr

Yea Jane Chu

Svetlana Levitan

In each iteration of the process of mining association rules from transaction data by a cluster of computing systems, each mapper node in the cluster receives a split of the transaction data. Each mapper node scans the split to count an absolute support value of each candidate itemset for current search level(s), and passes the candidate itemsets and their support values to reducer nodes in the cluster. The number of reducer nodes will be determined adaptively based on the number of the candidate itemsets and the number of maximum available resource nodes in the cluster. Each reducer node combines the absolute support value of each candidate itemset, and finds frequent itemsets among them using a minimum support threshold. For each frequent itemset it finds, the reducer node creates association rule(s) satisfying a minimum confidence threshold, and exports all discovered frequent itemsets and association rules to a file system for storage.

Mining association rules in the map-reduce framework

Provided are techniques for computing a task result. A processing data set of records is created, wherein each of the records contains data specific to a sub-task from a set of actual sub-tasks and contains a reference to data shared by the set of actual sub-tasks, and wherein a number of the records is equivalent to a number of the actual sub-tasks in the set of actual sub-tasks. With each mapper in a set of mappers, one of the records of the processing data set is received and an assigned sub-task is executed using the received one of the records to generate output. With a single reducer, the output from each mapper in the set of mappers is reduced to determine a task result.

Growing community of inventors

Shaanxi, China

Dong Liang