Nec Corporation

Nec C Orporation

Dotdata, Inc.

Ting Chen

Kazuyo Narita

Ryohei Fujimaki

Takuya Araki

Yukitaka Kusumura

Yusuke Muraoka

Masato Asahara

A table acquiring meansacquires a first table including prediction objects and first attributes, and a second table including second attributes. A receiving meansreceives a similarity function and condition for similarity used to calculate the similarity between the first attribute and the second attribute. A feature generating meansgenerates feature candidates able to affect a prediction object using a combination condition for combining a record in the first table including the value of a first attribute satisfying the condition with a record in the second table including the value of a second attribute satisfying the similarity calculated with the value of the first attribute and the value of the second attribute using the similarity function, and using a reduction method for a plurality of records in the second table and a reduction condition represented by the column to be aggregated. A feature selecting meansselects an optimum feature for the prediction from the feature candidates.

Feature value generation device, feature value generation method, and feature value generation program

One-to-one join is widely used in machine learning and business intelligent applications. Disclosed herein is an efficient method for one-to-one join to reduce memory usage and thus disk I/O accesses with limited memory. Disclosed method outputs and removes a pair of tuples immediately when they are matched to each other to allow join results to be generated without reading entire tables. Meanwhile, disclosed method increases matching rate for in memory blocks through the predication of data distribution patterns based on both statistics and history block matching information.

Method for efficient one-to-one join

To reduce the overall computation time of a batch of queries, multiple query optimization in SQL-on-Hadoop systems groups multiple MapReduce jobs converted from queries into a single one, thus avoiding redundant computations by taking sharing opportunities of data scan, map function and map output. SQL-on-Hadoop converts a query into a DAG of MapReduce jobs and each map function is a part of query plan composed of a sequence of relational operators. As each map function is a part of query plan which is usually complex and heavy, disclosed method creates a cost model to simulate the computation time which takes both I/O cost for reading/writing input file and intermediate data and CPU cost for the computation of map function into consideration. A heuristic algorithm is disclosed to find near-optimal integrated query plan for each group based on an observation that each query plan is locally optimal.

Multiple query optimization in SQL-on-Hadoop systems

Growing community of inventors

Tokyo, Japan

Ting Chen