International Business Machines Corporation

Qiang Xu

Zhenhua Hu

Hao Hai Ma

Wentao Tang

Liang Xue

Wei Duan

Jiang Lu

Wei Jie Wang

Jia Ji

The present invention discloses a method and system for processing semantic fragments. Some embodiments of the present invention provides a method for processing semantic fragments. The method comprises: obtaining a plurality of groups of semantic fragments, the plurality of groups of semantic fragments at least including a first group of semantic fragments generated from a first data processing flow and a second group of semantic fragments generated from a second data processing flow, the first data processing flow being different from the second data processing flow; and merging the first group of semantic fragment and the second group of semantic fragment based on semantic equivalence. A corresponding system is also disclosed.

Method and system for processing semantic fragments

A shuffler receives information associated with partition segments of map task outputs and a pipeline policy for a job running on a computing device. The shuffler transmits to an operating system of the computing device a request to lock partition segments of the map task outputs and transmits an advisement to keep or load partition segments of map task outputs in the memory of the computing device. The shuffler creates a pipeline based on the pipeline policy, wherein the pipeline includes partition segments locked in the memory and partition segments advised to keep or load in the memory, of the computing device for the job, and the shuffler selects the partition segments locked in the memory, followed by partition segments advised to keep or load in the memory, as a preferential order of partition segments to shuffle.

Optimization of map-reduce shuffle performance through shuffler I/O pipeline actions and planning

Optimization of Map-Reduce shuffle performance through snuffler I/O pipeline actions and planning

Growing community of inventors

Shannxi, China

Qiang Xu