Systems and/or methods for leveraging in-memory storage in connection with the shuffle phase of mapreduce

Abstract

Certain example embodiments relate to a computer system for performing a map reduce sequence. Nodes therein include at least one processor and memory and are divided into at least mapper and reducer nodes. Each mapper node executes a map function on input to generate intermediate output elements. Each said intermediate output element includes a first key-value pair. Each element key includes associated map and reduce task identifiers. Each element value includes substantive data, organized as another key-value pair. The intermediate output elements are stored to memory. Each reducer node: retrieves at least intermediate output element values from the memory of a given mapper node, using specified map and reduce task identifiers; stores the retrieved element values to its memory; executes a reduce function on the retrieved element values, in order; and outputs a result from the reduce function for the map reduce sequence. Disk operations advantageously are reduced or eliminated.