Liveramp, Inc.

Joseph Shannon Duncan

Pavan Roy Marupally

Adam Zimmerman

A method for improving fault tolerance of MapReduce computing systems processing extremely large data sets employs a custom mapper script that outputs the results to the shuffle and reduce processes. The Reducer writes its results into multiple files that are then compressed. The resulting files are processed by a map-only Reducer job that aggregates and reduces the results of a single file into a single output. The non-block compression prevents the file from being split across multiple mappers. The job can recover if a single mapper fails without requiring that each Map node be rerun, thus improving fault tolerance.

Fault tolerant big data processing

A system and method for efficiently matching into an extremely large data set uses two data stores. A search data store contains identifiers that correspond to localized portions of the main data store, along with matching values for that portion of the main data store. To perform a matching operation, the search data store is first searched for a matching value. Then the identifier for the localized portion of the main data store is retrieved from the search data store. Using this identifier, a search may be performed against only the corresponding localized portion of the main data store rather than the entire main data store.

Growing community of inventors

Conway, AR, United States of America

Joseph Shannon Duncan