Mapr Technologies, Inc.

Hewlett Packard Enterprise Development LP

Amit Ashoke Hadke

Chandra Guru Kiran Babu Sanapala

Uppaluri Vijaya Saradhi

Mandayam C Srivas

Pindikura Ravindra

Vivekanand Vellanki

Arvind Arun Pande

Lohit Vijaya Renu

Sathya Kavacheri

Jason D Frantz

A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Also addressed is the use of distributed transactions in a map-reduce system; the use of local and distributed snapshots; replication, including techniques for reconciling the divergence of replicated data after a crash; and mirroring.

Map-reduce ready distributed file system

The disclosed embodiments relate to a computer-implemented method for column-oriented access to data. The method can include inserting data into a data store. The data is randomly or sequentially retrievable from the data store by ordering keys for a table in a key-value store and recursively dividing a key space of said table into tablets that each have a range of possible keys. The tablets each contain partitions for key sub-ranges and each partition contains segments. Further, operations on tablets are distributed on different nodes and operations on partitions or segments are handled by using different threads.

Key-value store for map reduce system

A key-value store provides column-oriented access to data in a distributed and fault tolerant manner. Data can be inserted into the data store and data can be retrieved either randomly or sequentially from the data store at high rates. Keys for a table are ordered and the entire table is divided into key ranges. Each key range is handled by a table which itself is divided into key ranges called a partition. Partitions are also divided into segments. Such recursive division into smaller and smaller key ranges provides parallelism. At the highest level, operations on tablets can be distributed to different nodes. At lower levels, different threads can handle operations on individual segments. Large-scale restructuring operations can be decomposed into operations on individual segments so that a global lock on larger objects does not need to be kept across the entire operation.

Table format for map reduce system

A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Key-value stores relate keys to data for such purposes as directories, container location maps, and offset maps in compressed files.

Growing community of inventors

San Jose, CA, United States of America

Amit Ashoke Hadke