Xcalar, Inc.

Jerene Yang

Vikram Joshi

Michael A Brown

Brent Lim Tze Hao

Techniques are described herein for creating an algorithm for batch mode processing against big data. The techniques involve receiving one or more user commands from a set number of commands that correspond one-to-one with a set number of low-level database operations. In a preferred embodiment, the set of database operations includes only FILTERS, SORTS, AGREGGATES, and JOINS. In the algorithm formation process, database operations are performed on a sample population of records. The user drills down to a set of useful records by performing database operations against the results of the previous database operations. While the database cluster is receiving operations, the system is tracking the operations in a dependency graph. The chains selected within the dependency graph indicate which operations are used to create the algorithm. To generate the algorithm, the database cluster reverse engineers the logic for performing those operations against big data.

Data driven relational algorithm formation for execution against big data

Techniques are described herein for performing database operations against location and access transparent metadata units called fat pointers organized into globally distributed data structures. The fat pointers are created by extracting values corresponding to a particular key and paring each value with a reference to the local location and server that has the native format record containing the value. The fat pointers may be transferred to any server in the cluster, even if the server is different from the server that has the native format record. In general, most operations are performed against fat pointers rather than the native format records. This allows the cluster to perform work against arbitrary types of data efficiently and in a constant amount of time despite the variable sizes and structures of records.

Growing community of inventors

San Jose, CA, United States of America

Jerene Yang