The Mathworks, Inc.

Richard Amos

Thomas Lane

Jocelyn Martin

Penelope Anderson

Wei Wang

Michael Procopio

James Stewart

Lucio Andrade-Cetto

Aaditya Kalsi

Anandan Rangasamy

Yashwanth Annapureddy

Nicholas Haddad

Edric Mark Ellis

Aaron J Powers

Danil Kirsanov

Kari Sortland

Benjamin Tordoff

According to some embodiments, a method for making input data available for processing by one or more processors comprises storing one or more parameters, wherein the one or more parameters comprise information identifying a location of the input data; and creating a datastore object using the one or more parameters, wherein the datastore object interfaces the input data and includes a read method for reading a chunk, the chunk being a subset of the input data, and having a size that does not exceed a memory size assigned to the one or more processors. According to some embodiments, the one or more parameters further comprise one or more of a type of the input data; a format of the input data; an offset for reading from the input data; a size of the chunk; a condition for determining the chunk; and a query for deriving the input data.

Datastore mechanism for managing out-of-memory data

A scheduler is provided for tall algorithms that include control flow statements that are dependent on tall computations. Thunks may be defined for the tall algorithms and the boundaries or limits of the thunks may be determined by places where tall variables are required to be gathered. The scheduler executes the tall algorithms by calling individually the thunks from the tall algorithms and pausing when a gather operation is required. The scheduler collectively gathers the tall variables from different tall algorithms by using an optimizer to combine I/O operations for the tall algorithms. The scheduler may put back the gathered variables into the context of each tall algorithm, and the scheduler may resume with the next thunk for each tall algorithm. This process may be repeated until all tall algorithms are finished.

Scheduler for tall-gathering algorithms that include control flow statements

A device may include one or more processors. The device may receive program code that identifies operations to be performed with regard to a data set to determine an output value. The device may identify a plurality of read operations corresponding to the operations. The plurality of read operations may be performed to obtain respective inputs of the operations. The device may generate a first data structure that identifies the operations and the plurality of read operations. The device may identify two or more operations, of the operations, that receive a particular input. The device may generate a second data structure that includes a combined operation. The combined operation may be generated based on the two or more operations, and may be associated with fewer read operations than the two or more operations. The device may determine the output value based on the second data structure.

Big data read-write reduction

According to some embodiments, a method for processing input data comprises creating a MapReducer object corresponding to a MapReduce environment; and receiving, by a MapReduce interface, a plurality of input parameters comprising the input data; a mapper function; a reducer function; and the MapReducer object; and using the MapReduce interface to process the input data by one or more processors in the MapReduce environment using the mapper function and the reducer function. According to some embodiments, the method further comprises creating a second MapReducer object, wherein the second MapReducer object corresponds to a second MapReduce environment; receiving, by the MapReduce interface, the second MapReducer object in place of the first MapReducer object; and utilizing the MapReduce interface to process the input data by the one or more processors in the second MapReduce environment using the mapper function and the reducer function.

Unified mapreduce framework for large-scale data processing

Growing community of inventors

St. Ives, United Kingdom

Richard Amos