Adaptive caching and dynamic delay scheduling for in-memory data analytics

Abstract

In various embodiments, techniques for improving function of in-memory processing systems include identifying data stored in a distributed filesystem, including data to be processed by an in-memory processing application and data not to be processed by the in-memory processing application; identifying one or more partitions of the data corresponding to the data to be processed; and selectively transferring the one or more partitions from the distributed filesystem to a memory of the in-memory processing application. Techniques may also include determining an average remote execution time of the in-memory processing system; tracking a waiting time of a job pending processing on a local node of the in-memory processing system; comparing the tracked waiting time of the job pending processing; and demoting a locality of the job if waiting time of the job exceeds the average remote execution time of the in-memory processing system. Corresponding systems and computer-program products are also disclosed.