Machine learning pipeline performance acceleration with optimized data access interfaces using in-memory and distributed store

Abstract

Systems and methods are provided for utilization of optimal data access interface usage in machine learning pipelines. Examples of the systems and methods disclosed herein include identifying data access interfaces comprising at least a first data access interface for a persistent storage distributed across a plurality of storage nodes and at least a second data access interface for an in-memory object store, and receiving, from a compute node, a data operation request as part of a machine learning pipeline. Additionally, performance metrics are obtained for the plurality of access interfaces, and based on a type of data operation request, the data operation is executed using a data access interface selected from the plurality of data access interface based on the performance metrics and providing an object handle to the compute node.