Systems and methods for auto-tiered data storage for data intensive applications

Abstract

Method and system for training a machine learning model based on a training dataset formed by data objects distributed across a virtual object storage service. The method comprises fetching from the virtual object storage service, the training dataset; copying the fetched training dataset on a first local storage device and maintaining a list of modifications executed on the training dataset that occurred on the virtual object storage service. The method comprises, upon receiving a request to initiate training of the machine learning model, generating a synchronized training dataset mirroring the training dataset stored in the virtual object storage service; storing the synchronized training dataset in a second local storage device; and fetching training data from the synchronized training dataset stored in the second local storage device as the training of the machine learning model is executed.