Markov decision process for efficient data transfer

Abstract

Techniques are disclosed for improving transfer speed for a plurality of files (e.g., image files) by using a Markov decision process to determine an optimal number of parallel instances of transfer stages and optimal file batch sizes for each instance. The transfer (e.g., import or export) operation involves different stages that are each optimized using the algorithm. The stages include a file fetch operation, a file processing operation, and a database update operation. Each of the stages may have multiple parallel instances to process many files at the same time. The Markov decision process uses a reward structure to determine the optimal number of parallel instances for each stage and the number of files operated on at each instance at any given moment in time. The process is dynamic and adaptable to any system environment since it does not rely on any particular hardware or operating system configuration.