Method and system for generating machine learning training data streams using unstructured data

Abstract

Techniques described herein relate to a method for managing training data. The method includes making a determination that the first stream request is associated with unstructured data; in response to the determination: obtaining a manifest associated with the unstructured data based on the stream request, wherein the manifest comprises metadata; loading the unstructured data into a cache using the manifest; merging the metadata with the unstructured data to generate training data, wherein the training data comprises a plurality of training data examples; generating augmented training data using the training data and the stream specification; generating a mini-batch sequence using the augmented training data and the stream specification; creating a mini-batch sequence queue and a stream endpoint; and streaming the mini-batch sequence using the mini-batch sequence queue and the stream endpoint, wherein the mini-batch sequence is used by a training environment to train a machine learning model.