Conditional data sourcing and curation for generating datasets in deep learning model training

Abstract

In various examples, a technique for performing conditional data sourcing and curation includes retrieving, by a plurality of processing nodes, a plurality of content items from one or more data sources, wherein each processing node retrieves a different subset of the plurality of content items from the one or more data sources. The technique also includes applying a first set of filters to metadata associated with the content items to generate a plurality of filtered content items. The technique further includes generating, based on a subset of the metadata associated with the filtered content items and a second set of filters, mappings between the filtered content items and text descriptions for the filtered content items and storing, based on the mappings, the filtered content items in association with the text descriptions in one or more data stores.