Maintaining data in memory for transformation and quality checking when reading from a source and loading to a target

Abstract

Data transformation and data quality checking is provided by reading data from a source datastore and storing the data into memory, performing in-memory processing of the data stored in memory, where the data is maintained in-memory for performance of the in-memory processing thereof, and where the in-memory processing includes performing one or more transformations on the data stored in memory, in which the data stored in memory is transformed and stored back into the memory and applying one or more data quality rules to the data stored in-memory, and based on performing the in-memory processing of the data stored and maintained in memory for the in-memory processing, loading to a target datastore at least some of the data processed by the in-memory processing.