Building a data query engine that leverages expert data preparation operations

Abstract

A method, system and computer program product for building a data query engine. Initial taxonomies that describe and categorize data are built by expert users (e.g., data scientists) employing machine learning algorithms. The data is also indexed and stored in an index. Queries are then received from non-expert users to query the data based on data categorization from built taxonomies and the indexing. After the queries are executed using the machine learning algorithms in an environment (e.g., Hadoop®), the results of the queries are rated for relevance, precision and accuracy. The machine learning algorithms are also rated based on the number of successful queries. Those machine learning algorithms with a rating above a threshold are identified to be utilized to scan new data to be stored in the index to provide a new environment that replaces the initial environment.