Learned resource consumption model for optimizing big data queries

Abstract

Methods, systems, apparatuses, and computer program products are provided for evaluating a resource consumption of a query. A logical operator representation of a query generated to be executed (e.g., obtained from a query generating entity) may be determined. The logical operator representation may be transformed to a plurality of different physical operator representations for executing the query. A plurality of resource consumption models may be applied to each of the physical operator representations to determine a resource consumption estimate for the physical operator representation. The resource consumption models may be trained in different manners based at least on a history of query executions, such that each model may have different granularity, coverage and/or accuracy characteristics in estimating a resource consumption of a query. Based on the determined resource consumption estimates for the physical operator representations, a particular one of the physical operator representations may be selected to execute the query.