Automatic configuration of parameters across multiple compute engines in data lakehouse environments

Abstract

The disclosed methods provide automatic configuration of system parameters across multiple compute engines within a data lakehouse environment. The process begins by obtaining configuration parameters from both a first and second compute engine. Utilizing a pre-trained large language model (LLM), the method analyzes the first set of parameters to establish contextual definitions, which are compiled into a configuration definition file. Similarly, the second set of parameters is analyzed and documented in another configuration definition file. The method then determines correspondences between parameters from the two sets by comparing their respective definition files. Based on these correspondences, the second compute engine automatically applies parameter values from the first set to the second set. This approach streamlines the transition between compute engines, ensuring efficient parameter mapping and application, thereby optimizing performance and reducing manual configuration efforts.