Universal format for multiple storage layer compatibility

Abstract

One of the key challenges that entities face when adopting data lakehouse for data storage is selecting a storage format for their data. However, choosing a single storage format to standardize is a difficult task, and entities often end up using different storage formats and different data processing services that may provide support for a particular storage format but not for the other. The disclosure addresses these challenges by recognizing the fact that different data storage formats are built on data files of the same type. The main differences are the metadata layer that stores metadata files describing the properties of these data files. The metadata for a second data storage format is automatically generated against a single copy of data files. As a result, entities can use the most suitable tool for each data workload and operate on a single data source, with interoperability across different storage ecosystems.