Generating observability metrics for data lake usage based on data layer activity logs

Abstract

Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.