Amazon Technologies, Inc.

Arnab Dutta

Daniel Opincariu

Ramanathan Muthiah

Anas Fattahi

Srinivasan V Rajagopalan

Rajasuba Subramanian

Ranil Pavithran Muzhangathu

Deepan Chakravarthy Vijayarangam

In a data lake, a control data object is defined. The control object defines the processes and relationships of processes associated with a data set in the data lake. The control has states that are tied to and adapt in response to state changes of the associated data set. A control can have a control type. The system automatically carries forward enabled processes from one data set version to the next data set version. The system uses the control definition to execute processes, such as compaction or data quality scans, on data sets in the data lake.

Federated execution of data lake processes

Techniques are described for employing a graph-based analysis to determine a configuration of datasets to be stored on data storage systems in a data warehouse environment. Associations between datasets may be determined based on the parsing of join statements or other types of statements in jobs that are executed on the data storage systems. A graph may be generated that describes the associations among datasets. A greedy breadth-first traversal of the graph may be performed to determine sets of associated datasets. A utilization metric describing a weight of storing the datasets may be determined and employed to identify a data storage system on which to store a set of associated datasets, given the storage and processing capacity of the data storage system.

Storage configuration in data warehouses

Growing community of inventors

Seattle, WA, United States of America

Arnab Dutta