Cloudera, Inc.

Sudhanshu Arora

Vikas Singh

Chang She

Philip Zeyliger

Naren Koneru

Marcelo Masiero Vanzin

Mark Donsky

Guang Yao Leng

Himabindu Vuppula

Techniques are disclosed for inferring design-time information based on run-time artifacts generated by services operating in a distributed computing cluster. In an embodiment, a metadata system extracts metadata including run-time artifacts generated by services in a distributed computing cluster while processing a workflow including multiple jobs. The extracted metadata is processed to identify entities and entity relationships which can then be used to generate lineage information. Using the lineage information, the metadata system can infer design-time information associated with the workflow. The inferred design-time information can then be utilized to, for example, recreate the workflow, recreate previous versions of the workflow, optimize the workflow, etc.

Design-time information based on run-time artifacts in a distributed computing cluster

Transient computing clusters can be temporarily provisioned in cloud-based infrastructure to run data processing tasks. Such tasks may be run by services operating in the clusters that consume and produce data including operational metadata. Techniques are introduced for tracking data lineage across multiple clusters, including transient computing clusters, based on the operational metadata. In some embodiments, operational metadata is extracted from the transient computing clusters and aggregated at a metadata system for analysis. Based on the analysis of the metadata, operations can be summarized at a cluster level even if the transient computing cluster no longer exists. Further relationships between workflows, such as dependencies or redundancies, can be identified and utilized to optimize the provisioning of computing clusters and tasks performed by the computing clusters.

Design-time information based on run-time artifacts in transient cloud-based distributed computing clusters

Information based on run-time artifacts in a distributed computing cluster

Growing community of inventors

Sunnyvale, CA, United States of America

Sudhanshu Arora