Dell Products Inc.

Inga Sogaard

Ofir Ezrielev

Hanna Yehuda

Hana Schuster Smith

The technology described herein is directed towards mapping the storage tiers of various multi-cloud vendors to an enterprise's storage tiers and the enterprise's associated storage tier datasets, based on enterprise storage tiering definition data. Once mapped, the storage tier datasets are populated based on the vendor's per-tier metadata that are associated with the enterprise's data stored on that vendor's storage tiers. Similar mapping is performed for each other vendor's storage tiers. Once the storage tier datasets are populated based on the corresponding metadata, the metadata's variable parameters (e.g., performance metrics) for the enterprise's stored data can be presented according to the enterprise's own defined tiers and labels for the parameters. The metadata information in the datasets can be viewed per vendor and per enterprise-defined tier dataset, such as for object storage monitoring, cost comparison among different vendors' tiers, and closest vendor-to-enterprise parity comparisons of like variables.

Multi-cloud tiering translation to enterprise tiers

Methods and systems for managing operation of a data pipeline are disclosed. To manage the operation, a system may include one or more data sources, a data manager, and one or more downstream consumers. Changes to a system of representation of information in data requested by the downstream consumers may cause the data pipeline to provide unusable data to the downstream consumers. To remediate the change, a translation schema may be obtained to translate data based on an unexpected system of representation of information to data based on an expected system of representation of information. The translation schema may be tested and, if the translation schema is determined successful, a translation layer may be added to the data pipeline based on the translation schema.

Remediating a change in a system of representation of information in data used by a data pipeline

Methods and systems for managing operation of a data pipeline are disclosed. To manage the data pipeline, a system may include one or more data sources, a data repository, and one or more downstream consumers. Updates to the data pipeline may cause the data pipeline to become misaligned. To avoid misalignment and, therefore, failure of the data pipeline, information regarding the operation of the data pipeline may be distributed and used to decide how to update the data pipeline. The update may be evaluated through simulation prior to implementation.

Managing data pipelines using graphical user interfaces

Methods and systems for managing operation of a data pipeline are disclosed. To manage the data pipeline, a system may include one or more data sources, a data repository, and one or more downstream consumers. As new data becomes available for use in the data pipeline, the new data may be automatically evaluated for utility. The evaluation may be made through establishment of a shadow pipe. The shadow pipe may allow for comparison of operation of the data pipeline with the new data against operation of the data pipeline without the new data.

Managing integration of new data for data pipelines

Methods and systems for managing operation of a data pipeline are disclosed. To manage the data pipeline, a system may include one or more data sources, a data repository, and one or more downstream consumers. Updates to the data pipeline may cause the data pipeline to become misaligned. To avoid misalignment and, therefore, failure of the data pipeline, any updates intended for implementation in the data pipeline may first be implemented in a digital twin of the data pipeline. The digital twin and the data pipeline may operate concurrently using substantially identical data. The performance of the digital twin following implementation of the potential update may be compared to the performance of the data pipeline without implementation of the potential update to obtain a performance delta. If the performance delta meets a threshold, the potential update may cause failure of the data pipeline and may be remediated prior to implementation.

System and method for managing a data pipeline using a digital twin

Methods and systems for managing operation of a data pipeline are disclosed. To manage operation of a data pipeline when a portion of data is inaccessible may require generating synthetic portion of data to generalize the inaccessible portion of data. Prior to the generation of the synthetic portion of data, an analysis of an intended use of the inaccessible portion of data may be performed. The analysis may reduce the likelihood of generation and use of synthetic portion of data that is unreliable. Once obtained, the synthetic portion of data may be analyzed to determine a likelihood that the synthetic portion of the data may successfully generalize the inaccessible portion of data. When the synthetic portion of data is determined to meet or exceed quality criteria, the synthetic portion of data may be utilized by the data pipeline.

Growing community of inventors

Wichita, KS, United States of America

Inga Sogaard