Hewlett Packard Enterprise Development LP

Annmary Justine Koomthanam

Suparna Bhattacharya

Paolo Faraboschi

Martin Foltin

John Michael Czerkowicz

Sergey Serebryakov

Aalap Tripathy

Jothivelavan Sivashanmugam

Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages/processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).

Method to track and clone data artifacts associated with distributed data processing pipelines

Example implementations relate to partial file system instances. In an example, a subset of objects of a source file system instance on a source system are replicated to a target system to form a partial file system instance on the target system comprised of the subset of objects. Each of the objects of the source file system instance is identified by a signature based on content of each of the objects and the objects exhibit a hierarchical relationship to a root object in the file system instance. An unmaterialized object is dynamically added to the partial file system instance by replicating the corresponding object from the source file system instance. The target system is asynchronously updated from the source file system instance based on a comparison of the partial file system instance to the source file system instance.

Partial file system instances

Examples described herein relate to In some examples, a least recently used (LRU) list used to evict nodes from a cache is traversed, the nodes referencing data blocks storing data in a storage device, the nodes being leaf nodes in a tree representing a file system, and for each traversed node, determining a stride length as an absolute value of a difference between a block offset value of the traversed node and a block offset value of a previously traversed node, comparing the stride length to a proximity threshold, and updating a sequential access pattern counter based at least in part on the comparing; and proactively prefetching nodes from the storage device to the cache when the sequential access pattern counter indicates a detected pattern of sequential accesses to nodes.

Detecting sequential access patterns and proactive caching in a tree-based file system

Growing community of inventors

Bengaluru, India

Annmary Justine Koomthanam