Ab Initio Technology LLC

Dusan Radivojevic

Joel M Gould

Erik Bator

Anthony M Yeracaris

Andrew Schon

Joel Gould

Tim Wakeling

Robert Parks

David Clemens

Neil Galarneau

Marshall Alan Isman

Adam Harris Weiss

Jonah Egenolf

Gregg Yost

Peter Moss

Mazin Woodrow Khader

Richard Mellon

John MacLean

Paul Veiser

Among other things, we describe a method of receiving a portion of metadata from a data source, the portion of metadata describing nodes and edges; generating instances of a data structure representing the portion of metadata, at least one instance of the data structure including an identification value that identifies a corresponding node, one or more property values representing respective properties of the corresponding node, and one or more pointers to respective identification values, each pointer representing an edge associated with a node identified by the corresponding respective identification value; storing the instances of the data structure in random access memory; receiving a query that includes an identification of at least one particular element of data; and using at least one instance of the data structure to cause a display of a computer system to display a representation of lineage of the particular element of data.

Generating, accessing, and displaying lineage metadata

A method implemented by a data processing system for enabling a system to pipeline or otherwise process data in conformance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in conformance with the specified criteria.

Meta-data driven ingestion and data processing

Techniques for using finite state machines (FSMs) to implement workflows in a data processing system comprising at least one data store storing data objects and a workflow management system (WMS). The WMS is configured to perform: determining a current value of an attribute of a first data object by accessing the current value in the at least one data store; identifying, using the current value and metadata specifying relationships among at least some of the data objects, an actor authorized to perform a workflow task for the first data object; generating a GUI through which the actor can provide the input that the workflow task is to be performed; and in response to receiving, from the actor and through the GUI, input specifying that the workflow task is to be performed: performing the workflow task; and updating the current workflow state of the first FSM to a second workflow state.

Finite state machines for implementing workflows for data objects managed by a data processing system

In general, metadata is stored in a data storage system. Summary data identifying one or more characteristics of each of multiple metadata objects stored in the data storage system is computed, and the summary data characterizing a given metadata object in association with the given metadata object is stored. A visual representation is generated of a diagram including nodes representing respective metadata objects and relationships among the nodes. Generating the visual representation includes superimposing a representation of a characteristic identified by the summary data characterizing a given metadata object in proximity to the node representing the given metadata object.

Visualizing relationships between data elements and graphical representations of data element attributes

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for building reports. One of the methods includes creating a model based on relational structured data, the structured data including data structures, each data structure having data elements, each data element having fields, each field having a name. The method includes generating a hierarchy of objects in model, the hierarchy organizing objects the with respect to a starting object according to relationship fields on the objects. The method includes generating a user interface including elements for one or more of the objects in the hierarchy, wherein the user interface enables a user to create a report and filter the report using the new name. The method includes receiving a user selection of an element from the elements. The method also includes generating a report.

Building reports

A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Systems and methods for determining relationships among data elements

An identification of a directed graph is received that includes data transformation nodes that represent computations that transform data elements and one or more data nodes that represent data elements, and includes directed links that represent lineage relationships; and computing summary information based on paths in the directed graph, and storing the summary information in one or more summary objects. The computing includes: receiving designation of interest for a plurality of the nodes of the directed graph; and generating one or more summary objects for remaining nodes not included in the plurality of nodes of interest, a first summary object including summary information based on a first path between a first node of interest and a second node of interest that does include one or more of the remaining nodes and does not include any nodes of interest other than the first and second nodes.

Data lineage summarization

In general, a specification of multiple contexts that are related according to a hierarchy is received. Relationships are determined among three or more metadata objects, and at least some of the metadata objects are grouped into one or more respective groups. Each of at least some of the groups is based on a selected one of the contexts and is represented by a node in a diagram. Relationships among the nodes are determined based on the relationships among the metadata objects in the groups represented by the nodes, and a visual representation is generated of the diagram including the nodes and the relationships among the nodes.

Visualizing relationships between data elements

Managing lineage information includes processing a request for a representation of data lineage for a first node of a number of nodes. The processing includes determining an association between the first node and at least a first tag identifier of a number of tag identifiers, and determining a first subset of at least one and fewer than all of a number of possible tag values for the first tag identifier, and traversing nodes along a first lineage path of directed links from the first node to determine a data lineage for the first node. Determining the data lineage includes, for each traversed node determining whether to add the traversed node to the data lineage or to exclude the traversed node from the data lineage based at least in part on any tag identifiers or tag values associated with the traversed node.

Filtering data lineage diagrams

Managing lineage information includes processing a specification of a directed graph to associate nodes with information for processing requests for a representation of data lineage. The processing includes: identifying a first set of one or more nodes of the directed graph corresponding to normalizing data elements being stored in a data store and de-normalizing data elements being retrieved from the data store; and associating a first plurality of nodes connected to the first set of one or more nodes and a second plurality of nodes connected to the first set of one or more nodes with at least one tag identifier having a plurality of possible tag values, where the number of possible tag values is at least as large as the number of data elements being normalized, and where nodes representing different data elements in a de-normalized record are associated with different values of the tag identifier.

Growing community of inventors

North Andover, MA, United States of America

Dusan Radivojevic