Ab Initio Technology LLC

Ab Initio Software Corporation

Richard A Shapiro

Craig W Stanfill

Joseph Skeffington Wholey, Iii

Stephen A Kukolich

Adam Harris Weiss

Joel M Gould

Andrew Fiske Roberts

Brond Larson

Edward Alan Bach

Adam Weiss

Iii Joseph Skeffington Wholey

A method for generating an updated graph based on a base graph includes identifying differences between a base graph and each of a source and a target graph. The specification of each graph defines nodes representing data processing components defining operations for processing data, and links representing flow of data. The identifying includes: comparing specifications of base and source graphs to identify a first set of differences between the base and source graph, and comparing specifications of base and target graphs to identify a second set of differences between the base and the target graph. The method includes characterizing a correspondence between the differences in the first set and the differences in the second set, determining a set of combined differences based on the correspondence and the first and second sets of differences, and generating the updated graph based on the base graph and the set of combined differences.

Updating executable graphs

A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each connecting an output port of an upstream component of the plurality of components to an input port of a downstream component of the plurality of components. Prepared code is generated representing subsets of the plurality of components, including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of linked components; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in that formed subset to be performed according to information embedded in the prepared code for that formed subset.

Compiling graph-based program specifications

Entity data represent entities related according to a hierarchy, with at least one root entity at a highest level and one or more entities at one or more levels below the highest level. One or more entities each have multiple instances, and at least one attribute having respective elements for each of the instances. Input data includes expressions that express functions of arguments, with one or more of the arguments referencing an attribute of an entity represented in the entity data. Processing expressions includes: (a) analyzing an expression to determine, for each of one or more attributes, at least one of: (1) a position in the hierarchy of an entity of the attribute, or (2) a number of elements of a result of a function applied to an argument that references the attribute, and (b) evaluating the expression, or indicating an error based on a result of analyzing the expression.

Managing computations for hierarchical entities

A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.

Executing graph-based program specifications

Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Controlling data processing tasks

A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

Data processing graph compilation

A first component of a graph-based program specification includes an output control port. A second component includes an input control port and an input data port. A third component includes an output data port. The output control port is connected to the input control port, and the output data port is connected to the input data port. The first component includes control code that when executed causes the output control port to provide, to the input control port, at least one of suppression information or invocation information. The second component includes control code that when executed causes a computing system configured by the graph-based program specification to begin processing data received at the input data port in response to the invocation information if no suppression information is received at the input control port before the invocation information is received at the input control port.

Specifying control and data connections in graph-based programs

A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Managing invocation of tasks

A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration between ports of linked components, corresponding to transfer of control or transfer of a single data element, and a second type of link configuration between ports of linked components, corresponding to transfer of multiple data elements. A compiler generates a target program specification including control code representing at least one control graph including graph nodes representing the components, where at least two are connected based on links of the first type. A computing node initiates execution of the target program specification, and manages computing resources for links of the second type, the computing resources including at least one of: (1) a buffer for storing data elements provided by an output port, or (2) a buffer for storing data elements provided to an input port.

Growing community of inventors

Arlington, MA, United States of America

Richard A Shapiro