Ab Initio Technology LLC

Ab Initio Software Corporation

Paul Bay

Joel M Gould

H Mark Bromley

Brond Larson

Carl Richard Feynman

Ephraim Meriwether Vishniac

Marshall Alan Isman

John L Richardson

Craig W Stanfill

Daniell Garrick Stevens

Carl Offner

Tsvetan Tsvetanov

Dan Teven

Ephraim Vishniac

Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Managing interfaces for sub-graphs

Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

In certain aspects, a method includes: receiving a first graph that includes components and flows, the components representing operations performed on data records, the flows representing flows of data records between components; receiving a specification that is separate from the first graph, the specification defining one or more insertions, each of the insertions associated with a flow of the first graph; generating one or more components that each corresponds to one of the insertions; and generating a second graph that includes components and flows that correspond to at least some of the components and flows of the first graph and the one or more generated components.

Debugging a graph

Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Data profiling

A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Managing storage of individually accessible data units

Processing data includes accepting information characterizing values of a first field in records of a first data source and information characterizing values of a second field in records of a second data source. Quantities characterizing a relationship between the first field and the second field are computed based on the accepted information. Information relating the first field and the second field is presented.

Joint field profiling

Processing data includes identifying a plurality of subsets of fields of data records of a data source, determining co-occurrence statistics for each of the plurality of subsets, and identifying one or more of the plurality of subsets as having a functional relationship among the fields of the identified subset.

Functional dependency data profiling

A data processing system and method that provides checkpointing and permits a continuous flow of data processing by allowing each process to return to operation after checkpointing, independently of the time required by other processes to checkpoint their state. Checkpointing in accordance with the invention makes use of a command message from a checkpoint processor that sequentially propagates through a process stage from data sources through processes to data sinks, triggering each process to checkpoint its state and then pass on a checkpointing message to connected &ldquo;downstream&rdquo; processes. This approach provides checkpointing and permits a continuous flow of data processing by allowing each process to return to normal operation after checkpointing, independently of the time required by other processes to checkpoint their state.

Growing community of inventors

Arlington, MA, United States of America

Paul Bay