Ab Initio Technology LLC

Ab Initio Software Corporation

Tim Wakeling

Mark Buxbaum

Ian Schechter

Matthew Darcy Atterbury

Joel M Gould

Ann M Wollrath

Bryan Phil Douros

David Parmenter

Jennifer M Farver

Joshua Goldshlag

Michael G Mulligan

Dusan Radivojevic

Joyce L Vigneau

Anthony M Yeracaris

Erik Bator

Mark Staknis

Dino LaChiusa

Harry Michael Wolfson

Brad Lee Miller

Frank Lynch

David W Parmenter

Craig W Stanfill

Joseph Skeffington Wholey, Iii

Adam Harris Weiss

Brond Larson

Glenn John Allin

Muhammad Arshad Khan

David E Wald

Dima V Feinhaus

Martin A Serrano

Glen John Allin

Anthony Yeracaris

Joseph Skeffington Iii Wholey

Joseph Skeffington Wholey

Dino Lachiusa

A method implemented by a data processing system including: accessing the container image that includes the first application and a second application; determining, by the data processing system, the number of parallel executions of the given module of the first application; for the given module, generating a plurality of instances of the container image in accordance with the number of parallel executions determined, for each instance, configuring that instance to execute the given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and for at least one of the plurality of configured instances, causing, by the second application of that configured instance, communication between the data processing system and the one or more of the host systems executing that configured instance.

Dynamic distribution of container images

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for workload automation and job scheduling information. One of the methods includes obtaining job dependency information, the job dependency information specifying an order of execution of a plurality of jobs. The method also includes obtaining data lineage information that identifies dependency relationships between data stores and transformation, wherein at least one transformation accepts data from a first data store and produces data for a second data store. The method also includes creating links between the job dependency information and the data lineage information. The method also includes determining an impact of a change in a planned execution of an application of the plurality of applications based on the job dependency information, the created links, and the data lineage information.

Workload automation and data lineage analysis

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Processing data from multiple sources

Mapping data stored in a data storage system for use by a computer system includes processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data. At least one of the dataflow graphs receives a flow of data from at least one input dataset and at least one of the dataflow graphs provides a flow of data to at least one output dataset. A mapper identifies one or more sets of datasets. Each dataset in a given set matches one or more criteria for identifying different versions of a single dataset. A user interface is provided to receive a mapping between at least two datasets in a given set. The mapping received over the user interface is stored in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

Mapping instances of a dataset within a data management system

A method of managing components in a processing environment is provided. The method includes monitoring (i) a status of each of one or more computing devices, (ii) a status of each of one or more applications, each application hosted by at least one of the computing devices, and (iii) a status of each of one or more jobs, each job associated with at least one of the applications; determining that one of the status of one of the computing devices, the status of one of the applications, and the status of one of the jobs is indicative of a performance issue associated with the corresponding computing device, application, or job, the determination being made based on a comparison of a performance of the computing device, application, or job and at least one predetermined criterion; and enabling an action to be performed associated with the performance issue.

Integrated monitoring and control of processing environment

In general, a specification of multiple contexts that are related according to a hierarchy is received. Relationships are determined among three or more metadata objects, and at least some of the metadata objects are grouped into one or more respective groups. Each of at least some of the groups is based on a selected one of the contexts and is represented by a node in a diagram. Relationships among the nodes are determined based on the relationships among the metadata objects in the groups represented by the nodes, and a visual representation is generated of the diagram including the nodes and the relationships among the nodes.

Visualizing relationships between data elements

Providing a user interface for configuring a computer-executable application includes receiving a specification defining: relationships among user interface elements, the relationships based on dependencies between components of a dataflow graph that includes multiple nodes representing components of the dataflow graph and links between the nodes representing flows of data between the components, parameters defining respective characteristics of the components of the dataflow graph, and variables defining respective characteristics of the user interface elements. During operation of a user interface, user interface elements are displayed based on the relationships defined in the specification.

Specifying user interface elements

Managing task execution includes: receiving a specification of a plurality of tasks to be performed by respective functional modules; processing a flow of input data using a dataflow graph that includes nodes representing data processing components connected by links representing flows of data between data processing components; in response to at least one flow of data provided by at least one data processing component, generating a flow of messages; and in response to each of the messages in the flow of messages, performing an iteration of a set of one or more tasks using one or more corresponding functional modules.

Task managing application for performing tasks based on messages received from a data processing application initiated by the task managing application

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic graph performance monitoring. One of the methods includes receiving input data by the data processing system, the input data provided by an application executing on the data processing system. The method includes determining a characteristic of the input data. The method includes identifying, by the application, a dynamic component from multiple available dynamic components based on the determined characteristic, the multiple available dynamic components being stored in a data storage system. The method includes processing the input data using the identified dynamic component. The method also includes determining one or more performance metrics associated with the processing.

Dynamic component performance monitoring

A method for supporting communication between a client and a server includes receiving a first message from a client. The method also includes creating an object in response to the first message. The method also includes sending a response to the first message to the client. The method also includes receiving changes to the object from a server. The method also includes storing the changes to the object. The method also includes receiving a second message from the client. The method also includes sending the stored changes to the client with a response to the second message.

Techniques for capturing execution time data in dataflow graphs

Growing community of inventors

Andover, MA, United States of America

Tim Wakeling