Ab Initio Technology LLC

Ian Schechter

Glenn John Allin

Tim Wakeling

Ann M Wollrath

Garth Allen Dickie

Marshall Alan Isman

Jonah Egenolf

David Parmenter

Jennifer M Farver

J Skeffington Wholey

Joshua Goldshlag

David E Wald

David Fournier

Amit Weisman

Carl Richard Feynman

Edward Alan Bach

Robert Parks

Cory Christopher James Fantasia

Matthew Douglas Becker

Alexander Lavrentyev

Joseph Skeffington Wholey

David W Parmenter

A method includes accessing a schema that specifies relationships among datasets, computations on the datasets, or transformations of the datasets, selecting a dataset from among the datasets, and identifying, from the schema, other datasets that are related to the selected dataset. Attributes of the datasets are identified, and logical data representing the identified attributes and relationships among the attributes is generated. The logical data is provided to a development environment, which provides access to portions of the logical data representing the identified attributes. A specification that specifies at least one of the identified attributes in performing an operation is received from the development environment. Based on the specification and the relationships among the identified attributes represented by the logical data, a computer program is generated to perform the operation by accessing, from storage, at least one dataset having the at least one of the attributes specified in the specification.

Generation of optimized logic from a schema

A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.

Dataset multiplexer for data processing system

Techniques for generating a dataflow graph include generating a first dataflow graph with a plurality of first nodes representing first computer operations in processing data, with at least one of the first computer operations being a declarative operation that specifies one or more characteristics of one or more results of processing of data, and transforming the first dataflow graph into a second dataflow graph for processing data in accordance with the first computer operations, the second dataflow graph including a plurality of second nodes representing second computer operations, with at least one of the second nodes representing one or more imperative operations that implement the logic specified by the declarative operation, where the one or more imperative operations are unrepresented by the first nodes in the first dataflow graph.

Editor for generating computational graphs

A method includes executing a program that specifies operations and accessing a translation file that includes instructions for translating the language of the program into a language of a database. The translation file specifies operations in the language of the program that are supported by the database and the semantic meaning of the supported operations in the language of the database. Operations of the program that are unsupported by the database are processed by the program. Operations of the program that are supported by the database are determined from the translation file, and a portion of the program representing the supported operations is translated, using the translation file, into the language of the database and transmitted to the database. Data resulting from execution, within the database, of the translated portion of the program representing the operations that are supported by the database is received by the program.

Transforming operations of a computer program for execution at a database

Techniques for allowing a user to invoke lifecycle operations on a data store configured in accordance with a data management scheme, without knowledge of the data management scheme. Techniques include receiving user input from a user interface; identifying, in the received user input, an indication of a first lifecycle operation corresponding to a first entry in a template; based on information in the first entry, identifying in the received user input a first configuration value associated with the first lifecycle operation; formatting the first configuration value as at least a portion of at least one argument to a first command; and invoking, via the first command with the at least one argument, a first computer program configured to perform the first lifecycle operation on the data store.

Techniques for performing lifecycle operations on a data store

In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Processing data from multiple sources

One method includes receiving a database query, receiving information about a database table in data storage populated with data elements, producing a structural representation of the database table that includes a formatted data organization reflective of the database table and is absent the data elements of the database table, and providing the structural representation and the database query to a plan generator capable of producing a query plan representing operations for executing the database query on the database table. Another method includes receiving a query plan from a plan generator, the plan representing operations for executing a database query on a database table, and producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan.

Managing data queries

Among other things, we describe a computer-implemented method, performed by a data processing system, of executing a computer program based on a query that is expressed in accordance with a query language applicable to a relational database, the computer program executed based at least in part on data stored in a tangible, non-transitory computer-readable medium, the executing including receiving a SQL query, where the SQL query includes an identifier associated with a resource that is external to the data processing system, generating a computer program based on the SQL query, and executing the computer program, causing transmitting one or more instructions to the resource, the instructions defining operations other than operations of the SQL query, and receiving data from the resource in response to the instructions.

Querying a data source on a network

Techniques for monitoring execution of structured query language (SQL) queries. The techniques include: obtaining a SQL query; generating a query plan for the SQL query, the query plan comprising a plurality of operations to be performed by the data processing system including at least a first operation and a second operation; displaying a graphical user interface (GUI) containing a plurality of GUI elements including a first GUI element representing the first operation and a second GUI element representing the second operation; during execution of the SQL query, gathering tracking information for the SQL query including gathering first tracking information for the first operation and second tracking information for the second operation; and displaying, in the GUI, at least some of the first tracking information in association with the first GUI element and at least some of the second tracking information in association with the second GUI element.

Growing community of inventors

Sharon, MA, United States of America

Ian Schechter