International Business Machines Corporation

Torrent Systems, Inc.

Qliktech International Ab

Jerry Lee Callen

Michael James Beckerle

Shyam R Mudambi

Allen M Razdow

Steven Rosenthal

David Albert Blair

Lawrence A Bookman

Robert Louis Krawitz

Robert Louise Krawitz

Daniel L Leary

Jeffrey D Ives

William V Kehoe

Steven R Frechette

James Richard Burns

Edward S A Zyzkowski

Seven Rosenthal

A method and system for developing data integration applications with reusable semantic types to represent and process application data. Methods include creating schemas to describe external data, creating semantic types to describe internal data, mapping schemas to semantic types, developing dataflows that configure input and output operations using schemas, mappings, and semantic types and all other transformation operations and functions based solely on semantic types, and executing dataflows defined in this manner.

Method and system for developing data integration applications with reusable semantic types to represent and process application data

A method, computer program product, and computing system for record sorting is described. The method may comprise splitting an incoming record into a separate key block and payload block. The method may further comprise storing the key block in a first memory. The method may also comprise assigning the payload block an address in a second memory at the beginning of a sort. Moreover, the method may store, with the key block in the first memory, the address of the payload block in the second memory. Additionally, the method may store the payload block at the address in the second memory.

Record sorting

With a continuous source of data relating to transactions, the data may be segmented and processed in a data flow arrangement, optionally in parallel, and the data may be processed without storing the data in an intermediate database. Data from multiple sources may be processed in parallel. The segmentation also may define points at which aggregate outputs may be provided, and where checkpoints may be established.

Segmentation and processing of continuous data streams using transactional semantics

Provided are a techniques for determining a frequency distribution for a set of records. A count table of frequency distributions is built in memory for each field in the set of records, wherein each record of each count table includes a field identifier, a field value, and a count of a number of times the field value occurs in the set of records, and wherein the field identifier concatenated with the field value comprises a composite key value. It is determined that at least one count table of frequency distributions is approaching a maximum amount of memory allocated to that count table. The records of the at least one count table that is approaching the maximum amount of memory are sent for sorting and additional counting, wherein the records include composite key values.

Method for computing frequency distribution for many fields in one pass in parallel

A system provides an environment for parallel programming by providing a plurality of modular parallelizable operators stored in a computer readable memory. Each operator defines operation programming for performing an operation, one or more communication ports, each of which is either an input port for providing the operation programming a data stream of records, or an output port for receiving a data stream of records from the operation programming and an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port. An interface enables users to define a data flow graph by giving instructions to select a specific one of the operators for inclusion in the graph, or instructions to select a specific data object, which is capable of supplying or receiving a data stream of one or more records, for inclusion in the graph, or instructions to associate a data link with a specific communication port of an operator in the graph, which data link defines a path for the communication of a data stream of one or more records between its associated communications port and either a specific data object or the specific communication port of another specific operator in said graph. The execution of a data flow graph equivalent to that defined by the users is automatically parallelized by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator, and by automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input.

Apparatuses and methods for programming parallel computers

Growing community of inventors

Cambridge, MA, United States of America

Jerry Lee Callen