International Business Machines Corporation

Torrent Systems, Inc.

Michael James Beckerle

Amir Bar-Or

Jerry Lee Callen

Allen M Razdow

Shyam R Mudambi

Steven Rosenthal

David Albert Blair

Lawrence A Bookman

Anthony Passera

Jeffrey D Ives

Robert Louis Krawitz

Edward S Zyszkowski

Michael John Carney

John R Thorp

Robert Louise Krawitz

Daniel W Kohn

Daniel L Leary

James Richard Burns

Edward S A Zyzkowski

Seven Rosenthal

Provided are techniques for increasing transaction processing throughput. A transaction item with a message identifier and a session identifier is obtained. The transaction item is added to an earliest aggregated transaction in a list of aggregated transactions in which no other transaction item as the same session identifier. A first aggregated transaction in the list of aggregated transactions that has met execution criteria is executed. In response to determining that the aggregated transaction is not committing, the aggregated transaction is broken up into multiple smaller aggregated transactions and a target size of each aggregated transaction is adjusted based on measurements of system throughput.

Transaction aggregation to increase transaction processing throughput

Transaction aggregation to increase transaction processing throughout

Provided are techniques for pipeline optimization based on polymorphic schema knowledge. A hierarchical document to be processed by a pipeline of transformations is received. It is determined whether a next downstream transformation accesses content of each schema node in an associated input schema, wherein the input schema is a polymorphic schema. In response to determining that the next downstream transformation is not accessing the content of each schema node in the associated input schema, data items corresponding to each schema node that the next downstream transformation is not accessing are collected into a single compressed event. The collected items are passed to the next downstream transformation as the single compressed event.

Pipeline optimization based on polymorphic schema knowledge

Provided are techniques for processing data items. A limit on the number of dequeue operations allowed in a current step of processing for a queue-like data structure is set, wherein the number of allowed dequeue operations limit at least one of an amount of CPU resources and an amount of memory resources to be used by an operator. The operator to perform processing is selected and the operator is activated by passing control to the operator, which then dequeues data constrained by the limits set. In response to receiving control back from the operator, the data structure size is examined to determine whether the operator made forward progress in that the operator enqueued or dequeued at least one data item.

Asynchronous data structure pull application programming interface (API) for stream systems

Provided are techniques for compilation of hierarchical data processing. A data flow diagram including one or more operators, wherein each operator includes at least one of an incoming arc and an outgoing arc, is received. For each operator, for each incoming arc, it is validated that an arc input formal schema is compatible with a schema rooted in a context node in an arc input actual schema, and, for each outgoing arc, an arc output formal schema is computed based on operator logic and operator inputs and an arc output actual schema is computed from the arc input actual schema by replacing the context node of the arc input actual schema with the arc output formal schema.

Compilation model for processing hierarchical data in stream systems

Provided are techniques for optimizing the processing of hierarchical data. A linear processing graph is received, wherein the linear processing graph includes a plurality of operators, wherein each operator in the plurality is connected to at least one other operator by an arc, wherein hierarchical data flows on arcs, wherein the operators read and replace identified subregions within the hierarchical data flowing into the operators on the arcs, and wherein the operators do not modify the hierarchical data outside of these identified subregions. For each operator in the linear processing graph, a minimal set of dependent upstream operators on which that operator depends is found by examining how the identified subregions are created in the linear processing graph through obtaining a set of operators on which that operator depends, by analyzing dependencies carried by a set of vector nodes of the hierarchical data in an input schema of the operator, and, for each of the vector nodes, by analyzing an associated set of scalar nodes, wherein finding the minimum set of operators includes taking into consideration data preservation characteristics of the plurality of operators and taking into consideration structural-order preservation characteristics of the plurality of operators. The linear processing graph is rewritten to create a new graph that expresses dependencies based on the minimal set of dependent upstream operators for each operator.

Optimization model for processing hierarchical data in stream systems

A system and method for implementing a unified model for integration systems is presented. A user provides inputs to an integrated language engine for placing operator components and arc components onto a dataflow diagram. Operator components include data ports for expressing data flow, and also include meta-ports for expressing control flow. Arc components connect operator components together for data and control information to flow between the operator components. The dataflow diagram is a directed acyclic graph that expresses an application without including artificial boundaries during the application design process. Once the integrated language engine generates the dataflow diagram, the integrated language engine compiles the dataflow diagram to generated application code.

System and method for simulating data flow using dataflow computing system

With a continuous source of data relating to transactions, the data may be segmented and processed in a data flow arrangement, optionally in parallel, and the data may be processed without storing the data in an intermediate database. Data from multiple sources may be processed in parallel. The segmentation also may define points at which aggregate outputs may be provided, and where checkpoints may be established.

Segmentation and processing of continuous data streams using transactional semantics

Provided are a techniques for determining a frequency distribution for a set of records. A count table of frequency distributions is built in memory for each field in the set of records, wherein each record of each count table includes a field identifier, a field value, and a count of a number of times the field value occurs in the set of records, and wherein the field identifier concatenated with the field value comprises a composite key value. It is determined that at least one count table of frequency distributions is approaching a maximum amount of memory allocated to that count table. The records of the at least one count table that is approaching the maximum amount of memory are sent for sorting and additional counting, wherein the records include composite key values.

Method for computing frequency distribution for many fields in one pass in parallel

A computer system splits a data space to partition data between processors or processes. The data space may be split into sub-regions which need not be orthogonal to the axes defined the data space's parameters, using a decision tree. The decision tree can have neural networks in each of its non-terminal nodes that are trained on, and are used to partition, training data. Each terminal, or leaf, node can have a hidden layer neural network trained on the training data that reaches the terminal node. The training of the non-terminal nodes' neural networks can be performed on one processor and the training of the leaf nodes' neural networks can be run on separate processors. Different target values can be used for the training of the networks of different non-terminal nodes. The non-terminal node networks may be hidden layer neural networks. Each non-terminal node automatically may send a desired ratio of the training records it receives to each of its child nodes, so the leaf node networks each receives approximately the same number of training records. The system may automatically configures the tree to have a number of leaf nodes equal to the number of separate processors available to train leaf node networks. After the non-terminal and leaf node networks have been trained, the records of a large data base can be passed through the tree for classification or for estimation of certain parameter values.

Computer system and computerized method for partitioning data for parallel processing

A performance monitor represents execution of a data flow graph by changing performance information along different parts of a representation of that graph. If the graph is executed in parallel, the monitor can show parallel operator instances, associated datalinks, and performance information relevant to each. The individual parallel processes executing the graph send performance messages to the performance monitor, and the performance monitor can instruct such processes to vary the information they send. The monitor can provides 2D or 3D views in which the user can change focus, zoom and viewpoint. In 3D views, parallel instances of the same operator are grouped in a 2D array. The data rate of a datalink can be represented by both the density and velocity of line segments along the line which represent it. The line can be colored as a function of the datalink's source or destination, its data rate, or the integral thereof. Alternatively, a histogram can be displayed along each datalink's line, displaying information about the rate of, total of, or value of a field in, the data sent, at successive intervals. The user can click on objects to obtain additional information, such as bar charts of statistics, detailed performance listings, or invocation of a debugger. The user can selectively collapse representations of graph objects into composite representations, highlight objects which are out of records or which have flow blockages; label operators; turn off the display of objects; and record and playback the performance information.

Apparatuses and methods for monitoring performance of parallel computing

A system provides an environment for parallel programming by providing a plurality of modular parallelizable operators stored in a computer readable memory. Each operator defines operation programming for performing an operation, one or more communication ports, each of which is either an input port for providing the operation programming a data stream of records, or an output port for receiving a data stream of records from the operation programming and an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port. An interface enables users to define a data flow graph by giving instructions to select a specific one of the operators for inclusion in the graph, or instructions to select a specific data object, which is capable of supplying or receiving a data stream of one or more records, for inclusion in the graph, or instructions to associate a data link with a specific communication port of an operator in the graph, which data link defines a path for the communication of a data stream of one or more records between its associated communications port and either a specific data object or the specific communication port of another specific operator in said graph. The execution of a data flow graph equivalent to that defined by the users is automatically parallelized by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator, and by automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input.

Apparatuses and methods for programming parallel computers

Checkpointing of operations on data may be provided by partitioning the data into temporal segments. Operations may be performed on the temporal segments and checkpoints may be established by storing a persistent indication of the segment being processed. The entire processing state need not be saved. If a failure occurs, processing can be restarted using the saved indication of the segment to be processed. Such data partitioning and checkpointing may be applied to relational databases, databases with dataflow operation and/or parallelism and other database types with or without parallel operation.

Growing community of inventors

Needham, MA, United States of America

Michael James Beckerle