International Business Machines Corporation

Other

Robert Evan Strom

Chitra Dorai

Daniel Charles Sturman

Michael James Ward

Kevan Lee Miller

Tushar Deepak Chandra

Guruduth Somasekhara Banavar

Mark C Astley

Roman Ginis

Ying Li

Huining Feng

Edith H Stern

Oded Eddie Sturman

Geetika Tewari Lakshmanan

Wei Zheng

Marc Adam Kaplan

Joshua S Auerbach

Sumeer K Bhola

Steven E Massey

Yuanyuan Zhao

Atul Prakash

Christopher North

Sai Zeng

Tien Huynh

Daniel M Yellin

Guruduth Somesekhara Banavar

Shaula Yemini

Yuhui Jin

Jeong-Hyon Hwang

Om Prakash Damani

Michal Z Ukelson

Wei Tao

Andrey Khorlin

Lu Tian

A method and system for achieving time-awareness in the highly available, fault-tolerant execution of components in a distributed computing system, without requiring the writer of these components to explicitly write code (such as entity beans or database transactions) to make component state persistent. It is achieved by converting the intrinsically non-deterministic behavior of the distributed system to a deterministic behavior, thus enabling state recovery to be achieved by advantageously efficient checkpoint-replay techniques. The system is deterministic by repeating the execution of the receiving component by processing the messages in the same order as their associated timestamps and time-aware by allowing adjustment of message execution based on time.

System and method for time-aware run-time to guarantee timeliness in component-oriented distributed systems

Methods and apparatus for integrating relational and hierarchical data, schema definitions, and queries in a data processing system are provided. One or more schema definitions are converted into an intermediate schema language component of an intermediate data language when one or more schema definitions are provided. One or more query expressions are converted into an intermediate query language component of the intermediate data language when one or more query expressions are provided. The intermediate schema language component or the intermediate query language component is compiled in an intermediate data language processing engine into a run-time representation in accordance with a relational-hierarchical analysis.

Method and apparatus for integrating relational and hierarchical data

A distributed messaging system supporting stateful subscriptions is provided. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. Stateful subscriptions may refer to one or more message histories and may include more complex expressions. Therefore, subscribers may receive different information than that provided in the published messages. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of transform objects and view objects. The messaging system builds a structure containing all transform objects and view objects needed for all intermediate and subscribed views of all subscriptions. This messaging system uses this structure to allocate transform objects and view objects to broker machines in the overlay network.

Distributed messaging system supporting stateful subscriptions

A method comprises estimating a performance of a processing system using queueing theory wherein the processing system comprises a network that accepts at least one input message stream from a plurality of publishing clients, transforms the at least one input message stream into at least one output message stream utilizing the plurality of transforms distributed in a first assignment among the plurality of brokers, and delivers the at least one output message stream to at least one of the plurality of subscribing clients, performing optimization upon the assignment of the plurality of transforms using the estimated performance to produce a second assignment, and reassigning the plurality of transforms in accordance with the second assignment.

Method for predicting performance of distributed stream processing systems

System and method for time-aware run-time to guarantee time

Controlling live migration of a running application between a source machine and a target machine is provided. At least one application characteristic and at least one network characteristic are obtained. An objective function that defines a cost to migrate from the source machine to the target machine as a function such as a weighted combination of at least one cost property is obtained. The objective function is optimized using the at least one application characteristic and the at least one network characteristic to determine a policy specifying when to switch execution of the running application from the source machine to the target machine. Live migration of the application from the source machine is controlled to send pages from the source machine to the target machine in accordance with the policy, and to switch the execution of the application from the source machine to the target machine at a state specified by the policy.

Optimizing migration policy during live virtual memory migration

Computer implemented method, system and computer usable program code for achieving transactional quality of service in a transactional object store system. A transaction is received from a client and is executed, wherein the transaction comprises reading a read-only derived object, or reading or writing another object, and ends with a decision to request committing the transaction or a decision to request aborting the transaction. Responsive to a decision to request committing the transaction, wherein the transaction comprises writing a publishing object, events are delivered to event stream processing queries, and are executed in parallel with executing of the transaction. Responsive to a decision to request committing a transaction that comprises reading a read-only derived object, a validation is performed to determine whether the transaction can proceed to be committed, whether the transaction should abort, or whether the validation should delay waiting for one or more event stream processing queries to complete.

Transactional quality of service in event stream processing middleware

A computer implemented method, system and computer program product for delivering published messages to at least one subscriber in a publish-subscribe messaging system. A computer implemented method for delivering published messages to a subscriber includes receiving a plurality of message streams, and analyzing at least one message from each of the plurality of message streams in accordance with criteria established by the subscriber specifying a message content and a message content quality. At least one message that satisfies the criteria is delivered to the subscriber.

Method and system for mediating published message streams for selective distribution

Method, system and computer readable program code for dynamic and evolutionary component placement in an event processing system having producers, consumers, a plurality of nodes between the producers and the consumers, and a flow graph representing operator components to be executed between the producers and the consumers. A description of a change to the system is received. At each node, next-hop neighbor nodes for each consumer are identified. A routing value is assigned to each next-hop neighbor node for each consumer and the routing values are updated according to an update rule that represents a chromosome in a routing probe. The update rule in a routing probe is selectively updated from a plurality of update rules at the consumer. The probability of selecting a particular update rule is reinforced or decayed based on the success of an update rule in allowing routing probes to create many different efficient routes. At each producer, nests of scouting probes are adaptively selected from an available set of nests and dispatched to execute hypothetical placement of a query by an independent agent called a 'leader'. A placement of the operator components that minimizes performance cost of the system relative to the hypothetical placement is selected. Each scouting probe contains chromosomes that guide placement. Scouting probes in two different nests have different chromosomes. The performance cost of the hypothetical changed placement is evaluated and the performance evaluation is used to evolve at least one chromosome of a scouting ant in each nest.

Dynamic and evolutionary placement in an event-driven component-oriented network data processing system

The present invention solves the disadvantages of the prior art and provides a distributed messaging system supporting stateful subscriptions. A stateful publish-subscribe system extends the functionality of the content-based approach to include more general state-valued expressions. A plurality of broker machines is provided to deliver messages sent by publishing clients toward subscribing clients based upon the contents of the messages and stateful transformations requested by the subscribing clients. Subscription specifications are analyzed by a compiler and converted into a collection of message transforms and views. A deployment service component deploys tasks to optimize system performance. A monitoring services component detects a possible need to reconfigure. A performance optimization service component computes new assignment of transforms. A continuous deployment service implements a redeployment protocol that installs changes to transform placement while the existing publish-subscribe system continues to operate.

Continuous feedback-controlled deployment of message transforms in a distributed messaging system

Method, system and computer usable program code for dynamic component placement in an event processing system having producers, consumers, a plurality of nodes between the producers and the consumers, and a flow graph representing computational components to be executed between the producers and the consumers. A description of a change to the system is received. At each node, next-hop neighbor nodes for each consumer are identified. A routing value is assigned to each next-hop neighbor node for each consumer. Using the routing values in a context of the change, a performance cost of the system is estimated based on hypothetical changed placements of the computational components at nodes along paths from a producer to a consumer through the next-hop neighbor nodes for each consumer, and a changed placement of the computational components that minimizes performance cost of the system relative to the hypothetical changed placements is selected.

Dynamic component placement in an event-driven component-oriented network data processing system

A method and system for achieving highly available, fault-tolerant execution of components in a distributed computing system, without requiring the writer of these components to explicitly write code (such as entity beans or database transactions) to make component state persistent. It is achieved by converting the intrinsically non-deterministic behavior of the distributed system to a deterministic behavior, thus enabling state recovery to be achieved by advantageously efficient checkpoint-replay techniques. The method comprises: adapting the execution environment for enabling message communication amongst and between the components; automatically associating a deterministic timestamp in conjunction with a message to be communicated from a sender component to a receiver component during program execution, the timestamp representative of estimated time of arrival of the message at a receiver component. At a component, tracking state of that component during program execution, and periodically checkpointing the state in a local storage device. Upon failure of a component, the component state is restored by recovering a recent stored checkpoint and re-executing the events occurring since the last checkpoint. The system is deterministic by repeating the execution of the receiving component by processing the messages in the same order as their associated timestamp.

Distributed, fault-tolerant and highly available computing system

Method, apparatus and computer program product for fault recovery in a distributed stateful publish-subscribe system. The system includes the capability of recovering from failures that may occur when a stateful publish-subscribe service is implemented on an overlay network. Such failures may include, for example, temporary crashes of broker machines, and network errors causing messages to possibly be lost, duplicated or delivered out of order. The system requires stable storage logging only when a published event enters the system, and requires that logged messages be retrieved from stable storage only in the event all brokers between a failed link or broker and the publishing sites have failed. The publish-subscribe system of the present invention does not require that broker-to-broker connections use reliable FIFO protocols, such as TCP/IP, but may advantageously use faster, less reliable protocols.

Recovery in a distributed stateful publish-subscribe system

Systems, methods and computer program products for optimizing an objective function (e.g. the overall end-to-end runtime latency) of a flow graph of message transformations which are placed onto an overlay network of broker machines by replicating stateless transformations in a guided manner. Exemplary embodiments include a guided replication method, including given an initial transformation graph and a network topology, running a placement algorithm to obtain top N best task-to-broker assignments, identifying the R-task which forms the performance bottleneck, checking if the objective function improvement drops below a designated threshold, if yes, then selecting the optimal task-to-broker assignment and the program terminates; otherwise, replicating the R-task, optimally distributing the data load among R-task's replicas, deriving a set of starting points from the above obtained top N task-to-broker assignments, and re-executing the placement algorithm with these starting points in an iterative fashion.

Systems, methods and computer program products for improving placement performance of message transforms by exploiting guided replication

A method comprises estimating a performance of a processing system using queuing theory wherein the processing system comprises a network that accepts at least one input message stream from a plurality of publishing clients, transforms the at least one input message stream into at least one output message stream utilizing the plurality of transforms distributed in a first assignment among the plurality of brokers, and delivers the at least one output message stream to at least one of the plurality of subscribing clients, performing optimization upon the assignment of the plurality of transforms using the estimated performance to produce a second assignment, and reassigning the plurality of transforms in accordance with the second assignment.

Growing community of inventors

Ridgefield, CT, United States of America

Robert Evan Strom