International Business Machines Corporation

Honghui Xia

Zhen Liu

Li Feng Zhang

Mark Steven Squillante

Shun-Zheng Yu

Hao Yang

Daniel Manuel Dias

Richard Pervin King

Fan Ye

Hanhua Feng

Luis Alfonso Lastras-Montano

Frederick Douglis

Parijat Dube

Nagui Halim

Donald F Towsley

Bin Rong

Akshay Kashyap

James Andrew Broberg

Lei Ying

Petar Momcilovic

Identifying traffic patterns to web sites based on templates that characterize the arrival of traffic to the web sites is provided. Based on these templates, determinations are made as to which web sites should be co-located so as to optimize resource allocation. Web sites whose templates are complimentary, i.e. a first web site having a peak in arrival traffic at time tand a second web site that has a trough in arrival traffic at time t, are designated as being candidates for co-location. In addition, the templates identified for the traffic patterns of web sites are used to determine thresholds for offloading traffic to other servers. These thresholds include a first threshold at which offloading should be performed, a second threshold that takes into consideration the lead time needed to begin offloading, and a third threshold that takes into consideration a lag time needed to stop offloading of traffic.

Co-location and offloading of web site traffic based on traffic pattern recognition

A failure recovery framework to be used in cooperative data stream processing is provided that can be used in a large-scale stream data analysis environment. Failure recovery supports a plurality of independent distributed sites, each having its own local administration and goals. The distributed sites cooperate in an inter-site back-up mechanism to provide for system recovery from a variety of failures within the system. Failure recovery is both automatic and timely through cooperation among sites. Back-up sites associated with a given primary site are identified. These sites are used to identify failures within the primary site including failures of applications running on the nodes of the primary site. The failed applications are reinstated on one or more nodes within the back-up sites using job management instances local to the back-up sites in combination with previously stored state information and data values for the failed applications. In additions to inter-site mechanisms, each one of the plurality of sites employs an intra-site back-up mechanism to handle failure recoveries within the site.

Methods and apparatus for effective on-line backup selection for failure recovery in distributed stream processing systems

Arrangements and methods for developing a software toolkit that can be used to design or obtain parameters for a sensor network. High-level guidelines on the basic relations between sensor network parameters like number of sensors, degree of quantization at each sensor, and the distortion requirements, based on a deep analysis on two basic coding possibilities (multiplexed point-to-point, distributed) are contemplated. By evaluating tradeoffs among the various parameters, an optimization framework to obtain the most cost-effective design with required quantization capabilities pertaining to given distortion criterion is provided.

Method and apparatus for cost-effective design of large-scale sensor networks

A system and method for resource allocation includes, in a network having nodes and links, injecting units of flow for at least one commodity at a source corresponding to the at least one commodity. At each node, queue heights, associated with the at least one commodity, are balanced for queues associated with each of one or more outgoing paths associated with that node. An amount of commodity flow is pushed across a link toward a sink, where the amount of commodity flow is constrained by a capacity constraint. Flow that reached the sink is absorbed by draining the queues.

Distributed resource allocation in stream processing systems

A method to estimate demand of a market for a service is disclosed. The method includes selecting a set of feasible service offerings to offer for sale to the market from a set of candidate service offerings, observing a response of the market to the set of feasible service offerings offered for sale, and estimating a demand of the market for the service based upon the observed response. Each service offering of the set of feasible service offerings and the set of candidate service offerings is defined by a price and a service level.

Market demand estimation method, system, and apparatus

Methods and systems are provided for optimally trading off replication overhead and consistency levels in distributed data replication where nodes are organized in a hierarchy. The root node has the original data that need to be replicated at all other nodes, and the replicated copies have a freshness threshold that must be satisfied. The data are propagated through periodic updates in the hierarchy. Each node periodically sends data to its child nodes. Given the freshness threshold, an algorithm and its distributed protocol can determine the optimal update period for each link of the hierarchy such that the freshness threshold is satisfied for every node and the overall replication overhead is minimized. The systems and methods can be used in any scenario where replicated data have consistency requirements, such as in a replicate overlay assisted resource discovery system.

System and apparatus for optimally trading off the replication overhead and consistency level in distributed applications

Methods and apparatus operating in a stream processing network perform load shedding and dynamic resource allocation so as to meet a pre-determined utility criterion. Load shedding is envisioned as an admission control problem encompassing source nodes admitting workflows into the stream processing network. A primal-dual approach is used to decompose the admission control and resource allocation problems. The admission control operates as a push-and-pull process with sources pushing workflows into the stream processing network and sinks pulling processed workflows from the network. A virtual queue is maintained at each node to account for both queue backlogs and credits from sinks. Nodes of the stream processing network maintain shadow prices for each of the workflows and share congestion information with neighbor nodes. At each node, resources are devoted to the workflow with the maximum product of downstream pressure and processing rate, where the downstream pressure is defined as the backlog difference between neighbor nodes. The primal-dual controller iteratively adjusts the admission rates and resource allocation using local congestion feedback. The iterative controlling procedure further uses an interior-point method to improve the speed of convergence towards optimal admission and allocation decisions.

Distributed joint admission control and dynamic resource allocation in stream processing networks

Apparatus and methods for classifying web sites are provided. With the apparatus and methods, traffic data is obtained for a plurality of web sites. This patterns, or templates, for each web site are generated based on this traffic data and the patterns are clustered into classes of web sites using a clustering algorithm. The clusters, or classes, are then profiled to generate a template for each class. The template for each class is generated by first shifting the patterns for each web site that is part of the class to compensate for effects like time zone differences, if any, and then identifying a pattern that is most similar to all of the patterns in the class. Once the template for each class is generated, this template is then used with traffic data from a new web site to classify the new web site into one of the existing classes. In other words, when traffic data for a new web site is received, a pattern for the traffic data of the new web site is generated and compared to the templates for the various classes. If a matching class template is identified, the new web site is classified into the corresponding class. If the pattern for the new web site does not match any of the existing templates, a new template and class may be generated based on the pattern for the new web site.

Apparatus and methods for classification of web sites

Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network is provided. The methods and system include determining a data transmission cost f; determining a data processing cost f; determining a data storage cost f; and determining a data query Q which minimizes f(f+f+f) for a system of networked data processing resources.

Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network

Apparatus and methods for identifying traffic patterns to web sites based on templates that characterize the arrival of traffic to the web sites are provided. Based on these templates, determinations are made as to which web sites should be co-located so as to optimize resource allocation. Specifically, web sites whose templates are complimentary, i.e. a first web site having a peak in arrival traffic at time tand a second web site that has a trough in arrival traffic at time t, are designated as being candidates for co-location. In addition, the present invention uses the templates identified for the traffic patterns of web sites to determine thresholds for offloading traffic to other servers. These thresholds include a first threshold at which offloading should be performed, a second threshold that takes into consideration the lead time needed to begin offloading, and a third threshold that takes into consideration a lag time needed to stop all offloading of traffic to the other servers.

Apparatus and methods for co-location and offloading of web site traffic based on traffic pattern recognition

Method for modeling the performance of an Information Technology system are disclosed. The method includes the steps of receiving performance data of the system; receiving data of transaction flows and system topology; and inferring service demand parameters based on the received data. If closed form expressions are available to characterize the system, an optimization algorithm based on minimum distance between predicted and measured response times and may be used to obtain the performance parameters. Alternatively, a discrete event simulator together with a set of meta-heuristic search methods may be applied to obtain the optimized performance parameters.

Growing community of inventors

Briarcliff Manor, NY, United States of America

Honghui Xia