International Business Machines Corporation

Qubole, Inc.

Vijay Mann

Ravi Kothari

Shivkumar Kalyanaraman

Anil Kumar Vishnoi

Kalapriya Kannan

Pradipta De

Girish Bhimrao Chafle

Sunil Chandra

Mohan Dhawan

Kshiteej Sharad Mahajan

Rishabh Poddar

Anilkumar Vishnoi

Manish Satyapal Gupta

Sugata Ghosal

Raghavendra Singh

Shripad Jayant Nadgowda

Mangala Gowri Nanda

Pankaj Kankar

Aakash S Iyer

Naga R Samineni

Praveen Jayachandran

Neeran Mohan Karnik

Partha Dutta

Mudit Verma

Rahul Garg

Praveen Kumar

Ranganathan Vidya

Shubham Tagra

Rajat Venkatesh

Ankit Dixit

Rajiv Sachdev

Ting Yao

Dhruv Sharma

Raunaq Morarka

Mayank Mishra

Akanksha Gupta

The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.

System and method for scheduling and running interactive database queries with service level agreements in a multi-tenant processing system

Methods, systems, and computer program products for diagnosing faults in stateless distributed computing platforms are provided herein. A computer-implemented method includes processing a plurality of communication messages exchanged between two or more distributed components within a stateless distributed system; determining a message context for each of the plurality of communication messages based on one or more message identifiers extracted from each of the plurality of communication messages; grouping the plurality of communication messages into multiple clusters pertaining to message invocation, wherein said grouping is based on the determined message context for each of the plurality of communication messages; generating a control flow graph by creating one or more connections across the multiple clusters; and detecting a causal chain of events associated with a detected fault in the stateless distributed system by navigating the control flow graph.

Diagnosing faults in stateless distributed computing platforms

Methods, systems, and computer program products for software identification, dependency analysis, and manipulation of file system layers of container images are provided herein. A computer-implemented method includes extracting multiple layer files from an input image; identifying software components in each of the extracted layer files; manipulating the input image based on said identifying and user-defined parameters, wherein said manipulating comprises (i) selecting a sub-set of the layer files for retention, (ii) removing a sub-set of the layer files, and (iii) modifying one or more of the layer files selected for retention and/or inserting one or more new layer files in place of one or more of the removed layer files; and outputting a modified version of the input image that includes the sub-set of the multiple layer files selected for retention and any new layer files inserted in place of removed layer files.

Layer identification and dependency analysis for management of images

Methods and arrangements for verifying controller actions in a clustered software defined network. A contemplated method includes: intercepting at least one message at the primary controller; wherein the at least one intercepted message comprises a message that induces one or more changes in the network; intercepting at least one cache update in the network; replicating the at least one intercepted message and the at least one cache update; directing the at least one replicated message and the at least one replicated cache update to a verification subsystem; and verifying the at least one replicated message and the at least one replicated cache update in the verification subsystem. Other variants and embodiments are broadly contemplated herein.

Verifying controller actions in software-defined networks with controller clusters

Methods, systems, and computer program products for container storage migration are provided herein. A computer-implemented method includes provisioning a copy-on-write union filesystem on the target host with (i) one or more network filesystem-mounted directories from the source host set to read-only and (ii) at least one directory set to read-write; identifying one or more files that have been (i) updated by a given container and (ii) copied over by the copy-on-write union filesystem; removing the one or more identified files from a directory path on the target host; copying, from the source host into the directory path on the target host, each of one or more files that have not been updated by the given container; merging the copy-on-write union filesystem and the directory path on the target host; and initiating the given container on the target host based on the merging.

Container storage migration

Methods, systems, and computer program products for implementing a media distribution network for live streaming are provided herein. A computer-implemented method includes directing live video content from a sender device to a first video processing node hosted on a first server within a network, wherein the first server is located within a given geographical proximity of the sender device; determining locations within the network for hosting additional video processing nodes within a given geographical proximity of a receiver device; transmitting the live video content from the first video processing node to an additional video processing node within the network by implementing (i) a multicast overlay network over the network and (ii) a data transfer protocol between the video processing nodes in conjunction with the multicast overlay network; and transmitting the live video content from the additional video processing node to the receiver device associated with the additional video processing node.

Media distribution network for live streaming

Techniques for enabling high-performance computing are provided. The techniques include resizing a logical partition in a non-dedicated compute cluster server to enable high-performance computing, wherein a high performance computing application is executed such that the high performance computing application is configured to complete execution of each of one or more application threads at a similar time as a slowest thread in the cluster, and wherein the non-dedicated compute cluster comprises one or more servers and the logical partition is created by partitioning one or more server resources.

Proportional resizing of a logical partition based on a degree of performance difference between threads for high-performance computing on non-dedicated clusters

Methods and arrangements for securing a software defined network. One or more control messages are intercepted. Information is extracted from the one or more control messages to create a global network state model, and there is determined, from the extracted information, presence of at least one prospective modification to the global network state model. Thereupon, a determination is made as to whether the at least one prospective modification presents a threat to security of the software defined network. Other variants and embodiments are broadly contemplated herein.

Securing of software defined network controllers

Techniques, systems, and articles of manufacture for managing network utility of applications on cloud data centers. A method includes computing a network utility value for each of multiple applications in a data center network, determining a virtual machine placement and an application flow routing within the data center network for each of the multiple applications which have a computed network utility value below a pre-determined network utility threshold, and based on said determining, re-routing one or more of the multiple application flow routings and performing a virtual machine migration of one or more virtual machines associated with at least one of the multiple applications which have a computed network utility value below a pre-determined network utility threshold.

Managing network utility of applications on cloud data centers

Methods and arrangements for providing multicast communication. There is defined a first multicast tenant group comprising a plurality of hosts distributed over a plurality of locations. Communication is established with respect to the first multicast tenant group via: defining a routing tree, and using a multicast address space for communication with the hosts via the routing tree. The multicast address space is reused with respect to a second multicast tenant group. Other variants and embodiments are broadly contemplated herein.

Secure, multi-tenancy aware and bandwidth-efficient data center multicast

According to embodiments of the disclosure, methods, apparatuses, and computer products for controller based network resource management are disclosed. The method may include receiving a flow at a network controller, determining a flow parameter for the flow, and determining, via the network controller, an idle timeout for a flow rule, corresponding to the flow based on the flow parameter. Determining a previous idle timeout for the flow in response to determining that the flow has been received before, and determining an activity factor for the flow rule in response to the previous idle timeout being greater than or equal to the second value. The apparatus may be a network controller including a processor, and a memory configured to receive a flow, determine a flow parameter for the flow, and determine an idle timeout for the flow rule based on the flow parameter.

Controller based network resource management

Techniques, systems, and articles of manufacture for reducing switch state size in flow-based networks. A method includes installing a first hop rule on a first hop switch in a network for a given flow that originates from the first hop switch, wherein the rule adds tags onto a given packet that matches a description of the given flow, wherein the tags correspond to output ports used for forwarding the given packet on each subsequent switch in the network on the path of the given flow; installing and analyzing forwarding rules at each subsequent switch in the network on the path of the given flow to confirm a match with the given packet at each subsequent switch based only on the outermost tag on the given packet; and removing the outermost tag from the given packet and forwarding said given packet to the output port corresponding to said removed outermost tag.

Reducing switch state size in flow-based networks

Providing automated quality-of-service ('QoS') for virtual machine migration across a shared data center network, including: identifying, by a virtual machine migration module, a virtual machine to migrate from a source host to a destination host; determining a migration cost for migrating the virtual machine from the source host to the destination host; identifying based on the migration cost a least congested network path between the source host and the destination host and a bandwidth to be used for migration such that migration completes within a user specified time limit; and initiating transmission of the virtual machine from the source host to the destination host over the least congested network path, including reserving bandwidth across the shared data center network for transferring the virtual machine from the source host to the destination host in dependence upon the migration cost.

Growing community of inventors

Haryana, India

Vijay Mann