International Business Machines Corporation

Kevin Minerley

Andrew C M Hicks

Ryan Thomas Rawlins

Dale E Blue

Andrew C Hicks

Daniel Nicolas Gisolfi

Abdolreza Salahshour

David Matthew Loewenstern

Ma Sheng

Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (S) is computed based on the binary execution results. in response to a success rate of an attribute-value pair being below a predetermined threshold, a subset of testcases that use the attribute-value pair is identified. Further, sets of code paths for the subset of testcases are identified, each set of code path respectively corresponding to a testcase from the subset of testcases. Further, an intersection of the sets of code paths is determined. Code paths of the SUT that are in the intersection, are highlighted to represent a soft failure with the SUT.

System testing infrastructure with hidden variable, hidden attribute, and hidden value detection

A method for testing a system under test (SUT) in an active environment includes receiving, by a testing system, a code path of the SUT that causes a soft failure in the active environment. The soft failure occurs in the active environment during execution of the SUT based at least on a parameter of the active environment. The method further includes generating, by the testing system, multiple tests for testing the SUT, the tests generated based on a coverage model of the SUT, wherein the coverage model uses several attributes. The method further includes selecting, by the testing system, from the generated tests, a set of tests that are associated with the code path. The method further includes executing, by the testing system, only the set of tests that are selected on the SUT to analyze a cause of the soft failure.

System testing infrastructure for analyzing and preventing soft failure in active environment

A method for testing a system under test (SUT) in an active environment includes executing, by the testing system, on the SUT, a test from a set of tests. The method further includes, monitoring a first execution time to complete the test on the SUT in the active environment. Based on the first execution time being different than a second execution time of the test, marking, by the testing system, a code path associated with the test. The second execution time is a duration to complete execution of the test on the SUT in a clean execution environment. The method further includes communicating, by the testing system, the code path for analyzing a soft failure of the SUT in the active environment, wherein the soft failure occurs in the active environment during execution of the SUT based at least on a parameter of the active environment.

System testing infrastructure for detecting soft failure in active environment

Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (S) is computed based on the binary execution results. The method further includes outputting, to a user, in response to the success rate Sof the attribute-value pair being below a predetermined threshold, an identification of one or more testcases that use the attribute-value pair, wherein the one or more testcases are to be used for diagnosing a soft failure associated with the SUT.

A system and related method comprise using a processor for executing a plurality of tests associated with a covering array of a test framework of the software test system on a first version of a system under test (SUT). For each of the plurality of tests, on a current test, the method comprises determining a current success rate value (SRV) for the current test that represents a success rate of the current test for the first version of the SUT. The method further comprises combining the current SRV of the first version of the SUT and current SRVs of the current test for prior versions of the SUT into a current test eigenvector associated with the current test. The method further comprises converting the current test eigenvector into a first eigenvalue that represents a health, accuracy, and quality of the first version of the SUT.

Verification of software test quality using hidden variables

Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (S) is computed based on the binary execution results. An attribute is selected in response to a set of success rates corresponding to a set of attribute-value pairs that includes said attribute are all below a predetermined threshold. The set of testcases is executed using another set of test vectors using additional values for the selected attribute. For each execution of the set of testcases, for each attribute-value pair, a second non-binary success rate (S') is recorded. If the predetermined threshold is now satisfied, a user is notified of the additional values for the attribute that were detected.

A computer processing device receives computer readable data to derive computer executable rules for mining and constructing situation categories. The received data is transformed into a predetermined standard format if the received data is not already in the predetermined standard format. The predetermined standard formatted data is parsed, and an outer, iterative loop is performed until at least one predetermined stopping criterion is met. An inner iterative loop is performed within the outer iterative loop until all desired subsets of data are processed. During the inner iterative loop, selected subsets of data are labeled with labels associated with corresponding previously labeled subsets of data. New computer executable rules are generated for mining and constructing situation categories from the labeled subsets of data. Keyword list classifiers are transformed using the stored labeled subsets of data.

Rules generation for IT resource event situation classification

An off-line knowledge acquisition process takes IT resource messages and automatically generates a set of rules used to provide situation categories for the resource messages/events. The off-line knowledge acquisition process generates an event-to-situation mapping file for efficiently mapping situation to events in runtime. Rules are fed back into a knowledge repository and process for reuse. The off-line knowledge acquisition process provides methods to reiterate the process of mining (autonomically and/or by human interaction) to improve the rules and confidence level assigning the situation categories. A runtime categorizer component uses the event-to-situation mapping file to add situation categories to the IT resource events. The runtime categorizer uses a plurality of annotator components each capable of adding a situation annotation to incoming messages based on rules generated by the off-line knowledge acquisition process. An assessment component chooses the best situation category from those annotated by the annotator components.

IT event data classifier configured to label messages if message identifiers map directly to classification categories or parse for feature extraction if message identifiers do not map directly to classification categories

Classifying a message includes receiving a message to be classified, wherein the message includes a message identifier. If the message identifier uniquely maps to a corresponding classification category, the message is labeled with the identified classification category. If the message identifier does not map directly to a corresponding classification category, the message to be classified is parsed and a plurality of features from the parsed message are identified, wherein at least one classification rule is compared to the plurality of features. Each classification rule that matches to the plurality of features is rated and a classification category is identified from the rating, wherein the message is labeled with the identified classification category.

Growing community of inventors

Red Hook, NY, United States of America

Kevin Minerley