International Business Machines Corporation

Dale E Blue

Ryan Thomas Rawlins

Andrew C M Hicks

Andrew C Hicks

Kevin Minerley

Daniel Nicolas Gisolfi

Steven M Partlow

Brian Mo

Rachel Brill

Jacob Thomas Snyder

Eitan D Farchi

Yoram Simha Adler

Noam Behar

Orna X Orna Raz-Pelleg

Shmuel Ur

Jeff Jones

Deborah A Furman

Rachel Tzoref-Brill

Max M Maurer

James E Koopman

James McGurl

Adam L Salvatori

Ruy E Tiapula De Alencar

William H Bleier, Jr

Kevin J Brady

Techniques for testing and fault detection are disclosed. These techniques include generating a set of test cases for a system under test (SUT), the set of test cases based on attribute-value pairs modeled as input to the SUT. The techniques further include augmenting the set of test cases. This includes locating a missing counterpart for a first combination of values in a first test case in the set of test cases, based on identifying a number of instances of the first combination of values in the set of test cases, generating a new test case based on modifying the first test case to act as the missing counterpart, and adding the new test case to the set of test cases. The techniques further include identifying a fault for the SUT based on executing the augmented set of test cases.

Self diagnosing test suite using a double pair-wise combinatorial test solution

Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (S) is computed based on the binary execution results. in response to a success rate of an attribute-value pair being below a predetermined threshold, a subset of testcases that use the attribute-value pair is identified. Further, sets of code paths for the subset of testcases are identified, each set of code path respectively corresponding to a testcase from the subset of testcases. Further, an intersection of the sets of code paths is determined. Code paths of the SUT that are in the intersection, are highlighted to represent a soft failure with the SUT.

System testing infrastructure with hidden variable, hidden attribute, and hidden value detection

A computer-implemented method for test case priority selection includes initiating execution of a set of test cases associated with a system under test. Further, the set of test cases are executed according to a first order of execution, and during the execution, at runtime a failing test case, from the set of test cases is determined. Further, at runtime, from the set of test cases, a subset of test cases that are not yet executed is identified. Further, at runtime, a priority value is assigned to each test case from the subset of test cases, the priority value based on a similarity of the test case with the failing test case. Further, at runtime, the set of test cases is reorganized to execute the subset of test cases in a second order of execution based on priority values that are assigned.

Real time fault localization using combinatorial test design techniques and test case priority selection

A method for testing a system under test (SUT) in an active environment to identify cause of a soft failure includes recording a first difference vector by executing a set of test cases on a baseline system and monitoring performance parameters of the baseline system before and after executing the test cases. Each performance record represents differences in the performance parameters of the baseline system from before and after the execution of a corresponding test case. The method further includes, similarly, recording a second difference vector by executing the test cases on the SUT and monitoring performance parameters of the SUT before and after executing the test cases. The method further includes identifying an outlier performance record from the second difference vector by comparing the difference vectors and further, determining a root cause of the soft failure by analyzing a test case corresponding to the outlier.

Stress test impact isolation and mapping

A method for testing a system under test (SUT) in an active environment includes receiving, by a testing system, a code path of the SUT that causes a soft failure in the active environment. The soft failure occurs in the active environment during execution of the SUT based at least on a parameter of the active environment. The method further includes generating, by the testing system, multiple tests for testing the SUT, the tests generated based on a coverage model of the SUT, wherein the coverage model uses several attributes. The method further includes selecting, by the testing system, from the generated tests, a set of tests that are associated with the code path. The method further includes executing, by the testing system, only the set of tests that are selected on the SUT to analyze a cause of the soft failure.

System testing infrastructure for analyzing and preventing soft failure in active environment

A method for testing a system under test (SUT) in an active environment includes executing, by the testing system, on the SUT, a test from a set of tests. The method further includes, monitoring a first execution time to complete the test on the SUT in the active environment. Based on the first execution time being different than a second execution time of the test, marking, by the testing system, a code path associated with the test. The second execution time is a duration to complete execution of the test on the SUT in a clean execution environment. The method further includes communicating, by the testing system, the code path for analyzing a soft failure of the SUT in the active environment, wherein the soft failure occurs in the active environment during execution of the SUT based at least on a parameter of the active environment.

System testing infrastructure for detecting soft failure in active environment

A method for identifying optimal tests includes defining functional coverage by a test suite based on a functional coverage model of a System Under Test (SUT). The test suite includes a plurality of tests. The functional coverage model includes a plurality of attributes, each attribute having a set of possible values. The functional coverage model defines possible combinations of values of the attributes as covered by the test suite. A subset of the possible combinations of values is determined. A subset of the plurality of tests is selected that is operative to cover the determined subset of the possible combinations of values. A plurality of trees is generated to represent the selected subset of tests. Complexity of the generated trees is analyzed based on user-specified criteria. An optimal tree is selected from the subset of the plurality of trees to achieve the objective of the test suite.

Customizable test set selection using code flow trees

Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (S) is computed based on the binary execution results. The method further includes outputting, to a user, in response to the success rate Sof the attribute-value pair being below a predetermined threshold, an identification of one or more testcases that use the attribute-value pair, wherein the one or more testcases are to be used for diagnosing a soft failure associated with the SUT.

According to one or more embodiments of the present invention, a computer-implemented method for machine code analysis includes executing a set of test cases associated with a software product. The method further includes determining a failing test case, from the set of test cases. The method further includes identifying a portion of a machine code of the software product, the portion of the machine code corresponding to the failing test case. The method further includes analyzing the portion of the machine code to identify a pattern of machine code causing the failing test case to fail. The method further includes searching the machine code of the software product to find the identified pattern of machine code.

Machine code analysis for identifying software defects

Systems, methods, and computer-readable media for identifying a champion test case that provides an increased likelihood of exposing a fault and expanding a set of test cases to include the champion test case are described. The fault may occur in a System Under Test (SUT), which may be a hardware system or a software system. A weight may be assigned to a champion test case that indicates the relative strength of the test case in detecting faults. The weight may be used to influence the selection of other test cases.

Champion test case generation

Systems, methods, and computer-readable media are described for utilizing breakpoint value-based fingerprints of failing regression test cases to determine specific components of a System Under Test (SUT) that are causing a fault such as specific lines of source code. A failing test case from a regression run is selected and fault localization and inverse combinatorics techniques are employed to generate a set of failing test cases around the selected failing test case. A set of test fingerprints corresponding to the set of failing test cases is compared to a set of test fingerprints corresponding to a set of passing test cases to determine breakpoints that are indicated as being encountered during execution of at least one failing test case and that are not encountered during execution of any of the passing test cases. Specific lines of source code that correspond to these breakpoints are then identified as causing the fault.

Fault detection using breakpoint value-based fingerprints of failing regression test cases

A method for testing a system under test (SUT) in an active environment includes generating, by a testing system, a set of tests for testing the SUT, the tests generated based on a coverage model of the SUT, wherein the coverage model uses several attributes. The method further includes creating, by the testing system, a minimal set of tests from the generated tests by selecting tests for a disjoint set of attributes from the several attributes of the coverage model. The method further includes executing, by the testing system, the minimal set of tests on the SUT for analyzing a soft failure of the SUT in the active environment, wherein the soft failure occurs in the active environment during execution of the SUT based at least in part on a parameter of the active environment.

System testing infrastructure for analyzing soft failures in active environment

A method for testing a system under test (SUT) in an active environment includes generating, by a testing system, several tests for testing the SUT. The tests are generated based on a coverage model of the SUT, which includes multiple attributes. The method further includes creating, by the testing system, a minimal set of tests from the tests by selecting tests that do not exceed a predetermined performance threshold. The method further includes executing, by the testing system, the minimal set of tests on the SUT for analyzing a soft failure of the SUT in the active environment. The soft failure occurs in the active environment during execution of the SUT based at least in part on a performance parameter of the active environment.

Growing community of inventors

Poughkeepsie, NY, United States of America

Dale E Blue