International Business Machines Corporation

Globalfoundries Inc.

Itai Segall

Rachel Tzoref-Brill

Aviad Zlotnick

Eitan D Farchi

Edward Barnes Boden

Tatyana Veksler

Ehud Trainin

Danny Harnik

Julian Satran

Avi Ziv

Howard M Hess

Alexander Ivrii

Dalit Naor

Sharon Keidar-Barner

Ariel Jonatan Birnbaum

Muli Ben-Yehuda

Miriam Allalouf

Steven Mittermaier

Hsu Chien-Jen

Techniques for monitoring computer program test design are described herein. The techniques include a method that specifies a subset of tuples in a combinatorial test model as supervised. At least some of the supervised tuples are confirmed using expert data. The method may include determining a metric indicating progress of the combinatorial test modeling process based on the supervised tuples and the confirmed tuples.

Progress metric for combinatorial models

A computer-implemented method for test planning and test case generation, includes collecting, by a processor, a plurality of requirements, creating, by the processor, a plurality of content space specification files that includes the plurality of requirements, processing, by the processor, the plurality of content space specification files to generate a plurality of user stories, outputting, by the processor, the plurality of user stories and integrating, by the processor, the user stories into test planning and test case generation.

Integration of a software content space with test planning and test case generation

A methods, apparatus and product for explaining partially illegal combinations in combinatorial models. The method comprising: obtaining a combinatorial model defining a legal test space, the combinatorial model comprising a set of attributes, a respective domain for each attribute defining possible values for the attribute, and a set of restrictions, wherein the restrictions define a combination of values of the attributes that are illegal and are excluded from the legal test case; obtaining a partially illegal combination defining value assignments to a portion of the attributes; automatically identifying an extension of the partially illegal combination, wherein the extension is excluded from the legal test space, wherein the extension can be modified to become legal by changing a portion of the value assignments defined by the partially illegal combination; and outputting the extension.

Explaining partially illegal combinations in combinatorial models

A method executable on one or more processors for modeling a test space is provided. The method may include defining a coverage model including a set of variables. The method may also include selecting one or more variables within at least one subset of a plurality of subsets of the set of variables. The selection may be according to an interaction level requirement defined for at least one or more of the subsets, whereby the interaction level corresponds to a coverage of the test space that covers a plurality of possible combinations of the one or more variables at multiple levels. Furthermore, respective values for the one or more selected variables within the subset of the set of variables may be assigned. The method may also include one or more definitions for value combinations for said variables with assigned values.

Modeling test space for system behavior with optional variable combinations

A method for modeling a test space is provided. The method comprises defining a coverage model including one or more attributes, wherein respective values for the attributes are assigned, one or more definitions of value properties for said attributes with assigned values, and one or more requirements that limit combination of attribute values that are legal for the model, wherein at least one of said requirements is defined using at least one value property.

Attribute value properties for test selection with cartesian product models

A method for refining a test plan is provided. The method comprises defining a coverage model including: one or more variables, wherein respective values for the variables are assigned, and one or more definitions for value combinations for said variables with assigned values, wherein zero or more of said value combinations are defined according to one or more restrictions for the purpose of generating a test plan to test a system for which the coverage model is constructed; determining zero or more uncovered value combinations in the test plan; and providing means to update the test plan.

Test planning based on dynamic coverage analysis

A method, apparatus and computer-implemented method, the method comprising: receiving a combinatorial testing model comprising an attribute collection, the attribute collection comprising a predetermined number of initial attributes, one or more possible values for each of the initial attributes, a coverage requirement, and an order coverage requirement for the initial attributes, the order coverage requirement related to predetermined relative order in which the attribute values are to be applied; and providing two or more tests in which each of the initial attributes is assigned a value from the possible values, wherein the tests satisfy the order coverage requirement for the at least two attributes.

Test planning with order coverage requirements

An apparatus and computer-implemented method for determining relevance of assignments in combinatorial models, the method comprising: receiving an attribute collection, the attribute collection comprising one or more attributes and one or more possible values for each of attributes; receiving pone or more restrictions, each restriction indicating one or more values for one or more attributes; receiving one or more assignments comprising one or more assigned values for one or more of the attributes; and determining whether the assignment is legal, illegal or partially-legal, wherein an illegal assignment is an assignment which violates a constraint by itself; a legal assignment is an assignment which is not illegal, and for every extension thereof which is illegal, a combination of values assigned to other attributes violates a constraint by itself; and a partially-legal assignment is an assignment which is neither legal nor illegal.

Method and apparatus for determining relevance of assignments in combinatorial models

A method for modeling a test space comprising defining a coverage model including: one or more variables, wherein respective values for the variables are assigned, and restrictions based on which valid variable value combinations are determined for the purpose of testing the model, wherein at least two values that are assignable to the one or more variables are merged to reduce number of variable values in the coverage model.

Automatic designation of equivalent variable values

A method, system and product for explaining illegal combinations in combinatorial models. The method comprising obtaining a combinatorial model and an illegal combination that is excluded from the model by one or more restrictions, utilizing a Satisfiability solver on a satisfiability formula that encodes the legal test space and that assigns values to attributes as defined by the illegal combination, whereby the satisfiability solver provides an indication of unsatisfiability of the satisfiability formula and an UNSAT core comprising a subset of clauses defined by the satisfiability formula which are unsatisfiable; and identifying the one or more restrictions by mapping the clauses of the UNSAT core with clauses encoding the set of restrictions.

Explaining illegal combinations in combinatorial models

A computer-implemented method, computerized apparatus, and computer program product for test validation planning. The computer-implemented method, performed by a processor, comprising: having a test validation activity to be performed to validate results of two or more tests of a test suite; and automatically determining, by a processor, a subset of the two or more tests for which to perform the test validation activity; whereby avoiding performing duplicate validation activities. Optionally, for each test of the test suite a valuation of a set of functional attributes is available, and a subset of the functional attributes is deemed as relevant functional attributes with respect to the test validation activity. In such an embodiment, said determining is based on the valuation of the relevant functional attributes.

Test validation planning

Systems and methods for modeling test space for verifying system behavior, using one or more auxiliary variables, are provided. The method comprises implementing a functional coverage model including: one or more attributes, wherein respective values for the attributes are assigned according to a test plan, and one or more constraints defining restrictions on value combinations assigned to the attributes, wherein the restrictions are Boolean expressions defining whether said value combinations are valid; determining a set of valid value combinations for the attributes that satisfy the restrictions to define the test space for verifying the system behavior; and determining relevant auxiliary variables and a corresponding function for said auxiliary variables to reduce the complexity associated with modeling the test space.

Utilizing auxiliary variables in modeling test space for system behavior

Obtaining a functional coverage model of a System Under Test (SUT) defining all functional coverage tasks of the SUT, wherein the functional coverage model defining a test-space with respect to functional attributes; obtaining a set of covered functional coverage tasks; encoding a covered Binary Decision Diagram (BDD) to represent the set of covered functional coverage tasks within the test-space; and manipulating the covered BDD to identify one or more coverage holes, wherein a coverage hole defines a set of coverage tasks in the test-space, all having a same combination of values to a subset of the functional attributes, that are not covered by the set of covered functional coverage task.

BDD-based functional coverage analysis

A computer-implemented method, apparatus, and computer program product for assisting in dynamic verification of a System Under Test (SUT). The method comprising obtaining a set of functional attributes and associated domains with respect to a System Under Test (SUT), and obtaining a set of restrictions over the functional attributes and associated domains. The method comprising encoding a Binary Decision Diagram (BDD) to represent a Cartesian cross-product test-space of all possible combinations of values of the functional attributes excluding combinations that are restricted by the set of restrictions, whereby the BDD symbolically represents the Cartesian cross-product test-space. The method may further comprise analyzing the Cartesian cross-product test-space by manipulating the BDD so as to assist in performing dynamic verification of the SUT.

Growing community of inventors

Tel Aviv, Israel

Itai Segall