International Business Machines Corporation

Globalfoundries Inc.

Avi Ziv

Michal Rimon

Shai Fine

Raviv Gal

Yoav Avraham Katz

Laurent Fournier

Shmuel Ur

Anatoly Koyfman

Eitan Marcus

Giora Simchoni

Elad Yom-Tov

Steven Mittermaier

Yehezkel Azatchi

Keren Zohar

Simon Rushton

Eitan D Farchi

Allon Adir

Debapriya Chatterjee

Rachel Tzoref-Brill

Alexander Ivrii

Itai Segall

Yehuda Naveh

Ziv Nevo

Gil Eliezer Shurek

Itai Jaeger

Christopher John Spandikow

Ronny Morad

John Martin Ludden

Eyal Bin

Alon Gluska

Ariel Jonatan Birnbaum

Yossi Malka

Ari Freund

Haim Kermany

Samuel Solomon Ackerman

Schmuel Ur

Moshe Levinger

Shmuol Ur

A computer that receives a set of names of coverage events. The computer creates, by a machine learning-based technique, groups from the set of received names of the coverage events based on the set of names of the coverage events. The computer generates a cross product coverage model from the created groups and identifies subgroups of uncovered events for each of the created groups.

Coverage analysis with event clustering

An example system includes a processor to receive a template-aware coverage data that tracks probabilities of events in a list of events being hit for a set of test-templates over a first and second predetermined period of time. The processor is to generate a hit prediction score for each combination of unhit event in the events and each test-template in the set of test-templates of the second predetermined period of time. The hit prediction score indicates a probability of an unhit event being hit by a particular test-template in a future third predetermined period of time based on the template-aware coverage data and similarities between the events and the test-templates. The processor is to generate a template score for each test-template based on the hit prediction scores for each test-template. The processor is to select a test-template from the set of test-templates based on the template score.

Selecting test-templates using template-aware coverage data

A method, apparatus and product for testing a system under development. The method comprises obtaining information including a current version of a code, and predicting that a subset of code elements from the current version of the code are bug-prone elements. The method comprises determining one or more coverage events corresponding to the bug-prone elements. The method further comprises determining a testing policy based on the one or more coverage events that correspond to the bug-prone elements, wherein the testing policy is determined based on a statistical analysis of coverage likelihood of tests generated based on a test template for each coverage event of the one or more coverage events. The method further comprises implementing the testing policy.

Risk-focused testing

A method, apparatus and product for reducing a number of test templates in a test suite. The method comprises determining, for a first test template of the test suite, a first probabilities vector comprising a first plurality of coverage probabilities with respect to a set of coverage events. The method comprises determining, for a second test template of the test suite, a second probabilities vector comprising a second plurality of coverage probabilities with respect to the set of coverage events. The method further comprises determining that the first test template is statistically dominant over the second test template based on the first probabilities vector and based on the second probabilities vector. The method further comprises providing an output based on the determination of the statistically dominant test template.

Diminution of test templates in test suites

A method, and apparatus and a computer program product for determining coverage in hardware verification based on relations between coverage events. The method comprises generating an over-approximation model of the hardware being verified to perform formal verification thereof with respect to a target coverage event being utilized in the verification process along with a set of coverage events. A score indicating an estimated conditional probability to hit the target coverage event in the verification process, given that the coverage event is hit in the verification process, may be determined for each coverage event based on the formal verification. The method further comprises selecting test suits to be executed in the verification process based on the scores and the test suits probability to hit each coverage event. The verification process may be the performed the selected test suits in order to cover the target coverage event.

Hardware verification based on relations between coverage events

A computer-implemented method, an apparatus and a computer program for automatically extracting useful information for functional verification. The method comprising performing repeatedly both operating an instruction generator associated with a Design Under Test (DUT), whereby a generated instruction is determined, the generated instruction having one or more instruction attributes; and collecting information relating to the generated instruction. Based on the generated instruction and the collected information, a classification technique is utilized to classify the information based on the instruction attributes.

Automatic identification of information useful for generation-based functional verification

A method, apparatus and product to be used in verification. The method comprising: based on a test generation input that defines a plurality of requirements automatically determining a mutated test generation input, wherein the mutated test generation input defining a mutated requirement which is absent from the test generation input, wherein the mutated requirement is based on a requirement of the plurality of requirements and contradicts, at least in part, the plurality of requirements; and generating one or more test-cases based on the mutated test generation input, whereby the one or more test-cases violate at least one requirement of the test generation input.

Mutations on input for test generation

A method, apparatus and product for generating elements based on generation streams. The method comprises: obtaining one or more generation streams, wherein the streams comprise elements, wherein each element is a formal specification of an operation that stimulates a system, wherein based on each of the generation streams one or more alternative stimuli for the system can be generated, which stimuli comprises operations according to the elements; and generating a stimuli in accordance with the one or more generation streams, wherein the stimuli comprises at least one hybrid operation, wherein the hybrid operation complies simultaneously with two or more elements of the one or more generation stream, whereby the stimuli is comprised of a number of operations that is smaller than a sum of the numbers of elements of the one or more generation streams.

Stream generation

Systems and methods for modeling test space for verifying system behavior, using one or more auxiliary variables, are provided. The method comprises implementing a functional coverage model including: one or more attributes, wherein respective values for the attributes are assigned according to a test plan, and one or more constraints defining restrictions on value combinations assigned to the attributes, wherein the restrictions are Boolean expressions defining whether said value combinations are valid; determining a set of valid value combinations for the attributes that satisfy the restrictions to define the test space for verifying the system behavior; and determining relevant auxiliary variables and a corresponding function for said auxiliary variables to reduce the complexity associated with modeling the test space.

Utilizing auxiliary variables in modeling test space for system behavior

A method, apparatus and product. The method comprising automatically determining an abstract CSP based on a formally defined problem having interconnected sub-problems, wherein the abstract CSP corresponds to the problem, wherein the abstract CSP has a reduced complexity in comparison to a CSP representing the problem, wherein the abstract CSP captures the interconnection between the sub-problems and reduces the details of each sub-problem, wherein the abstract CSP comprises constraints over variables, wherein each variable having an associated domain; and repeatedly: (1) propagating constraints of the abstract CSP to reduce domains of the abstract CSP; (2) selecting a sub-problem to solve; (3) solving the sub-problem; and (4) updating the abstract CSP with values in accordance with the solution of the sub-problem. Whereby, a solution to the formally defined problem is determined based on the solutions to the sub-problems.

Solving constraint satisfaction problems having loosely interconnected sub-problems

Coverage analysis may be performed using reverse time. The coverage analysis may be based on last hit data. The last hit data may comprise a timestamp indicating a last time in which a coverage event was covered. Utilizing last hit data instead of first hit data as is known in the art enables distinction between coverage goals that were never covered and coverage goals that were not covered lately.

Using reverse time for coverage analysis

A logic verification program, method and system that segments simulation results and then processes the resulting segments separately, and optionally in parallel, reduces memory and other system requirements and improves efficiency of verification of digital logic designs. The verification process fixes up event dependency check for past-directed checkers by including additional information with each segment after an initial segment that describes at least a portion of a state of the logic design, so that resultant events in the current segment that are caused by events in the previous segment(s) can be traced back to those events. Future directed checks are fixed-up by either repeating a failed check with a concatenation of the current segment and a next segment, or by providing an overlap between segments to ensure that the expected time duration between a causative event and the resulting event are included within the same segment file.

Techniques for segmenting of hardware trace and verification of individual trace segments

Method, apparatus and product for performing instruction-by-instruction checking on an acceleration platform. The method comprising: simulating by a hardware accelerator an execution of a testcase on a circuit design enhanced by a tracer module, wherein during the simulation the tracer module is configured to collect and record information regarding instruction which are completed by the circuit design and regarding register value modifications; and off-loading the recorded information from the hardware accelerator to a computerized apparatus, whereby based on the off-loaded recorded information, the computerized apparatus can perform an instruction-by-instruction checking that each recorded register modification is justified by an instruction which is was completed prior to the register modification.

Instruction-by-instruction checking on acceleration platforms

An operation of a processor in respect to transactions is checked by simulating an execution of a test program, and updating a transaction order graph to identify a cycle. The graph is updated based on a value read during an execution of a first transaction and a second transaction that is the configured to set the memory with the read value. The test program comprises information useful for identifying the second transaction.

Growing community of inventors

Haifa, Israel

Avi Ziv